I am trying to make my json file that looks like this
{
"database_details": {
"age": "false",
"help": "xxxx",
"host": "",
"servername": "fra02",
"db_name": "config_tools",
"pass": "xxxx",
"user": "default",
},
{
"age": "false",
"help": "xxxx",
"host": "",
"servername": "fra03",
"db_name": "config_tools",
"pass": "xxxx",
"user": "default",
}
}
to look like
{
"database_details": {
"fra02":{
"age": "false",
"help": "xxxx",
"host": "",
"servername": "fra02",
"db_name": "config_tools",
"pass": "xxxx",
"user": "default",
},
"fra03": {
"age": "false",
"help": "xxxx",
"host": "",
"servername": "fra02",
"db_name": "config_tools",
"pass": "xxxx",
"user": "default",
}
}
}
I have tried jq but not sure if that is the right approach?
I have thought about using awk or sed but not sure that is the cleanest route?
Assuming valid JSON like
{
"database_details": [
{
"age": "false",
"help": "xxxx",
"host": "",
"servername": "fra02",
"db_name": "config_tools",
"pass": "xxxx",
"user": "default"
},
{
"age": "false",
"help": "xxxx",
"host": "",
"servername": "fra03",
"db_name": "config_tools",
"pass": "xxxx",
"user": "default"
}
]
}
you can use the filter
.database_details |= (map({key: .servername, value: .}) | from_entries)
The array associated with database_details is turned into an array of key/value pairs, which from_entries turns into a new object that |= assigns back to the key database_details in the original object.
Related
How to write the JSON schema for both of the payloads by combining in a single schema alert_type: "change" with alert_sub_type: "check change"
alert_type: "change" with alert_sub_type: "check change"
{
"alert_details": {
"event_date": "2021-04-26T12:30:80Z",
"camount": "789",
"profile": {
"name": {
"first_name": "xxxx",
"last_name": "xxxx",
"middle_name": "xxx"
}
},
"check_cash_date": "2021-04-26",
"profile_address": {
"name": "xxxxx",
"street_address": "xxxxxx",
"city": "xxxx",
"state": "CA",
"zip_code": "90021"
},
"alert_id": {
"id": "abc123",
"subject": "abc123",
"abc_id": "abc123"
}
},
"alert_sub_type": "check change",
"alert_type": "change",
"provider": "ABCD",
"view_date": "2020-11-03T10:15:30Z",
"status": "Read"
}
alert_type : "review" alert_sub_type: "review check"
{
"alert_details": {
"event_date": "2020-11-03T10:15:30Z",
"account_number": "*********xxx",
"check_start_number": "2",
"myprofile": {
"name": {
"first_name": "xxxx",
"last_name": "xxxx",
"middle_name": "M"
}
},
"order_shipped_date": "2021-04-23",
"myprofile_address": {
"name": "xxxxx",
"street_address": "xxxxx",
"city": "xxxx",
"state": "xxxx",
"zip_code": "90021"
},
"quantity_ordered": "12",
"alert_id": {
"id": "abc123",
"subject": "abc123",
"abc_id": "abc123"
}
},
"alert_sub_type": "review Check",
"alert_type": "review",
"provider": "abcd",
"view_date": "2020-11-03T10:15:30Z",
"status": "Read"
}
in this alert_type: "change" with alert_sub_type: "check change" mandatory fields are
camount: ""
profile: ""
check_cash_date: ""
profile_address_name: ""
in the alert_type : "review" alert_sub_type: "review check"
mandatory fields are :
check_cash_date: ""
myprofile: ""
"quantity_ordered"
I'm a JQ newb, and have been wrestling with this task for while.
Our source JSON file looks like this...
[
{
"id": "xxxx",
"title": "xxxx",
"created_at": "xxxx",
"updated_at": "xxxx",
"fields": [
{
"usernamevalue": "xxxx",
"reference": "xxxx"
},
{
"passwordvalue": "xxxx",
"reference": "xxxx"
},
{
"otherlabel": "xxxx",
"otherreference": "xxxx"
}
]
},
{
"id": "xxxx",
"title": "xxxx",
"created_at": "xxxx",
"updated_at": "xxxx",
"fields": [
{
"usernamevalue": "xxxx",
"reference": "xxxx"
},
{
"passwordvalue": "xxxx",
"reference": "xxxx"
},
{
"otherlabel": "xxxx",
"otherreference": "xxxx"
}
]
}
]
We are trying to end up with...
[
{
"title": "xxxx",
"updated_at": "xxxx",
"usernamevalue": "xxxx",
"passwordvalue": "xxxx"
},
{
"title": "xxxx",
"updated_at": "xxxx",
"usernamevalue": "xxxx",
"passwordvalue": "xxxx"
}
]
As such, the goals are...
End up with one, "flat" item for each entry -- so that we can then convert it to a CSV.
Only get the "title" and "updated_at" fields from the top level.
Only get fields with keys = "usernamevalue" and "passwordvalue" from the subordinate "fields" array, and move those fields to the top level.
Any guidance would be GREATLY appreciated!
(I've already spent more hours on this than I care to admit.)
THANK YOU
If usernamevalue and passwordvalue have only one occurrence as in your sample file, you can add the fields object, and then extract from it as you would with the outside keys:
jq 'map({id, updated_at} + (.fields | add | {usernamevalue, passwordvalue}))'
[
{
"id": "xxxx",
"updated_at": "xxxx",
"usernamevalue": "xxxx",
"passwordvalue": "xxxx"
},
{
"id": "xxxx",
"updated_at": "xxxx",
"usernamevalue": "xxxx",
"passwordvalue": "xxxx"
}
]
Demo
If your ultimate goal is a CSV output of the values, you'd rather go for an array than an object, which could be something like:
jq -r '.[] | [.id, .updated_at, (.fields | add | .usernamevalue, .passwordvalue)] | #csv'
"xxxx","xxxx","xxxx","xxxx"
"xxxx","xxxx","xxxx","xxxx"
Demo
You could parse the JSON object through their keys.
var jsonObject = [
{
"foo": "bar",
"foo1": "bar1",
"field": [
{
"username": "john#gm.com",
"context": "foob"
},
{
"password": "kek",
"context": "bar1"
},
{
"other": "foobar",
"context": "foobarbar"
}
]
}
]
for (var i = 0; i < jsonObject.field.length; i++)
{
for(var key in jsonObject.field[i])
{
if(key == "username")
{
jsonObject.usernamevalue = jsonObject.field[i].username;
} else if(key == "password")
{
jsonObject.passwordvalue = jsonObjecct.field[i].password;
}
}
}
//Afterwards, you can delete the field
delete jsonObject.field;
I have a long list of json file . I want to extract the subdomain of harvard.edu which is in the variable host in "host": "ceonlineb2b.hms.harvard.edu using bash . I would be happy if anyone can help out .Below is only a snippet of json file.
{
"data": {
"total_items": 3,
"offset": 0,
"limit": 1,
"items": [
{
"name": "ceonlineb2b.hms.harvard.edu",
"alexa": null,
"cert_summary": null,
"dns_records": {
"A": [
"3.221.168.206",
"54.174.253.3"
],
"AAAA": null,
"CAA": null,
"CNAME": [
"hms-moodleb2b-prod.cabem.com"
],
"MX": null,
"NS": null,
"SOA": null,
"TXT": null,
"SPF": null,
"updated_at": "2021-05-14T23:12:43.332816923Z"
},
"hosts_enrichment": [
{
"ip": "3.221.168.206",
"as_num": 14618,
"as_org": "amazon-aes",
"isp": "amazon.com",
"city_name": "ashburn",
"country": "united states",
"country_iso_code": "us",
"location": {
"lat": 39.0481,
"lon": -77.4728
}
},
{
"ip": "54.174.253.3",
"as_num": 14618,
"as_org": "amazon-aes",
"isp": "amazon.com",
"city_name": "ashburn",
"country": "united states",
"country_iso_code": "us",
"location": {
"lat": 39.0481,
"lon": -77.4728
}
}
],
"http_extract": {
"cookies": [
{
"domain": "",
"expire": "0001-01-01T00:00:00Z",
"http_only": true,
"key": "MoodleSession",
"max_age": 0,
"path": "/",
"security": true,
"value": "tqhmqc4muk513sad1bmnl3kocj"
}
],
"description": "",
"emails": null,
"final_redirect_url": {
"full_uri": "https://ceonlineb2b.hms.harvard.edu/login/index.php",
"host": "ceonlineb2b.hms.harvard.edu",
"path": "/login/index.php"
},
"extracted_at": "2020-10-04T20:55:26.043777194Z",
"favicon_sha256": "",
"http_headers": [
{
"name": "date",
"value": "Sun, 04 Oct 2020 20:55:25 GMT"
},
{
"name": "content-type",
"value": "text/html; charset=utf-8"
},
{
"name": "server",
"value": "Apache/2.4.46 () OpenSSL/1.0.2k-fips"
},
{
"name": "x-powered-by",
"value": "PHP/7.2.24"
},
{
"name": "content-language",
"value": "en"
},
{
"name": "content-script-type",
"value": "text/javascript"
},
{
"name": "content-style-type",
"value": "text/css"
},
{
"name": "x-ua-compatible",
"value": "IE=edge"
},
{
"name": "cache-control",
"value": "private, pre-check=0, post-check=0, max-age=0, no-transform"
},
{
"name": "pragma",
"value": "no-cache"
},
{
"name": "expires",
"value": ""
},
{
"name": "accept-ranges",
"value": "none"
},
{
"name": "set-cookie",
"value": "MoodleSession=tqhmqc4muk513sad1bmnl3kocj; path=/; secure;HttpOnly;Secure;SameSite=None"
}
],
"http_status_code": 200,
"links": [
{
"anchor": "Forgotten your username or password?",
"url": "https://ceonlineb2b.hms.harvard.edu/login/forgot_password.php",
"url_host": "ceonlineb2b.hms.harvard.edu"
},
{
"anchor": "Privacy Statement",
"url": "/local/staticpage/view.php?page=privacy-statement",
"url_host": ""
},
{
"anchor": "Terms of Service",
"url": "/local/staticpage/view.php?page=terms-of-service",
"url_host": ""
},
{
"anchor": "Copyright Information",
"url": "/local/staticpage/view.php?page=copyright-information",
"url_host": ""
}
],
"meta_tags": [
{
"name": "keywords",
"value": "moodle, HMS Postgraduate Courses: Log in to the site"
},
{
"name": "format-detection",
"value": "telephone=no"
},
{
"name": "robots",
"value": "noindex"
},
{
"name": "viewport",
"value": "width=device-width, initial-scale=1.0"
}
],
"robots_txt": "",
"scripts": [
"https://ceonlineb2b.hms.harvard.edu/theme/yui_combo.php?rollup/3.17.2/yui-moodlesimple-min.js",
"https://ceonlineb2b.hms.harvard.edu/lib/javascript.php/1589465014/lib/javascript-static.js",
"https://ceonlineb2b.hms.harvard.edu/lib/javascript.php/1589465014/lib/requirejs/require.min.js",
"https://ceonlineb2b.hms.harvard.edu/theme/javascript.php/hms/1589465013/footer"
],
"styles": [
"https://ceonlineb2b.hms.harvard.edu/theme/yui_combo.php?rollup/3.17.2/yui-moodlesimple-min.css",
"https://ceonlineb2b.hms.harvard.edu/theme/styles.php/hms/1589465013_1/all"
],
"title": "HMS Postgraduate Courses: Log in to the site"
},
"is_CNAME": null,
"is_MX": null,
"is_NS": null,
"is_PTR": null,
"is_subdomain": true,
"name_without_suffix": "ceonlineb2b.hms.harvard",
"updated_at": "2021-05-16T10:25:01.59086376Z",
"user_scan_at": null,
"whois_parsed": null,
"security_score": {
"score": 100
},
"cve_list": null,
"technologies": [
{
"name": "Moodle",
"version": ""
},
{
"name": "RequireJS",
"version": ""
}
],
"trackers": null,
"organizations": null
}
]
}
}
For json parsing on bash, I recommend checking out jq. It's lightweight and versatile.
We can use the -r flag to output only values.
Output the fields of each object with the keys in sorted order.
--raw-output / -r:
The structure of the JSON you provided has the subdomain at .data.items[].http_extract.final_redirect_url.host
{
"data": {
"items": [
{
"http_extract": {
"final_redirect_url": {
"full_uri": "https://ceonlineb2b.hms.harvard.edu/login/index.php",
"host": "ceonlineb2b.hms.harvard.edu",
"path": "/login/index.php"
},
...
I've saved your json to a file, se.json
Example extracting full domain with jq
jq -r '.data.items[].http_extract.final_redirect_url.host' se.json
Output
ceonlineb2b.hms.harvard.edu
To extract the subdomain, just perform a search/replace using sub().
sub(regex; tostring) sub(regex; string; flags)
Emit the string obtained by replacing the first match of regex in the input string with tostring, after interpolation. tostring should be a jq string, and may contain references to named captures. The named captures are, in effect, presented as a JSON object (as constructed by capture) to tostring, so a reference to a captured variable named "x" would take the form: "(.x)".
Extracting subdomain using jq
jq -r '.data.items[].http_extract.final_redirect_url.host | sub(".hms.harvard.edu";"")' se.json
Output
ceonlineb2b
I would like to validate the following json:
{
"endpoints": [
{
"ip_name": "plantuml_ip",
"ip": "172.18.0.2",
"ports": [
{
"name": "plantuml_port",
"port": 8080,
"proto": "HTTPS_TERM"
}
]
}
],
"subdomains": [
{
"name": "plantuml_port",
"value": "plantuml"
}
]
}
If proto contains value HTTPS_TERM, then subdomains with the same name should exists (plantuml_port). Here is an another valid example:
Valid:
{
"endpoints": [{
"ip_name": "plantuml_ip",
"ip": "172.18.0.2",
"ports": [{
"name": "plantuml_port",
"port": 8080,
"proto": "HTTPS_TERM"
}, {
"name": "random_port",
"port": 8181,
"proto": "HTTPS_TERM"
}, {
"name": "no_subdomain",
"port": 999,
"proto": "NO_SUBDOMAIN"
}]
}],
"subdomains": [{
"name": "plantuml_port",
"value": "plantuml"
}, {
"name": "random_port",
"value": "random"
}]
}
It is something like this possible? Thanks in advance!
I'm trying to push JSON from a service in to Firebase, and it's not accepting it.
I cant figure out why. JSONLint validates it as valid.
The error I get is:
error: "Invalid data; couldn't parse JSON object, array, or value."
Here's the JSON payload:
{
"app_id": "e4805b8d5a9f4032b0bb8b6d9c6726b8",
"archived": false,
"attachments": {
"form.xml": {
"content_type": "text/xml",
"length": 4500,
"url": "https://someurl.com/form.xml"
}
},
"build_id": "3c5703e20346462ebcb07a3f36d5fe9b",
"domain": "my-test",
"edited_by_user_id": null,
"edited_on": null,
"form": {
"#type": "data",
"#name": "Beneficiary Registration",
"#uiVersion": "1",
"#version": "1",
"#xmlns": "http://openrosa.org/formdesigner/123456",
"beneficiary_age": {
"beneficiary_age_num": "8",
"beneficiary_exact_age_ind": "N"
},
"beneficiary_gender_cd": "M",
"beneficiary_id": "Hhr-O6I3C5L-J3G1L",
"case": {
"#case_id": "a07972bc-1a34-4c84-90ea-91250639f2a4",
"#date_modified": "2020-03-17T20:57:26.986000Z",
"#user_id": "2275c340c48ac83b6852035b0a15b5d3",
"#xmlns": "http://someurl.org/case/transaction/v2"
},
"case_name": "Hhr-O6I3C5L-J3G1L|joel galager|Katsekera|Katsekera|Mpando|Katsekera|KE",
"existing_beneficiaries": "Jane Doe\n\nJoan Doe",
"existing_beneficiaries_label": "",
"household_information": {
"beneficiary_household_information_display": "",
"beneficiary_location_hierarchy_1": "KE",
"beneficiary_location_hierarchy_1_text": "Kenya",
"beneficiary_location_hierarchy_2": "HF0001",
"beneficiary_location_hierarchy_2_text": "Katsekera",
"beneficiary_location_hierarchy_3": "TA0001",
"beneficiary_location_hierarchy_3_text": "Mpando",
"beneficiary_location_hierarchy_4": "GHV0001",
"beneficiary_location_hierarchy_4_text": "Katsekera",
"beneficiary_location_hierarchy_5": "V0001",
"beneficiary_location_hierarchy_5_text": "Katsekera",
"correct_information": "",
"hh_first_name": "John",
"hh_full_name": "John Doe",
"hh_id_fk": "Hhr-O6I3C5L",
"hh_last_name": "Doe",
"household_information": ""
},
"meta": {
"#xmlns": "http://openrosa.org/jr/xforms",
"appVersion": "Formplayer Version: 2.47",
"app_build_version": 1,
"commcare_version": null,
"deviceID": "Formplayer",
"drift": "0",
"geo_point": null,
"instanceID": "123-321-232",
"timeEnd": "2020-03-17T20:57:26.986000Z",
"timeStart": "2020-03-17T20:57:13.958000Z",
"userID": "2275c340c48ac83b6852035b0a15b5d3",
"username": "some_username"
},
"name_group": {
"beneficiary_first_name": "joel",
"beneficiary_full_name": "joel galager",
"beneficiary_last_name": "galager"
},
"subcase_0": {
"case": {
"#case_id": "2a4bfe27-a5c3-4f3a-8540-5f8ded86db85",
"#date_modified": "2020-03-17T20:57:26.986000Z",
"#user_id": "abc123",
"#xmlns": "http://commcarehq.org/case/transaction/v2",
"create": {
"case_name": "Hhr-O6I3C5L-J3G1L|joel galager|Katsekera|Katsekera|Mpando|Katsekera|KE",
"case_type": "beneficiary_case",
"owner_id": "abc123"
},
"index": {
"parent": {
"#text": "a07972bc-1a34-4c84-90ea-91250639f2a4",
"#case_type": "household_case"
}
},
"update": {
"beneficiary_age_num": "8",
"beneficiary_exact_age_ind": "N",
"beneficiary_first_name": "joel",
"beneficiary_full_name": "joel galager",
"beneficiary_gender_cd": "M",
"beneficiary_id": "Hhr-O6I3C5L-J3G1L",
"beneficiary_last_name": "galager",
"beneficiary_location_hierarchy_1": "KE",
"beneficiary_location_hierarchy_1_text": "Kenya",
"beneficiary_location_hierarchy_2": "HF0001",
"beneficiary_location_hierarchy_2_text": "Katsekera",
"beneficiary_location_hierarchy_3": "TA0001",
"beneficiary_location_hierarchy_3_text": "Mpando",
"beneficiary_location_hierarchy_4": "GHV0001",
"beneficiary_location_hierarchy_4_text": "Katsekera",
"beneficiary_location_hierarchy_5": "V0001",
"beneficiary_location_hierarchy_5_text": "Katsekera",
"hh_id_fk": "Hhr-O6I3C5L"
}
}
}
},
"id": "d547ccea-503c-4c50-b974-38ed564ae78a",
"indexed_on": "2020-03-17T21:01:04.695654",
"initial_processing_complete": true,
"is_phone_submission": true,
"metadata": {
"appVersion": "Formplayer Version: 2.47",
"app_build_version": 1,
"commcare_version": null,
"deviceID": "Formplayer",
"drift": "0",
"geo_point": null,
"instanceID": "01010101",
"location": null,
"timeEnd": "2020-03-17T20:57:26.986000Z",
"timeStart": "2020-03-17T20:57:13.958000Z",
"userID": "2275c340c48ac83b6852035b0a15b5d3",
"username": "myusername"
},
"problem": null,
"received_on": "2020-03-17T20:57:27.179986Z",
"resource_uri": "",
"server_modified_on": "2020-03-17T20:57:27.382548Z",
"type": "data",
"uiversion": "1",
"version": "1"
}
This is probably because of #type inside form.
The # character can't be used in RTDB paths.
See How data is structured from the docs.
Keys cannot contain
.
$
#
[
]
/
ASCII control characters 0-31 or 127