I have the following JSON:
{
"overview_ui": {
"display_name": "my display name",
"long_description": "my long description",
"description": "my description"
}
}
I grab it like so:
overview_ui=$(jq -r ".overview_ui" service.json)
I then want to use it to replace content in another JSON file:
jq -r --arg updated_overview_ui_strings "${overview_ui}" '.overview_ui.${language} |= $updated_overview_ui_strings' someOtherFile.json
This works, however it also introduces visible newline \n and escape \ characters instead of actually preserving the newlines as newlines. Why does it do that?
"en": "{\n \"display_name\": \"my display name\",\n \"long_description\": \"my long description\",\n \"description\": \"my description\"\n}",
You have read the overview_ui variable in as a string (using --arg) so when you assigned it, you assigned that string (along with the formatting). You would either have to parse it as an object (using fromjson) or just use --argjson instead.
jq -r --argjson updated_overview_ui_strings "${overview_ui}" ...
Though, you don't really need to have to do this in multiple separate invocations, you can read the file in as an argument so you can do it in one call.
$ jq --argfile service service.json --arg language en '
.overview_ui[$language] = $service.overview_ui
' someOtherFile.json
I need to transform an array by adding additional objects -
I have:
"user_id":"testuser"
"auth_token":"abcd"
I need:
"key":"user_id"
"value":"testuser"
"key":"auth_token"
"value":"abcd"
I have been using jq but cant figure out how to do it. Do i need to transform this into a multi-dimensional array first?
I have tried multiple jq queries but cant find the most suitable
When i try using jq i get
jq: error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END (Unix shell quoting issues?) at , line 1
Your input is not json, it's just a bunch of what could be thought of as key/value pairs. Assuming your json input actually looked like this:
{
"user_id": "testuser",
"auth_token": "abcd"
}
You could get an array of key/value pair objects using to_entries.
$ jq 'to_entries' input.json
[
{
"key": "user_id",
"value": "testuser"
},
{
"key": "auth_token",
"value": "abcd"
}
]
If on the other hand your input was actually that, you would need to convert it to a format that can be processed. Fortunately you could read it in as a raw string and probably parse using regular expressions or basic string manipulation.
$ jq -Rn '[inputs|capture("\"(?<key>[^\"]+)\":\"(?<value>[^\"]*)\"")]' input.txt
$ jq -Rn '[inputs|split(":")|map(fromjson)|{key:.[0],value:.[1]}]' input.txt
You can use to_entries filter for that.
Here is jqplay example
Robust conversion of key:value lines to JSON.
If the key:value specifications would be valid JSON except for the
missing punctuation (opening and closing braces etc), then a simple and quite robust approach to converting these key:value pairs to a single valid JSON object is illustrated by the following:
cat <<EOF | jq -nc -R '["{" + inputs + "}" | fromjson] | add'
"user_id": "testuser"
"auth_token" : "abcd"
EOF
Output
{
"user_id": "testuser",
"auth_token": "abcd"
}
I am trying to form a JSON construct using jq that should ideally look like below:-
{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": [
"event"
],
"traffic_including": [
"unattributed_traffic"
],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
"attribution_campaign_id",
"attribution_creative",
"attribution_date_adjusted",
"attribution_date_utc",
"attribution_matched_by",
"attribution_matched_to",
"attribution_network",
"attribution_network_id",
"attribution_seconds_since",
"attribution_site_id",
"attribution_site_id",
"attribution_tier",
"attribution_timestamp",
"attribution_timestamp_adjusted",
"attribution_tracker",
"attribution_tracker_id",
"attribution_tracker_name",
"count",
"custom_dimensions",
"device_id_adid",
"device_id_android_id",
"device_id_custom",
"device_id_idfa",
"device_id_idfv",
"device_id_kochava",
"device_os",
"device_type",
"device_version",
"dimension_count",
"dimension_data",
"dimension_sum",
"event_name",
"event_time_registered",
"geo_city",
"geo_country",
"geo_lat",
"geo_lon",
"geo_region",
"identity_link",
"install_date_adjusted",
"install_date_utc",
"install_device_version",
"install_devices_adid",
"install_devices_android_id",
"install_devices_custom",
"install_devices_email_0",
"install_devices_email_1",
"install_devices_idfa",
"install_devices_ids",
"install_devices_ip",
"install_devices_waid",
"install_matched_by",
"install_matched_on",
"install_receipt_status",
"install_san_original",
"install_status",
"request_ip",
"request_ua",
"timestamp_adjusted",
"timestamp_utc"
]
}
What I have tried unsuccessfully thus far is below:-
json_construct=$(cat <<EOF
{
"api_key": "6AEC90B5-4169-59AF-7AC9-D655F83B4825",
"app_guid": "komacca-s-rewards-app-au-ios-production-cv8tx71",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv"
"columns_order": ["attribution_attribution_action","attribution_campaign","attribution_campaign_id","attribution_creative","attribution_date_adjusted","attribution_date_utc","attribution_matched_by","attribution_matched_to","attributio
network","attribution_network_id","attribution_seconds_since","attribution_site_id","attribution_tier","attribution_timestamp","attribution_timestamp_adjusted","attribution_tracker","attribution_tracker_id","attribution_tracker_name","
unt","custom_dimensions","device_id_adid","device_id_android_id","device_id_custom","device_id_idfa","device_id_idfv","device_id_kochava","device_os","device_type","device_version","dimension_count","dimension_data","dimension_sum","ev
t_name","event_time_registered","geo_city","geo_country","geo_lat","geo_lon","geo_region","identity_link","install_date_adjusted","install_date_utc","install_device_version","install_devices_adid","install_devices_android_id","install_
vices_custom","install_devices_email_0","install_devices_email_1","install_devices_idfa","install_devices_ids","install_devices_ip","install_devices_waid","install_matched_by","install_matched_on","install_receipt_status","install_san_
iginal","install_status","request_ip","request_ua","timestamp_adjusted","timestamp_utc"]
}
EOF)
followed by:-
echo "$json_construct" | jq '.'
I get the following error:-
parse error: Expected separator between values at line 10, column 15
I am guessing it is because of the string literal which spans to multiple lines that jq is unable to parse it.
Use jq itself:
my_formatted_json=$(jq -n '{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
...,
"timestamp_utc"
]
}')
Your input "JSON" is not valid JSON, as indicated by the error message.
The first error is that a comma is missing after the key/value pair: "delivery_format": "csv", but there are others -- notably, JSON strings cannot be split across lines. Once you fix the key/value pair problem and the JSON strings that are split incorrectly, jq . will work with your text. (Note that once your input is corrected, the longest JSON string is quite short -- 50 characters or so -- whereas jq has no problems processing strings of length 10^8 quite speedily ...)
Generally, jq is rather permissive when it comes to JSON-like input, but if you're ever in doubt, it would make sense to use a validator such as the online validator at jsonlint.com
By the way, the jq FAQ does suggest various ways for handling input that isn't strictly JSON -- see https://github.com/stedolan/jq/wiki/FAQ#processing-not-quite-valid-json
Along the lines of chepner's suggestion since jq can read raw text data you could just use a jq filter to generate a legal json object from your script variables. For example:
#!/bin/bash
# whatever logic you have to obtain bash variables goes here
key=XXXXXXXXXX-7AC9-D655F83B4825
guid=XXXXXXXXXXXXXX
# now use jq filter to read raw text and construct legal json object
json_construct=$(jq -MRn '[inputs]|map(split(" ")|{(.[0]):.[1]})|add' <<EOF
api_key $key
app_guid $guid
EOF)
echo $json_construct
Sample Run (assumes executable script is in script.sh)
$ ./script.sh
{ "api_key": "XXXXXXXXXX-7AC9-D655F83B4825", "app_guid": "XXXXXXXXXXXXXX" }
Try it online!
I'm tailing some logs, and to be able to read them easier, I use jq (http://stedolan.github.io/jq/), but either it's missing something, or I don't know how to do what I need to do.
So all lines are json, and currently I'm doing:
tail -f /path/to/log | jq .
Issue is, sometimes, I have stuff like this (when logging http responses):
{
"foo": "bar",
"fi": "bo",
"stream": "{\n \"json\": \"asAString\"\n}"
}
And obviously, would like to end up with something like this:
{
"foo": "bar",
"fi": "bo",
"stream": {
"json": "asAString"
}
}
Although we can assume in a first time I know the specific name of this fields that needs to be processed in a particular way, if you have an automated way to do that :-)
Thanks!
That's not a nested object... it's a json string. It's already pretty printed as it should. You need to parse the string if that's what you want.
.stream |= fromjson
For me what worked is to take your json string and pipe it to jq using the -r (raw) flag. for example if you have a json object where the value is a long stringified log message like
{"message":"some long string with \"escaped characters \n"}
You take that input and pipe it to jq like | jq -r .message