How to get a formatted json string from a json object? - json

I'm storing the output of cat ~/path/to/file/blah | jq tojson in a variable to be used later in a curl POST with JSON content. It works well, but it removes all line breaks. I understand line breaks are not supported in JSON, but I'd like them to be replaced with \n characters so when the data is used it isn't all one line.
Is there a way to do this?
Example:
{
"test": {
"name": "test",
"description": "blah"
},
"test2": {
"name": "test2",
"description": "blah2"
}
}
becomes
"{\"test\":{\"name\":\"test\",\"description\":\"blah\"},\"test2\":{\"name\":\"test2\",\"description\":\"blah2\"}}"
but I'd like it to look like
{\n \"test\": {\n \"name\": \"test\",\n \"description\": \"blah\"\n },\n \"test2\": {\n \"name\": \"test2\",\n \"description\": \"blah2\" \n }\n}
I'm actually only converting it to a JSON string so it is able to be posted as part of another JSON. When is it posted, I'd like it to have the format it had originally which can be achieved if there are \n characters.
I can do this manually by doing
cat file | sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' | sed 's/\"/\\"/g')
but this is not ideal.

tojson (or other json outputting filters) will not format the json. It will take on the usual compact form. There is a feature request out there for this so look out for that in a future version.
You could take advantage of jq's regular formatted output, but you'll want to stringify it. You could simulate stringifying by slurping in as raw input, the formatted output. This will read in all of the input as a single string. And since the input was just a json object, it'll produce a string representation of that object.
If you don't mind the extra jq calls, you could do this:
$ var=$(jq '.' input.json | jq -sR '.')
$ echo "$var"
"{\n \"test\": {\n \"name\": \"test\",\n \"description\": \"blah\"\n },\n \"test2\": {\n \"name\": \"test2\",\n \"description\": \"blah2\"\n }\n}\n"
Then of course if your input is already formatted, you could leave out the first jq call.

If your input contains only one JSON value, then jq isn't really buying you much here: all you need is to escape the few characters that are valid in JSON but that don't represent themselves in JSON strings, and you can easily do that using command-line utilities for general-purpose string processing.
For example:
perl -wpe '
s/\\/\\<backslash>/g;
s/\t/\\t/g;
s/\n/\\n/g;
s/\r/\\r/g;
s/"/\\"/g;
s/\\<backslash>/\\\\/g
' ~/path/to/file/blah

Related

jq injects literal newline and escape characters instead of an actual newline

I have the following JSON:
{
"overview_ui": {
"display_name": "my display name",
"long_description": "my long description",
"description": "my description"
}
}
I grab it like so:
overview_ui=$(jq -r ".overview_ui" service.json)
I then want to use it to replace content in another JSON file:
jq -r --arg updated_overview_ui_strings "${overview_ui}" '.overview_ui.${language} |= $updated_overview_ui_strings' someOtherFile.json
This works, however it also introduces visible newline \n and escape \ characters instead of actually preserving the newlines as newlines. Why does it do that?
"en": "{\n \"display_name\": \"my display name\",\n \"long_description\": \"my long description\",\n \"description\": \"my description\"\n}",
You have read the overview_ui variable in as a string (using --arg) so when you assigned it, you assigned that string (along with the formatting). You would either have to parse it as an object (using fromjson) or just use --argjson instead.
jq -r --argjson updated_overview_ui_strings "${overview_ui}" ...
Though, you don't really need to have to do this in multiple separate invocations, you can read the file in as an argument so you can do it in one call.
$ jq --argfile service service.json --arg language en '
.overview_ui[$language] = $service.overview_ui
' someOtherFile.json

transform json to add array objects

I need to transform an array by adding additional objects -
I have:
"user_id":"testuser"
"auth_token":"abcd"
I need:
"key":"user_id"
"value":"testuser"
"key":"auth_token"
"value":"abcd"
I have been using jq but cant figure out how to do it. Do i need to transform this into a multi-dimensional array first?
I have tried multiple jq queries but cant find the most suitable
When i try using jq i get
jq: error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END (Unix shell quoting issues?) at , line 1
Your input is not json, it's just a bunch of what could be thought of as key/value pairs. Assuming your json input actually looked like this:
{
"user_id": "testuser",
"auth_token": "abcd"
}
You could get an array of key/value pair objects using to_entries.
$ jq 'to_entries' input.json
[
{
"key": "user_id",
"value": "testuser"
},
{
"key": "auth_token",
"value": "abcd"
}
]
If on the other hand your input was actually that, you would need to convert it to a format that can be processed. Fortunately you could read it in as a raw string and probably parse using regular expressions or basic string manipulation.
$ jq -Rn '[inputs|capture("\"(?<key>[^\"]+)\":\"(?<value>[^\"]*)\"")]' input.txt
$ jq -Rn '[inputs|split(":")|map(fromjson)|{key:.[0],value:.[1]}]' input.txt
You can use to_entries filter for that.
Here is jqplay example
Robust conversion of key:value lines to JSON.
If the key:value specifications would be valid JSON except for the
missing punctuation (opening and closing braces etc), then a simple and quite robust approach to converting these key:value pairs to a single valid JSON object is illustrated by the following:
cat <<EOF | jq -nc -R '["{" + inputs + "}" | fromjson] | add'
"user_id": "testuser"
"auth_token" : "abcd"
EOF
Output
{
"user_id": "testuser",
"auth_token": "abcd"
}

jq construct with value strings spanning multiple lines

I am trying to form a JSON construct using jq that should ideally look like below:-
{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": [
"event"
],
"traffic_including": [
"unattributed_traffic"
],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
"attribution_campaign_id",
"attribution_creative",
"attribution_date_adjusted",
"attribution_date_utc",
"attribution_matched_by",
"attribution_matched_to",
"attribution_network",
"attribution_network_id",
"attribution_seconds_since",
"attribution_site_id",
"attribution_site_id",
"attribution_tier",
"attribution_timestamp",
"attribution_timestamp_adjusted",
"attribution_tracker",
"attribution_tracker_id",
"attribution_tracker_name",
"count",
"custom_dimensions",
"device_id_adid",
"device_id_android_id",
"device_id_custom",
"device_id_idfa",
"device_id_idfv",
"device_id_kochava",
"device_os",
"device_type",
"device_version",
"dimension_count",
"dimension_data",
"dimension_sum",
"event_name",
"event_time_registered",
"geo_city",
"geo_country",
"geo_lat",
"geo_lon",
"geo_region",
"identity_link",
"install_date_adjusted",
"install_date_utc",
"install_device_version",
"install_devices_adid",
"install_devices_android_id",
"install_devices_custom",
"install_devices_email_0",
"install_devices_email_1",
"install_devices_idfa",
"install_devices_ids",
"install_devices_ip",
"install_devices_waid",
"install_matched_by",
"install_matched_on",
"install_receipt_status",
"install_san_original",
"install_status",
"request_ip",
"request_ua",
"timestamp_adjusted",
"timestamp_utc"
]
}
What I have tried unsuccessfully thus far is below:-
json_construct=$(cat <<EOF
{
"api_key": "6AEC90B5-4169-59AF-7AC9-D655F83B4825",
"app_guid": "komacca-s-rewards-app-au-ios-production-cv8tx71",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv"
"columns_order": ["attribution_attribution_action","attribution_campaign","attribution_campaign_id","attribution_creative","attribution_date_adjusted","attribution_date_utc","attribution_matched_by","attribution_matched_to","attributio
network","attribution_network_id","attribution_seconds_since","attribution_site_id","attribution_tier","attribution_timestamp","attribution_timestamp_adjusted","attribution_tracker","attribution_tracker_id","attribution_tracker_name","
unt","custom_dimensions","device_id_adid","device_id_android_id","device_id_custom","device_id_idfa","device_id_idfv","device_id_kochava","device_os","device_type","device_version","dimension_count","dimension_data","dimension_sum","ev
t_name","event_time_registered","geo_city","geo_country","geo_lat","geo_lon","geo_region","identity_link","install_date_adjusted","install_date_utc","install_device_version","install_devices_adid","install_devices_android_id","install_
vices_custom","install_devices_email_0","install_devices_email_1","install_devices_idfa","install_devices_ids","install_devices_ip","install_devices_waid","install_matched_by","install_matched_on","install_receipt_status","install_san_
iginal","install_status","request_ip","request_ua","timestamp_adjusted","timestamp_utc"]
}
EOF)
followed by:-
echo "$json_construct" | jq '.'
I get the following error:-
parse error: Expected separator between values at line 10, column 15
I am guessing it is because of the string literal which spans to multiple lines that jq is unable to parse it.
Use jq itself:
my_formatted_json=$(jq -n '{
"api_key": "XXXXXXXXXX-7AC9-D655F83B4825",
"app_guid": "XXXXXXXXXXXXXX",
"time_start": 1508677200,
"time_end": 1508763600,
"traffic": ["event"],
"traffic_including": ["unattributed_traffic"],
"time_zone": "Australia/NSW",
"delivery_format": "csv",
"columns_order": [
"attribution_attribution_action",
"attribution_campaign",
...,
"timestamp_utc"
]
}')
Your input "JSON" is not valid JSON, as indicated by the error message.
The first error is that a comma is missing after the key/value pair: "delivery_format": "csv", but there are others -- notably, JSON strings cannot be split across lines. Once you fix the key/value pair problem and the JSON strings that are split incorrectly, jq . will work with your text. (Note that once your input is corrected, the longest JSON string is quite short -- 50 characters or so -- whereas jq has no problems processing strings of length 10^8 quite speedily ...)
Generally, jq is rather permissive when it comes to JSON-like input, but if you're ever in doubt, it would make sense to use a validator such as the online validator at jsonlint.com
By the way, the jq FAQ does suggest various ways for handling input that isn't strictly JSON -- see https://github.com/stedolan/jq/wiki/FAQ#processing-not-quite-valid-json
Along the lines of chepner's suggestion since jq can read raw text data you could just use a jq filter to generate a legal json object from your script variables. For example:
#!/bin/bash
# whatever logic you have to obtain bash variables goes here
key=XXXXXXXXXX-7AC9-D655F83B4825
guid=XXXXXXXXXXXXXX
# now use jq filter to read raw text and construct legal json object
json_construct=$(jq -MRn '[inputs]|map(split(" ")|{(.[0]):.[1]})|add' <<EOF
api_key $key
app_guid $guid
EOF)
echo $json_construct
Sample Run (assumes executable script is in script.sh)
$ ./script.sh
{ "api_key": "XXXXXXXXXX-7AC9-D655F83B4825", "app_guid": "XXXXXXXXXXXXXX" }
Try it online!

Ruby String Automatically Escaped

I am pulling JSON data from a website and am trying to output it within another JSON array.
The data is being pulled properly, but when I shove it into the array of my own, the data gets escaped...
Example:
"{\n \"company\": {\n \"name\": \"SomeName\", \n \"searching\": false, \n \"status\": \"LEAD\"\n}"
How can I prevent the \ and \n from inserting themselves everywhere?

Pretty print of nested stringified json in logs

I'm tailing some logs, and to be able to read them easier, I use jq (http://stedolan.github.io/jq/), but either it's missing something, or I don't know how to do what I need to do.
So all lines are json, and currently I'm doing:
tail -f /path/to/log | jq .
Issue is, sometimes, I have stuff like this (when logging http responses):
{
"foo": "bar",
"fi": "bo",
"stream": "{\n \"json\": \"asAString\"\n}"
}
And obviously, would like to end up with something like this:
{
"foo": "bar",
"fi": "bo",
"stream": {
"json": "asAString"
}
}
Although we can assume in a first time I know the specific name of this fields that needs to be processed in a particular way, if you have an automated way to do that :-)
Thanks!
That's not a nested object... it's a json string. It's already pretty printed as it should. You need to parse the string if that's what you want.
.stream |= fromjson
For me what worked is to take your json string and pipe it to jq using the -r (raw) flag. for example if you have a json object where the value is a long stringified log message like
{"message":"some long string with \"escaped characters \n"}
You take that input and pipe it to jq like | jq -r .message