I have the following json layout:
test.json
{
"end": 9,
"previous_page_uri": null,
"messages": [
{
"error_message": null,
"num_media": "1",
"status": "received"
},
{
"error_message": null,
"num_media": "2",
"status": "received"
}
],
"end1": "end page 1",
"end2": "end page 2"
}
I want to output the .messages object as csv, followed by the "end1" value.
Is there a way to do that in jq?
To produce the csv:
jq '.messages[] | [.error_message, .num_media, .status]|#csv' test.json
which produces this:
",\"1\",\"received\""
",\"2\",\"received\""
How can I add .end1?
You can use , Comma to concatenate the output of two filters and ( ) Parenthesis to specify them separately. For example with the sample input you provided the filter
( .messages[] | [.error_message, .num_media, .status] | #csv ), .end1
generates
",\"1\",\"received\""
",\"2\",\"received\""
"end page 1"
Try it online!
Related
Lets say I have an I/P json file as below. And I want to extract the O/P in a CSV format with the below fields. Specifically, I want to get the value of the key "Gamma" in the o/p if the key "Gamma" exists in "tags" map. If the key doesn't exists, it should just print a NULL value. The expected o/p is below.
generated_time,platform,id,,
2021-09-09:12:03:12,earth,2eeee67748,Ray,2021-08-25 09:41:06
2021-09-09:12:03:12,sun,xxxxx12334,NULL,2021-08-25 10:11:31
[
{
"generated_time": "generated_time",
"platform": "platform",
"id": "id"
},
{
"generated_time": "2021-09-09:12:03:12",
"platform": "earth",
"id": "2eeee67748",
"tags": {
"app": "map",
"Gamma": "Ray",
"null": [
"allow-all-humans"
]
},
"created": "2021-08-25 09:41:06"
},
{
"generated_time": "2021-09-09:12:03:12",
"platform": "sun",
"id": "xxxxx12334",
"tags": {
"component": "machine",
"environment": "hot",
"null": [
"aallow-all-humans"
]
},
"created": "2021-08-25 10:11:31"
}
]
jq has a builtin #csv which renders an array
as CSV with double quotes for strings, and quotes escaped by repetition.
If the additional quoting (as compared to your expected output) isn't an issue, the following
jq --raw-output '
# produce an array for each element in the input array
.[] | [
# containing the first three columns unchanged
.generated_time, .platform, .id,
# if the input element has a field named "tags"
if has("tags")
# then add two more columns and replace an inexistant Gamma with "NULL"
then (.tags.Gamma // "NULL", .created)
# otherwise add two empty columns instead
else (null, null) end
# and convert the array into CSV format
] | #csv
' input.json
will produce
"generated_time","platform","id",,
"2021-09-09:12:03:12","earth","2eeee67748","Ray","2021-08-25 09:41:06"
"2021-09-09:12:03:12","sun","xxxxx12334","NULL","2021-08-25 10:11:31"
I need to print some results with jq to take json.
This is an example:
{
"data": [
{
"time": 20201606,
"event": {
"ip": "127.0.1",
"hostname": "srv1",
"locations": [
"UK",
"site1"
],
"num": 1
}
},
{
"time": 202016034,
"event": {
"ip": "127.0.2",
"hostname": "srv2",
"locations": [
"UK",
"site2"
],
"num": 3
}
}
]
}
Like to generate this output "num, ip, hostname, locations":
1, srv1, 127.0.1, UK,site1
2, srv2, 127.0.2, HK,site2
3, srv3, 127.0.3, LO,site3
How can I print this via jq?
Join locations by a comma, and put the result into an array with other fields. Then join again by a comma followed by a space to get the desired output format. E.g.:
.data[].event | [
.num,
.hostname,
.ip,
(.locations | join(",")) ?
] | join(", ")
Use --raw-output/-r option in the command line invocation to get raw strings instead of JSON strings.
Online demo
At its core, you want to build an array consisting of the values you want:
$ jq '.data[].event | [.num, .hostame, .ip, .locations]' tmp.json
[
1,
null,
"127.0.1",
[
"UK",
"site1"
]
]
[
3,
null,
"127.0.2",
[
"UK",
"site2"
]
]
From there, it's a matter of formatting. First, let's turn the list of locations into a single string:
$ jq '.data[].event | [.num, .hostame, .ip, (.locations|join(","))]' tmp.json
[
1,
null,
"127.0.1",
"UK,site1"
]
[
3,
null,
"127.0.2",
"UK,site2"
]
Next, let's join those strings into a ", "-separated string.
$ jq '.data[].event | [.num, .hostame, .ip, (.locations|join(","))] | join(", ")' tmp.json
"1, , 127.0.1, UK,site1"
"3, , 127.0.2, UK,site2"
Finally, you can use the -r flag to output raw text rather than a JSON string value.
$ jq -r '.data[].event | [.num, .hostame, .ip, (.locations|join(","))] | join(", ")' tmp.json
1, , 127.0.1, UK,site1
3, , 127.0.2, UK,site2
I am new to Bash scripting and jq. I am trying to extract the key value pairs name and transcription.normalized from a JSON object.
I have learned how to get a list of all the values from name and normalized separately but it is not really what I am looking for.
cat submission.json | jq '.documents[] .pages[] .fields[] .name, .documents[] .pages[] .fields[] .transcription.normalized'
I am wondering if I need to perform some sort of loop but just not sure. I really want a single script that pulls those 2 fields in a format that I can easily dump to a CSV file.
This is the example of what the JSON looks like.
{
"id": 1,
"state": "complete",
"substate": null,
"exceptions": [],
"name": "Sender Account Number",
"output_name": null,
"field_definition_attributes": {
"required": false,
"data_type": "Account Number",
"multiline": false,
"consensus_required": false,
"supervision_override": null
},
"transcription": {
"raw": "1685-0441-1",
"normalized": "168504411",
"source": "machine_transcription",
"data_deleted": false,
"user_transcribed": null,
"row_index": null
},
"field_image_url": "/api/v4/image/be167a88-9d1d-43bc-82b2-3d96d8c06656?start_x=0.3110429607297866&start_y=0.1052441592299208&end_x=0.5696909842243418&end_y=0.16043316955780607"
}
You don't need a loop. And jq can produce CSV out of arrays.
jq -r '.documents[].pages[].fields[] | [.name, .transcription.normalized] | #csv' file
I have a messages.json file
[
{
"id": "title",
"description": "This is the Title",
"defaultMessage": "title",
"filepath": "src/title.js"
},
{
"id": "title1",
"description": "This is the Title1",
"defaultMessage": "title1",
"filepath": "src/title1.js"
},
{
"id": "title2",
"description": "This is the Title2",
"defaultMessage": "title2",
"filepath": "src/title2.js"
},
{
"id": "title2",
"description": "This is the Title2",
"defaultMessage": "title2",
"filepath": "src/title2.js"
},
]
I want to create an object
{
"title": "Dummy1",
"title1": "Dummy2",
"title2": "Dummy3",
"title3": "Dummy4"
}
from the top one.
So far I have
jq '.[] | .id' src/messages.json;
And it does give me the IDs
How do I add some random text and make the new object as above?
Can we also create a new JSON file and write the newly created object onto it using jq?
Your output included "title3" so I'll assume that you intended that the second occurrence of "title2" in the input was supposed to refer to "title3".
With this assumption, the following jq program seems to do what you want:
map( .id )
| . as $in
| reduce range(0;length) as $i ({};
. + {($in[$i]): "dummy\(1+$i)"})
In words, extract the values of .id, and then turn each into an object of the form: {(.id) : "dummy\(1+$i)"}
This uses string interpolation, and produces:
{
"title": "dummy1",
"title1": "dummy2",
"title2": "dummy3",
"title3": "dummy4"
}
reduce-free solution
map(.id )
| [., [range(0;length)]]
| transpose
| map( {(.[0]): "dummy\(.[1]+1)"})
| add
Output
Can we also create a new json file and write the newly created object onto it using jq?
Yes, just use output redirection:
jq -f program.jq messages.json > output.json
Addendum
I want a parent object "de" to the already created json file objects
You could just pipe either of the above solutions to: {de: .}
I got below output using: https://stackoverflow.com/a/40330344
(.issues[] | {key, status: .fields.status.name, assignee: .fields.assignee.emailAddress})
Output:
{
"key": "SEA-739",
"status": "Open",
"assignee": null
}
{
"key": "SEA-738",
"status": "Resolved",
"assignee": "user2#mycompany.com"
}
But I need to parse each and every line but it's tough to identify which assignee is for which key as far as key group is concerned. Is this possible to make one bunch in one row using jq?
Expected output:
{ "key": "SEA-739", "status": "Open", "assignee": null }
{ "key": "SEA-738", "status": "Resolved", "assignee": "user2#mycompany.com"}
OR
{ "SEA-739", "Open", null }
{ "SEA-738", "Resolved", user2#mycompany.com }
-c is what you likely need
Using the output you posted above, you can process it further:
jq -c . input
To Give;
{"key":"SEA-739","status":"Open","assignee":null}
{"key":"SEA-738","status":"Resolved","assignee":"user2#mycompany.com"}
Or you can just change your original command
FROM
jq -r '(.issues[] | {key, status: .fields.status.name, assignee: .fields.assignee.emailAddress})'
TO
jq -c '(.issues[] | {key, status: .fields.status.name, assignee: .fields.assignee.emailAddress})'
Not precisely an answer to the long version of the question, but for people who Googled this looking for other single line output formats from jq:
$ jq -r '[.key, .status, .assignee]|#tsv' <<<'
{
"key": "SEA-739",
"status": "Open",
"assignee": null
}
{
"key": "SEA-738",
"status": "Resolved",
"assignee": "user2#mycompany.com"
}'
outputs:
SEA-739 Open
SEA-738 Resolved user2#mycompany.com
#sh rather than #tsv returns:
'SEA-739' 'Open' null
'SEA-738' 'Resolved' 'user2#mycompany.com'
Additionally, there are other output formats to do things such as escape the output, like #html, or encode it, as with #base64. The list is available in the Format strings and escaping section of either the jq(1) man page or online at stedolan.github.io/jq/manual.