This question already has answers here:
How to use jq when the variable has reserved characters?
(3 answers)
Closed 6 months ago.
I have a JSON file that I am trying to query with jq. I am unable to retrieve the observations. I am trying to retieve each of the "observations using the following command and not able to get to the result:
cat sample3.json | jq .dataSets[0].series.0:0:0:0:0.observations.0[0]
I am able to retieve up to the series using:
cat sample3.json | jq .dataSets[0].series
But once I try to drill down further I am getting a compile error:
$ cat sample3.json | jq .dataSets[0].series.0:0:0:0:0
jq: error: syntax error, unexpected LITERAL, expecting end of file (Unix shell quoting issues?) at <top-level>, line 1:
.dataSets[0].series.0:0:0:0:0
jq: 1 compile error
I am not sure what I am doing wrong here....
The input file is:
{
"header": {
"id": "b8be2cd5-33bf-4687-9e81-eb032f6f8a71",
"test": false,
"prepared": "2022-09-01T13:30:57.013+02:00",
"sender": {
"id": "ECB"
}
},
"dataSets": [
{
"action": "Replace",
"validFrom": "2022-09-01T13:30:57.013+02:00",
"series": {
"0:0:0:0:0": {
"attributes": [
0,
null,
0,
null,
null,
null,
null,
null,
null,
null,
null,
null,
0,
null,
0,
null,
0,
0,
0,
0
],
"observations": {
"0": [
1.4529,
0,
0,
null,
null
],
"1": [
1.4472,
0,
0,
null,
null
],
"2": [
1.4591,
0,
0,
null,
null
]
}
}
}
}
],
"structure": {
"links": [
{
"title": "Exchange Rates",
"rel": "dataflow",
"href": "https://sdw-wsrest.ecb.europa.eu:443/service/dataflow/ECB/EXR/1.0"
}
],
"name": "Exchange Rates",
"dimensions": {
"series": [
{
"id": "FREQ",
"name": "Frequency",
"values": [
{
"id": "D",
"name": "Daily"
}
]
},
{
"id": "CURRENCY",
"name": "Currency",
"values": [
{
"id": "AUD",
"name": "Australian dollar"
}
]
},
{
"id": "CURRENCY_DENOM",
"name": "Currency denominator",
"values": [
{
"id": "EUR",
"name": "Euro"
}
]
},
{
"id": "EXR_TYPE",
"name": "Exchange rate type",
"values": [
{
"id": "SP00",
"name": "Spot"
}
]
},
{
"id": "EXR_SUFFIX",
"name": "Series variation - EXR context",
"values": [
{
"id": "A",
"name": "Average"
}
]
}
],
"observation": [
{
"id": "TIME_PERIOD",
"name": "Time period or range",
"role": "time",
"values": [
{
"id": "2022-08-29",
"name": "2022-08-29",
"start": "2022-08-29T00:00:00.000+02:00",
"end": "2022-08-29T23:59:59.999+02:00"
},
{
"id": "2022-08-30",
"name": "2022-08-30",
"start": "2022-08-30T00:00:00.000+02:00",
"end": "2022-08-30T23:59:59.999+02:00"
},
{
"id": "2022-08-31",
"name": "2022-08-31",
"start": "2022-08-31T00:00:00.000+02:00",
"end": "2022-08-31T23:59:59.999+02:00"
}
]
}
]
},
"attributes": {
"series": [
{
"id": "TIME_FORMAT",
"name": "Time format code",
"values": [
{
"name": "P1D"
}
]
},
{
"id": "BREAKS",
"name": "Breaks",
"values": []
},
{
"id": "COLLECTION",
"name": "Collection indicator",
"values": [
{
"id": "A",
"name": "Average of observations through period"
}
]
},
{
"id": "COMPILING_ORG",
"name": "Compiling organisation",
"values": []
},
{
"id": "DISS_ORG",
"name": "Data dissemination organisation",
"values": []
},
{
"id": "DOM_SER_IDS",
"name": "Domestic series ids",
"values": []
},
{
"id": "PUBL_ECB",
"name": "Source publication (ECB only)",
"values": []
},
{
"id": "PUBL_MU",
"name": "Source publication (Euro area only)",
"values": []
},
{
"id": "PUBL_PUBLIC",
"name": "Source publication (public)",
"values": []
},
{
"id": "UNIT_INDEX_BASE",
"name": "Unit index base",
"values": []
},
{
"id": "COMPILATION",
"name": "Compilation",
"values": []
},
{
"id": "COVERAGE",
"name": "Coverage",
"values": []
},
{
"id": "DECIMALS",
"name": "Decimals",
"values": [
{
"id": "4",
"name": "Four"
}
]
},
{
"id": "NAT_TITLE",
"name": "National language title",
"values": []
},
{
"id": "SOURCE_AGENCY",
"name": "Source agency",
"values": [
{
"id": "4F0",
"name": "European Central Bank (ECB)"
}
]
},
{
"id": "SOURCE_PUB",
"name": "Publication source",
"values": []
},
{
"id": "TITLE",
"name": "Title",
"values": [
{
"name": "Australian dollar/Euro"
}
]
},
{
"id": "TITLE_COMPL",
"name": "Title complement",
"values": [
{
"name": "ECB reference exchange rate, Australian dollar/Euro, 2:15 pm (C.E.T.)"
}
]
},
{
"id": "UNIT",
"name": "Unit",
"values": [
{
"id": "AUD",
"name": "Australian dollar"
}
]
},
{
"id": "UNIT_MULT",
"name": "Unit multiplier",
"values": [
{
"id": "0",
"name": "Units"
}
]
}
],
"observation": [
{
"id": "OBS_STATUS",
"name": "Observation status",
"values": [
{
"id": "A",
"name": "Normal value"
}
]
},
{
"id": "OBS_CONF",
"name": "Observation confidentiality",
"values": [
{
"id": "F",
"name": "Free"
}
]
},
{
"id": "OBS_PRE_BREAK",
"name": "Pre-break observation value",
"values": []
},
{
"id": "OBS_COM",
"name": "Observation comment",
"values": []
}
]
}
}
}
The .foo syntax cannot be used if the key name has anything but alphanumeric characters or the underscore, or if the first character of the key name is numeric.
Assuming you are using a recent version of jq,
you can always use the form: ."foo", which is actually an abbreviation of the basic form, .["foo"].
So assuming you're using a sufficiently recent version of jq, your query could begin with:
.dataSets[0].series."0:0:0:0:0"
If you are presenting the jq query on a command line, then you may have to escape the double-quotes appropriately, e.g. in a bash shell, by enclosing the jq query in single-quotes.
Given the following JSON (oversimplified for the sake of the example), I need to order the keys by their value. In this case, the order should be id > name > type.
{
"link": [{
"attributes": [{
"value": "ConfigurationElement",
"name": "type"
}, {
"value": "NAME1",
"name": "name"
}, {
"value": "0026a8b4-ced6-410e-9213-e3fcb28b3aab",
"name": "id"
}
],
"href": "href1",
"rel": "down"
}, {
"attributes": [{
"value": "0026a8b4-ced6-410e-9213-k23g15h2u1l5",
"name": "id"
}, {
"value": "ConfigurationElement",
"name": "type"
}, {
"value": "NAME2",
"name": "name"
}
],
"href": "href2",
"rel": "down"
}
],
"total": 2
}
EXPECTED RESULT:
{
"link": [{
"attributes": [{
"value": "0026a8b4-ced6-410e-9213-e3fcb28b3aab",
"name": "id"
}, {
"value": "NAME1",
"name": "name"
}, {
"value": "ConfigurationElement",
"name": "type"
}
],
"href": "href1",
"rel": "down"
}, {
"attributes": [{
"value": "0026a8b4-ced6-410e-9213-k23g15h2u1l5",
"name": "id"
}, {
"value": "NAME2",
"name": "name"
}, {
"value": "ConfigurationElement",
"name": "type"
}
],
"href": "href2",
"rel": "down"
}
],
"total": 2
}
I would be very grateful if anyone could help me out. I tried jq with -S and -s with sort_by(), but this example is way too complex for me to figure it out with my current experience with jq. Thank you a lot!
You can do:
jq '.link[].attributes|=sort_by(.name)'
The |= takes all the paths matched by .link[].attributes, i.e. each "attributes" array, and applies the filter sort_by(.name) to each of them, leaving everything else unchanged.
My original JSON is given below.
[
{
"id": "1",
"name": "AA_1",
"total": "100002",
"files": [
{
"filename": "8665b987ab48511eda9e458046fbc42e.csv",
"filename_original": "some.csv",
"status": "3",
"total": "100002",
"time": "2020-08-24 23:25:49"
}
],
"status": "3",
"created": "2020-08-24 23:25:49",
"filenames": "8665b987ab48511eda9e458046fbc42e.csv",
"is_append": "0",
"is_deleted": "0",
"comment": null
},
{
"id": "4",
"name": "AA_2",
"total": "43806503",
"files": [
{
"filename": "1b4812fe634938928953dd40db1f70b2.csv",
"filename_original": "other.csv",
"status": "3",
"total": "21903252",
"time": "2020-08-24 23:33:43"
},
{
"filename": "63ab85fef2412ce80ae8bd018497d8bf.csv",
"filename_original": "some.csv",
"status": "2",
"total": 0,
"time": "2020-08-24 23:29:30"
}
],
"status": "2",
"created": "2020-08-24 23:35:51",
"filenames": "1b4812fe634938928953dd40db1f70b2.csv&&63ab85fef2412ce80ae8bd018497d8bf.csv",
"is_append": "0",
"is_deleted": "0",
"comment": null
}
]
From this JSON I want to create new objects by combining fields from objects which have status: 2 and their files which also have the same pair, status: 2.
So, I am expecting a JSON array as below.
[
{
"id": "4",
"name": "AA_2",
"file_filename": "63ab85fef2412ce80ae8bd018497d8bf.csv",
"file_status": 2
}
]
So far I tried with this JQ filter:
.[]|select(.status=="2")|[{id:.id,file_filename:.files[].filename,file_status:.files[].status}]
But this produces some invalid data.
[
{
"id": "4", # want to remove this as file.status != 2
"file_filename": "1b4812fe634938928953dd40db1f70b2.csv",
"file_status": "3"
},
{
"id": "4",
"file_filename": "1b4812fe634938928953dd40db1f70b2.csv",
"file_status": "2"
},
{
"id": "4", # Repeat
"file_filename": "63ab85fef2412ce80ae8bd018497d8bf.csv",
"file_status": "3"
},
{
"id": "4", # Repeat
"file_filename": "63ab85fef2412ce80ae8bd018497d8bf.csv",
"file_status": "2"
}
]
How do I filter the new JSON using JQ and remove these duplicate objects?
By applying [] operator to files twice, you're running into a combinatorial explosion. That needs to be avoided, for example:
[ .[] | select(.status == "2") | {id, name} + (.files[] | select(.status == "2") | {file_filename: .filename, file_status: .status}) ]
Online demo
Trying to get unique values stored in items array for each group. somehow it's always mixed...
My JSON looks like this:
{
"start": 1534425916,
"stop": 1535030716,
"groups": [
{
"group": "transmission",
"data": {
"events": 665762,
},
"items": [
{
"item": "manualni",
"data": {
"events": 389158,
}
},
{
"item": "automaticka",
"data": {
"events": 276604,
}
}
]
},
{
"group": "vat",
"data": {
"events": 671924,
},
"items": [
{
"item": "ne",
"data": {
"events": 346221,
}
},
{
"item": "ano",
"data": {
"events": 325703,
}
}
]
}
]
}
Desired result is the following:
{
"id": "transmission",
"value": [
"manualni",
"automaticka",
]
}
{
"id": "vat",
"value": [
"ne",
"ano"
]
}
Tried with this filter on command line:
| jq '{id: .groups[].group, value: [.groups[].items[].item]}'
Which results in the above mentioned mixed up result:
{
"id": "transmission",
"value": [
"manualni",
"automaticka",
"ne",
"ano"
]
}
{
"id": "vat",
"value": [
"manualni",
"automaticka",
"ne",
"ano"
]
}
Any idea how to receive the uniquified values here? Thanks in advance!
This gets the desired result. I think the manual entry under .[] explains why it works.
jq '.groups[] | {"id": .group, "value": [.items[].item]}'
I have json
{
"file1": [{
"username": "myname",
"groupname": "mypassword",
"environment": [{
"name": "UMASK",
"value": "022"
},
{
"name": "DEBUG",
"value": "2"
}]
}]
}
and want to change the value of DEBUG to 5.
Tried with below command
jq .file1[0].environment sandeep.json |jq '(.[] |select(.name ==
"DEBUG") | .value) |= "5"'
this will return me specific portion of json like
[
{
"name": "UMASK",
"value": "022"
},
{
"name": "DEBUG",
"value": "5"
}
]
but I want to see full json with changed value
{
"file1": [{
"username": "myname",
"groupname": "mypassword",
"environment": [{
"name": "UMASK",
"value": "022"
},
{
"name": "DEBUG",
"value": "5"
}]
}]
}
Please suggest me
It should be:
jq '(.file1[].environment[]|select(.name=="DEBUG").value) |= 5' file.json
Output:
{
"file1": [
{
"username": "myname",
"groupname": "mypassword",
"environment": [
{
"name": "UMASK",
"value": "022"
},
{
"name": "DEBUG",
"value": 5
}
]
}
]
}