I have below json data. Need help in parsing it.
{
"e4624072-a9a2-4181-9649-550b9cfeb7cd||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "A1",
"data1": 1073741824,
"data2": 0,
"data3": 0
}
]
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "e4624072-a9a2-4181-9649-550b9cfeb7cd||220000d7f738801",
"endTime": 96285337200000
},
"0b1141b2-c2de-47c9-aa0c-2742f92b63f2||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B1",
"data1": 5368709120,
"data2": 5368709120,
"data3": 5368709120
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostYES_B1"
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostOff_B1"
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "0b1141b2-c2de-47c9-aa0c-2742f92b63f2||220000d7f738801",
"lifeCycle": 0
},
"23e529f9-b2f3-4730-9b28-4ee05ca678b6||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "A2",
"data1": 1073741824,
"data2": 0,
"data3": 0,
"lastUpdateTime": 1619541451476,
"origInitialVal": 1073741824
}
]
}
],
"name": [
"AMIT||220000d7f728802"
],
"id": "23e529f9-b2f3-4730-9b28-4ee05ca678b6||220000d7f738801",
"endTime": 96285337200000,
"lifeCycle": 0
},
"66b2229b-2c16-4d54-b2a8-fcc1baeaf51c||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B2",
"data1": 10737418240,
"data2": 10737418240,
"data3": 10737418240,
"lastUpdateTime": 1637766239807,
"origInitialVal": 10737418240
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "RMN_B2"
}
],
"name": [
"AMIT||220000d7f728801"
],
"startTime": 1637766000000,
"id": "66b2229b-2c16-4d54-b2a8-fcc1baeaf51c||220000d7f738801",
"endTime": 1637852400000,
"lifeCycle": 0
},
"b896eb1b-d6b0-432b-8925-af17431c0f3e||220000d7f738801": {
"List": [
{
"Insts": [
{
"DefId": "B3",
"data1": 2147483648,
"data2": 2147483648,
"data3": 2147483648,
"lastUpdateTime": 1635692405780
}
]
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostYES_B3"
},
{
"edigest": "MV6shIv5NE5vWkc0cx6Q/JTwid4=",
"csDefRef": "BoostOffCS_B3"
}
],
"name": [
"AMIT||220000d7f728801"
],
"id": "b896eb1b-d6b0-432b-8925-af17431c0f3e||220000d7f738801",
"lifeCycle": 0
}
}
Required output is as below :
AMIT||220000d7f728801,A1,1073741824,0,0
AMIT||220000d7f728801,B1,5368709120,5368709120,5368709120
AMIT||220000d7f728801,A2,1073741824,0,0
AMIT||220000d7f728801,B2,10737418240,10737418240,10737418240
AMIT||220000d7f728801,B3,2147483648,2147483648,2147483648
I tried to execute below jq to start with but it giving as mentioned in title. This might be due to DefId is not present at all the places. Similarly for data 1, data2 and data3.
jq -r '.[] | ."name", ."List"[]."Insts"."DefId"'
What is the right jq command in order to get the correct output?
Unfortunately you say nothing about constraints (guaranteed existence of fields, lengths of arrays, etc.) regarding your data structure. Therefore, simply based on what you have provided, this will extract the contents into your desired output format:
jq -r '.[] | [.name[0] , (.List[0].Insts[0] | .DefId, .data1, .data2, .data3)] | join(",")'
AMIT||220000d7f728801,A1,1073741824,0,0
AMIT||220000d7f728801,B1,5368709120,5368709120,5368709120
AMIT||220000d7f728801,A2,1073741824,0,0
AMIT||220000d7f728801,B2,10737418240,10737418240,10737418240
AMIT||220000d7f728801,B3,2147483648,2147483648,2147483648
Demo
You can dynamically walk within List.Insts array without error by using
.List[].Insts | select( . != null ) for the returning values of to_entries[]
while it's straightforward to extract the value for the name key.
Then join the components of the formed array through use of wrapper square brackets such as
jq -r 'to_entries[] | .value | [ .name[] , (.List[].Insts | select( . != null ) | .[] | .DefId, "\(.data1)", "\(.data2)", "\(.data3)" ) ] | join(",")'
Demo
Related
I have a JSON file who I am trying to convert to CSV using jq, but I've been having a lot of problems, this is the JSON:
{
"nhits": 2,
"parameters": {
"dataset": "real-time-bezettingen-fietsenstallingen-gent",
"rows": 10,
"start": 0,
"facet": [
"facilityname"
],
"format": "json",
"timezone": "UTC"
},
"records": [
{
"datasetid": "real-time-bezettingen-fietsenstallingen-gent",
"recordid": "d471594688a931ba8d81f8d883874a08cee84775",
"fields": {
"id": "48-2",
"freeplaces": 71,
"facilityname": "Braunplein",
"geo_point_2d": [
51.05406845807926,
3.723722319130363
],
"time": "2022-11-10T12:18:01+00:00",
"totalplaces": 116,
"occupiedplaces": 45,
"bezetting": 38
},
"geometry": {
"type": "Point",
"coordinates": [
3.723722319130363,
51.05406845807926
]
},
"record_timestamp": "2022-11-10T12:18:04.838Z"
},
{
"datasetid": "real-time-bezettingen-fietsenstallingen-gent",
"recordid": "d0121748cf31c7e1c02d99712bdf07cb33156689",
"fields": {
"id": "48-1",
"freeplaces": 65,
"facilityname": "Korenmarkt",
"geo_point_2d": [
51.05388288288933,
3.7214177570400473
],
"time": "2022-11-10T12:18:01+00:00",
"totalplaces": 235,
"occupiedplaces": 170,
"bezetting": 72
},
"geometry": {
"type": "Point",
"coordinates": [
3.7214177570400473,
51.05388288288933
]
},
"record_timestamp": "2022-11-10T12:18:04.838Z"
}
],
"facet_groups": [
{
"name": "facilityname",
"facets": [
{
"name": "Braunplein",
"count": 1,
"state": "displayed",
"path": "Braunplein"
},
{
"name": "Korenmarkt",
"count": 1,
"state": "displayed",
"path": "Korenmarkt"
}
]
}
]
}
I only want to have the columns facilityname, time, totalplaces, occupiedplaces and bezetting, I tried converting using the following command:
jq -r '["naam", "tijd", "totaalAantalPlaatsen", "bezettePlaatsen", "bezetting"] , .records[] | (.fields[] | [.facilityname, .time, .totalplaces, .occupiedplaces, .bezetting]) | #csv' data.json
But I get the error:
jq: error (at data.json:0): Cannot index array with string "fields"
Does anyone know what I'm doing wrong?
You just need some parentheses around the .records[] ... part
jq -r '
["name", "tijd", "totaalAantalPlaatsen", "bezettePlaatsen", "bezetting"],
(.records[].fields | [.facilityname, .time, .totalplaces, .occupiedplaces, .bezetting])
| #csv
' file.json
I have a json like this
[
{
"name": "hosts",
"ipaddress": "1.2.3.4",
"status": "UP",
"randomkey": "randomvalue"
},
{
"name": "hosts",
"ipaddress": "5.6.7.8",
"status": "DOWN",
"newkey": "newvalue"
},
{
"name": "hosts",
"ipaddress": "9.10.11.12",
"status": "RESTART",
"anotherkey": "anothervalue"
}
]
I want to merge the objects and looking for some output like this
[
{
"name": "hosts", //doesn't matter if it is ["hosts"]
"ipaddress": ["1.2.3.4", "5.6.7.8", "9.10.11.12"],
"status": ["UP", "DOWN", "RESTART"],
"randomkey": ["randomvalue"],
"newkey": ["newvalue"],
"anotherkey": ["anothervalue"]
}
]
I can hardcode each and every key and do something like this - { ipaddress: (map(.ipaddress) | unique ) } + { status: (map(.status) | unique ) } + { randomkey: (map(.randomkey) | unique ) }
The important ask here is the values are random and cannot be hardcoded.
Is there a way i can merge all the keys without hardcoding the key here?
Using reduce, then unique would be one way:
jq '[
reduce (.[] | to_entries[]) as {$key, $value} ({}; .[$key] += [$value])
| map_values(unique)
]'
[
{
"name": [
"hosts"
],
"ipaddress": [
"1.2.3.4",
"5.6.7.8",
"9.10.11.12"
],
"status": [
"DOWN",
"RESTART",
"UP"
],
"randomkey": [
"randomvalue"
],
"newkey": [
"newvalue"
],
"anotherkey": [
"anothervalue"
]
}
]
Demo
Using group_by and map, then unique again would be another:
jq '[
map(to_entries[]) | group_by(.key)
| map({key: first.key, value: map(.value) | unique})
| from_entries
]'
[
{
"anotherkey": [
"anothervalue"
],
"ipaddress": [
"1.2.3.4",
"5.6.7.8",
"9.10.11.12"
],
"name": [
"hosts"
],
"newkey": [
"newvalue"
],
"randomkey": [
"randomvalue"
],
"status": [
"DOWN",
"RESTART",
"UP"
]
}
]
Demo
I have file with 30, 000 JSON lines delimited by new line. I am using JQ to process it.
Below is each line schema (new.json).
{
"indexed": {
"date-parts": [
[
2020,
8,
13
]
],
"date-time": "2020-08-13T06:27:26Z",
"timestamp": 1597300046660
},
"reference-count": 42,
"publisher": "American Chemical Society (ACS)",
"issue": "3",
"content-domain": {
"domain": [],
"crossmark-restriction": false
},
"short-container-title": [
"Org. Lett."
],
"published-print": {
"date-parts": [
[
2005,
2
]
]
},
"DOI": "10.1021/ol047829t",
"type": "journal-article",
"created": {
"date-parts": [
[
2005,
1,
27
]
],
"date-time": "2005-01-27T05:53:29Z",
"timestamp": 1106805209000
},
"page": "383-386",
"source": "Crossref",
"is-referenced-by-count": 38,
"title": [
"Liquid-Crystalline [60]Fullerene-TTF Dyads"
],
"prefix": "10.1021",
"volume": "7",
"author": [
{
"given": "Emmanuel",
"family": "Allard",
"affiliation": []
},
{
"given": "Frédéric",
"family": "Oswald",
"affiliation": []
},
{
"given": "Bertrand",
"family": "Donnio",
"affiliation": []
},
{
"given": "Daniel",
"family": "Guillon",
"affiliation": []
}
],
"member": "316",
"container-title": [
"Organic Letters"
],
"original-title": [],
"link": [
{
"URL": "https://pubs.acs.org/doi/pdf/10.1021/ol047829t",
"content-type": "unspecified",
"content-version": "vor",
"intended-application": "similarity-checking"
}
],
"deposited": {
"date-parts": [
[
2020,
4,
7
]
],
"date-time": "2020-04-07T13:39:55Z",
"timestamp": 1586266795000
},
"score": null,
"subtitle": [],
"short-title": [],
"issued": {
"date-parts": [
[
2005,
2
]
]
},
"references-count": 42,
"alternative-id": [
"10.1021/ol047829t"
],
"URL": "http://dx.doi.org/10.1021/ol047829t",
"relation": {},
"ISSN": [
"1523-7060",
"1523-7052"
],
"issn-type": [
{
"value": "1523-7060",
"type": "print"
},
{
"value": "1523-7052",
"type": "electronic"
}
],
"subject": [
"Physical and Theoretical Chemistry",
"Organic Chemistry",
"Biochemistry"
]
}
For every DOI, I need to obtain the values of given and family key in the same cell of the same row of that DOI in the CSV/TSV format.
The expected output for the above json is (in CSV/TSV format):
|DOI| givenName|familyName|
|10.1021/ol047829t|Emmanuel; Frédéric; Bertrand; Daniel;|Allard; Oswald; Donnio; Guillon|
I am using the below command line but it is throwing error and when I try to alter I am unable to get CSV/TSV output at all.
cat new.json | jq -r "[.DOI, .publisher, .author[] | .given] | #tsv" > manage.tsv
The same logic applies for subject key also. I am using the below command line to output values of subject key to CSV but it is throwing only the first element (in this case only: "Physical and Theoretical Chemistry")
cat new.json | jq -c -r "[.DOI, .publisher, .subject[0]] | #csv" > manage.csv
Any pointers for right jq command line will be of great help.
Join given and family names by semicolons separately, then pass resulting strings as fields to the TSV filter.
["DOI", "givenName", "familyName"],
(inputs | [.DOI, (.author | map(.given), map(.family) | join("; "))])
| #tsv
Online demo
Note that you need to invoke JQ with -r and -n flags for this to work and produce a valid TSV output.
Can somebody help me to extract with | jq the following:
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"pod": "dev-cds-5c97cf7f78-sw6b9"
},
"values": [
[
1588204800,
"0.3561394483796914"
],
[
1588215600,
"0.3607968456046861"
],
[
1588226400,
"0.3813882532417868"
],
[
1588237200,
"0.6264355815408573"
]
]
},
{
"metric": {
"pod": "uat-cds-66ccc9685-b5tvh"
},
"values": [
[
1588204800,
"0.9969746974696218"
],
[
1588215600,
"0.7400881057270005"
],
[
1588226400,
"1.2298959318837195"
],
[
1588237200,
"0.9482296838254507"
]
]
}
]
}
}
I need to obtain all-values individually by given word dev-cds and not all the name dev-cds-5c97cf7f78-sw6b9.
Result desired:
{
"metric": {
"pod": "dev-cds-5c97cf7f78-sw6b9"
},
"values": [
[
1588204800,
"0.3561394483796914"
],
[
1588215600,
"0.3607968456046861"
],
[
1588226400,
"0.3813882532417868"
],
[
1588237200,
"0.6264355815408573"
]
]
}
You should first iterate over the result array. Check if the pod inside, metric object has the value that contains "dev-cds".
.data.result[] | if .metric.pod | contains("dev-cds") then . else empty end
https://jqplay.org/s/54OH83qHKP
I want to convert a complex JSON file into a simple JSON file using JQ. However, the query I'm using generates an incorrect output.
My (cut down) JSON file:
[
{
"id": 100,
"foo": [
{
"bar": [
{"type": "read"},
{"type": "write"}
],
"users": ["admin_1"],
"groups": []
},
{
"bar": [
{"type": "execute"},
{ "type": "read"}
],
"users": [],
"groups": ["admin_2"]
}
]
},
{
"id": 101,
"foo": [
{
"bar": [
{"type": "read"}
],
"users": [
"admin_3"
],
"groups": []
}
]
}
]
I need to generate a flatter JSON file and combine the users and groups into one field, similar to this:
[
{
"id": 100,
"users_groups": [
"admin_1",
"admin_2"
],
"bar": ["read"]
},
{
"id": 100,
"users_groups": ["admin_1"],
"bar": ["write"]
},
{
"id": 100,
"users_groups": ["admin_2"],
"bar": ["execute"]
},
{
"id": 101,
"users_groups": ["admin_3"],
"bar": ["read"]
}
]
Everything I try in JQ results in me getting an incorrect output (where admin_1 incorrectly has bar=execute and admin_2 incorrectly has bar=write), similar to the following:
[
{
"id": 100,
"users_groups": [
"admin_1",
"admin_2"
],
"bar": ["read", "write", "execute"]
},
{
"id": 101,
"users_groups": ["admin_3"],
"bar": ["read"]
}
]
I have tried many vairiats of this query - any idea what I should be doing instead?
cat file.json | jq -r '[.[] | select(has("foo")) |{"id", "users":(.foo[] | .users), "groups":(.foo[] | .groups), "bar":([.foo[].bar[] | .type])} ] '
The following filter groups by "type" as the question seems to require:
map(.id as $id
| [.foo[]
| {id: $id, bar: .bar[].type} +
{"users_groups": (.users + .groups)[]} ]
| group_by(.bar)
| map(.[0] + {"users_groups": [.[].users_groups]}) )
Output
[
[
{
"id": 100,
"bar": "execute",
"users_groups": [
"admin_2"
]
},
{
"id": 100,
"bar": "read",
"users_groups": [
"admin_1",
"admin_2"
]
},
{
"id": 100,
"bar": "write",
"users_groups": [
"admin_1"
]
}
],
[
{
"id": 101,
"bar": "read",
"users_groups": [
"admin_3"
]
}
]
]
Variations
To achieve the array-of-objects output format, simply tack on | [.[][]];
it would similarly be trivially easy to ensure that .bar is array-valued, though that might be pointless given that the grouping is by .type.