Select or exclude multiples object with an array of IDs - json

I have the following JSON :
[
{
"id": "1",
"foo": "bar-a",
"hello": "world-a"
},
{
"id": "2",
"foo": "bar-b",
"hello": "world-b"
},
{
"id": "10",
"foo": "bar-c",
"hello": "world-c"
},
{
"id": "42",
"foo": "bar-d",
"hello": "world-d"
}
]
And I have the following array store in a variable: ["1", "2", "56", "1337"] (note the IDs are string, and may contain any regular character).
So, thanks to this SO, I found a way to filter my original data. jq 'jq '[.[] | select(.id == ("1", "2", "56", "1337"))]' ./data.json (note the array is surrounded by parentheses and not brackets) produces :
[
{
"id": "1",
"foo": "bar-a",
"hello": "world-a"
},
{
"id": "2",
"foo": "bar-b",
"hello": "world-b"
}
]
But I would also liked to do the opposite (basically excluding IDs instead of selecting them). Using select(.id != ("1", "2", "56", "1337")) doesn't work and using jq '[. - [.[] | select(.id == ("1", "2", "56", "1337"))]]' ./data.json seems very ugly and it doesn't work with my actual data (an output of aws ec2 describe-instances).
So have you any idea to do that? Thank you!

To include them, you need to verify that the id is any of the values in the keep set.
$ jq --argjson include '["1", "2", "56", "1337"]' 'map(select(.id == $include[]))' ...
To exclude them, you need to verify that all values are not in your excluded set. But it might just be easier to take the original set and remove the items that are in the excluded set.
$ jq --argjson exclude '["1", "2", "56", "1337"]' '. - map(select(.id == $exclude[]))' ...

Here is a solution that uses inside. Assuming you run jq as
jq -M --argjson IDS '["1","2","56","1337"]' -f filter.jq data.json
This filter.jq
map( select([.id] | inside($IDS)) )
produces the ids from data.json that are in the $IDS array:
[
{
"id": "1",
"foo": "bar-a",
"hello": "world-a"
},
{
"id": "2",
"foo": "bar-b",
"hello": "world-b"
}
]
and this filter.jq
map( select([.id] | inside($IDS) | not) )
produces the ids from data.json that are not in the $IDS array:
[
{
"id": "10",
"foo": "bar-c",
"hello": "world-c"
},
{
"id": "42",
"foo": "bar-d",
"hello": "world-d"
}
]

Related

how to output all the keys and values from json using jq?

I am trying to out all the data from my json file that matches the value "data10=true" it does that but only grabs the names, how can i make it so it will output everything in my json file with anything that matches the "data10=true"?
this is what ive got data=$(jq -c 'to_entries[] | select (.value.data10 == "true")| [.key, .value.name]' data.json )
This is in my YAML template btw, running it as a pipeline in devops.
The detailed requirements are unclear, but hopefully you'll be able to use the following jq program as a guide:
..
| objects
| select( .data10 == "true" )
| to_entries[]
| select(.key != "data10")
| [.key, .value]
This will recursively (thanks to the initial ..) examine all the JSON objects in the input.
p.s.
If you want to make the selection based on whether .data10 is "true" or true, you could change the criterion to .data10 | . == true or . == "true".
jq 'to_entries | map(select(.value.data10=="true")) | from_entries' data.json
input data.json,
with false value:
{
"FOO": {
"data10": "false",
"name": "Donald",
"location": "Stockholm"
},
"BAR": {
"data10": "true",
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"data10": "true",
"name": "Jack",
"location": "Whereever"
}
}
output:
{
"BAR": {
"data10": "true",
"name": "Walt",
"location": "Stockholm"
},
"BAZ": {
"data10": "true",
"name": "Jack",
"location": "Whereever"
}
}
based on: https://stackoverflow.com/a/37843822/983325

Lookup filtering with jq

Giving a JSON string like this,
[
{
"id": 1,
"name": "Arthur",
"age": "21"
},
{
"id": 2,
"name": "Richard",
"age": "32"
}
]
How to filter by name and get the age?
E.g., given the name being "Richard", let jq return "32". Thx.
$ jq --arg name Richard '.[] | select(.name==$name) | .age' input.json
"32"
When using jq like this in Windows, the quoting would have to be appropriate for Windows.

parsing JSON with jq to return value of element where another element has a certain value

I have some JSON output I am trying to parse with jq. I read some examples on filtering but I don't really understand it and my output it more complicated than the examples. I have no idea where to even begin beyond jq '.[]' as I don't understand the syntax of jq beyond that and the hierarchy and terminology are challenging as well. My JSON output is below. I want to return the value for Valid where the ItemName equals Item_2. How can I do this?
"1"
[
{
"GroupId": "1569",
"Title": "My_title",
"Logo": "logo.jpg",
"Tags": [
"tag1",
"tag2",
"tag3"
],
"Owner": [
{
"Name": "John Doe",
"Id": "53335"
}
],
"ItemId": "209766",
"Item": [
{
"Id": 47744,
"ItemName": "Item_1",
"Valid": false
},
{
"Id": 47872,
"ItemName": "Item_2",
"Valid": true
},
{
"Id": 47872,
"ItemName": "Item_3",
"Valid": false
}
]
}
]
"Browse"
"8fj9438jgge9hdfv0jj0en34ijnd9nnf"
"v9er84n9ogjuwheofn9gerinneorheoj"
Except for the initial and trailing JSON scalars, you'd simply write:
.[] | .Item[] | select( .ItemName == "Item_2" ) | .Valid
In your particular case, to ensure the top-level JSON scalars are ignored, you could prefix the above with:
arrays |

Creating a CSV from json using jq, based on elements in array

I have the following json format that I need to convert to CSV
[{
"name": "joe",
"age": 21,
"skills": [{
"lang": "spanish",
"grade": "47",
"school": {
"name": "my school",
"url": "example.com/sp-school"
}
}, {
"lang": "english",
"grade": "87"
}]
},
{
"name": "sarah",
"age": 34,
"skills": [{
"lang": "french",
"grade": "47",
"school": {
"name": "my school",
"url": "example.com/sp-school"
}
}, {
"lang": "english",
"grade": "87"
}]
}, {
"name": "jim",
"age": 26,
"skills": [{
"lang": "spanish",
"grade": "60"
}, {
"lang": "english",
"grade": "66",
"school": {
"name": "eg school",
"url": "eg-school.com"
}
}]
}
]
to convert to csv
name,age,grade,school,url,file,line_number
joe,21,47,"my school","example.com/sp-school",sample.json,1
jim,26,60,"","",sample.json,3
So add the top level fields and the object from the skills array if lang=spanish and the school hash from the skills object for spanish if it exists
I'd also like to add the file and line number it came from.
I would like to use jq for the job, but can't figure out the syntax , anyone help me out ?
With your data in input.json, and the following jq program in tocsv.jq:
.[]
| [.name, .age] +
(.skills[]
| select(.lang == "spanish")
| [.grade, .school.name, .school.url, input_filename, input_line_number] )
| #csv
the invocation:
jq -r -f tocsv.jq input.json
yields:
"joe",21,"47","my school","example.com/sp-school","input.json",51
"jim",26,"60",,,"input.json",51
If you want the number-valued strings converted to numbers, you could use the "tonumber" filter. If you want the null-valued fields replaced by strings, use e.g. .school.name // ""
Of course this approach doesn't yield a very useful line number. One approach that would yield higher granularity would be to stream the individual objects into jq, but then you'd lose the filename. To recover the filename you could pass it in as an argument. So you would have a pipeline like so:
jq -c '.[]' input.json | jq -r --arg file input.json -f tocsv2.jq
where tocsv2.jq would be like tscsv.jq above but without the initial .[] |, and with $file instead of input_filename.
Finally, please also consider using the TSV format (#tsv) rather than the rather messy CSV format (#csv).

Bash JQ getting multiple values Issue in JSON file

I'm trying to parse a JSON file for getting multiple values. I know how to parse the specific values ( "A"/"B"/"C") in the array (.info.file.hashes[]).
For Example : When issuing the following command over the file b.json
jq -r '.info.file.hashes[] | select(.name == ("A","B","C")).value' b.json
Result :
f34d5f2d4577ed6d9ceec516c1f5a744
66031dad95dfe6ad10b35f06c4342faa
9df25fa4e379837e42aaf6d05d92012018d4b659
Where b.json:
{
"Finish": 1475668827,
"Start": 1475668826,
"info": {
"file": {
"Score": 4,
"file_subtype": "None",
"file_type": "Image",
"hashes": [
{
"name": "A",
"value": "f34d5f2d4577ed6d9ceec516c1f5a744"
},
{
"name": "B",
"value": "66031dad95dfe6ad10b35f06c4342faa"
},
{
"name": "C",
"value": "9df25fa4e379837e42aaf6d05d92012018d4b659"
},
{
"name": "D",
"value": "4a51cc531082d216a3cf292f4c39869b462bf6aa"
},
{
"name": "E",
"value": "e445f412f92b25f3343d5f7adc3c94bdc950601521d5b91e7ce77c21a18259c9"
}
],
"size": 500
}
}
}
Now, how can i get multiple values with "Finish", "Start" along with the hash values? I have tried issuing the command.
jq -r '.info.file.hashes[] | select(.name == ("A","B","C")).value','.Finish','.Start' b.json
and Im getting the result as:
f34d5f2d4577ed6d9ceec516c1f5a744
null
66031dad95dfe6ad10b35f06c4342faa
null
9df25fa4e379837e42aaf6d05d92012018d4b659
null
null
null
Expected Result :
f34d5f2d4577ed6d9ceec516c1f5a744
66031dad95dfe6ad10b35f06c4342faa
9df25fa4e379837e42aaf6d05d92012018d4b659
1475668827
1475668826
Literally just downloaded and read the manual
Try
jq '(.info.file.hashes[] |select(.name == ("A","B","C")).value), .Finish, .Start' b.json
"f34d5f2d4577ed6d9ceec516c1f5a744"
"66031dad95dfe6ad10b35f06c4342faa"
"9df25fa4e379837e42aaf6d05d92012018d4b659"
1475668827
1475668826
Note the brackets used for grouping the pipe separately from the Finish and Start values.