Filter by values using ./jq - json

Given the input of sizes:
[
{
"stock": 1,
"sales": 0,
"sizes": [
{
"countries": ["at", "be", "ch", "cy", "de", "ee", "es", "fi", "gr", "ie", "lu", "lv", "nl", "pl", "pt", "se", "si", "sk"],
"size": "EU 45,5"
},
{
"countries": ["it"],
"size": "EU 45,5"
},
{
"countries": ["fr"],
"size": "EU 45,5"
},
{
"countries": ["gb"],
"size": "EU 45,5"
}
]
}
]
I will like to get the same structure without the ones that countries hasn't "de" (Germany) and remove the field complete. Expected something like this:
[
{
"stock": 1,
"sizes": [
{
"size": "EU 45,5"
}
]
}
]
I tried this:
map(.sizes[] |= select(.countries | join(",") | contains("de"))) | map({ stock, sizes })
But the filter is not working properly, throwing jq: error (at <stdin>:48): Cannot iterate over null (null).
Tried has, in, contains, inside and nothing seems to work.
Also, how can I filter which field appears? With map({ stock, sizes }) countries still there. Can I do something like map({ stock, sizes: { size } })?

Here's a one-liner that answers your main question -- if you can't see how it works, try breaking it up into separate pieces:
map( .sizes |= map( select(.countries | index("de") ) | del(.countries) ))
Regarding the selection of fields, you can use del/1 as above, or sometimes simply using an expression such as {key1, key2} will do the trick. Consider also this function and the following example:
def query(queryobject):
with_entries( select( .key as $key | queryobject | has( $key ) ));
Example:
$ jq -c -n '{"a": 1, "b": null, "c":3} | query( {a,b,d} )'
{"a":1,"b":null}

Related

Use JQ to create new object where the key comes from one object and the value comes from another

I have the following input:
{
"Columns": [
{
"email": 123,
"name": 456,
"firstName": 789,
"lastName": 450,
"admin": 900,
"licensedSheetCreator": 617,
"groupAdmin": 354,
"resourceViewer": 804,
"id": 730,
"status": 523,
"sheetCount": 298
}
]
}
{
"Users": [
{
"email": "abc#def.com",
"name": "Abc Def",
"firstName": "Abc",
"lastName": "Def",
"admin": false,
"licensedSheetCreator": true,
"groupAdmin": false,
"resourceViewer": true,
"id": 521,
"status": "ACTIVE",
"sheetCount": 0
},
{
"email": "aaa#bbb.com",
"name": "Aaa Bob",
"firstName": "Aaa",
"lastName": "Bob",
"admin": false,
"licensedSheetCreator": true,
"groupAdmin": false,
"resourceViewer": false,
"id": 352,
"status": "ACTIVE",
"sheetCount": 0
}
]
}
I need to change the key for all key value pairs in users to match the value in Columns, like so:
{
"Columns": [
{
"email": 123,
"name": 456,
"firstName": 789,
"lastName": 450,
"admin": 900,
"licensedSheetCreator": 617,
"groupAdmin": 354,
"resourceViewer": 804,
"id": 730,
"status": 523,
"sheetCount": 298
}
]
}
{
"Users": [
{
123: "abc#def.com",
456: "Abc Def",
789: "Abc",
450: "Def",
900: false,
617: true,
354: false,
804: true,
730: 521,
523: "ACTIVE",
298: 0
},
{
123: "aaa#bbb.com",
456: "Aaa Bob",
789: "Aaa",
450: "Bob",
900: false,
617: true,
354: false,
804: false,
730: 352,
523: "ACTIVE",
298: 0
}
]
}
I don't mind if I update the Users array or create a new array of objects.
I have tried several combinations of with entries, to entries, from entries, trying to search for keys using variables but the more I dive into it, the more confused I get.
Elements of a stream are processed independently. So we have to change the input.
We could group the stream elements into an array. For an input stream, this can be achieved using --slurp/-s.[1]
jq -s '
( .[0].Columns[0] | map_values( tostring ) ) as $map |
(
.[0],
(
.[1:][] |
.Users[] |= with_entries(
.key = $map[ .key ]
)
)
)
'
Demo on jqplay
Alternatively, we could use --null-input/-n in conjunction with input and/or inputs to read the input.
jq -n '
input |
( .Columns[0] | map_values( tostring ) ) as $map |
(
.,
(
inputs |
.Users[] |= with_entries(
.key = $map[ .key ]
)
)
)
'
Demo on jqplay
Note that your desired output isn't valid JSON. Object keys must be strings. So the above produces a slightly different document than requested.
Note that I assumed that .Columns is always an array of one exactly one element. This is a nonsense assumption, but it's the only way the question makes sense.
For a stream the code generates, you could place the stream generator in an array constructor ([]). reduce can also be used to collect from a stream. For example, map( ... ) can be written as [ .[] | ... ] and as reduce .[] as $_ ( []; . + [ $_ | ... ] ).
The following has the merit of simplicity, though it does not sort the keys.
It assumes jq is invoked with the -n option and of course produces a stream of valid JSON objects:
input
| . as $Columns
| .Columns[0] as $dict
| input # Users
| .Users[] |= with_entries(.key |= ($dict[.]|tostring))
| $Columns, .
If having the keys sorted is important, then you could easily add suitable code to do that; alternatively, if you don't mind having the keys of all objects sorted, you could use the -S command-line option.

jq: count nest object values based on group by

Json:
[
{
"account": "1",
"cost": [
{
"usage":"low",
"totalcost": "2.01"
}
]
},
{
"account": "2",
"cost": [
{
"usage":"low",
"totalcost": "2.25"
}
]
},
{
"account": "1",
"cost": [
{
"usage":"low",
"totalcost": "15"
}
]
},
{
"anotheraccount": "a",
"cost": [
{
"usage":"low",
"totalcost": "2"
}
]
}
]
Results expected:
account cost
1 17.01
2 2.25
anotheraccount cost
a 2
I am able to pull out data but not sure how to aggregate it.
jq '.[] | {account,cost : .cost[].totalcost}'
Is there a way to do this in using jq, so I get all types of accounts and costs associated with them?
Two helper functions will help you get you to your destination:
def sigma( f ): reduce .[] as $o (null; . + ($o | f )) ;
def group( keyname ):
map(select(has(keyname)))
| group_by( .[keyname] )
| map({(keyname) : .[0][keyname],
cost: sigma(.cost[].totalcost | tonumber) })
;
With these, the following invocations:
group("account"),
group("anotheraccount")
yield:
[{"account":"1","cost":17.009999999999998},{"account":"2","cost":2.25}]
[{"anotheraccount":"a","cost":2}]
You should be able to manage the final formating step in jq.

Getting sum of values from object in array using jq

I'm trying to get the total amount for a specific value in a array.
The input would be something like
[{
"name": "mars",
"runner": [
{
"foo": null,
"idle": true
},
{
"foo": null,
"idle": true
},
{
"foo": null,
"idle": false
}
],
"name": "june",
"runner": [
{
"foo": null,
"idle": true
},
{
"foo": null,
"idle": true
},
{
"foo": null,
"idle": true
}
]
}]
The desired output
[ {"name" : "mars", "idle" : 2"},{"name" : "june", "idle" : 1"} ]
I tried using select and map, but I'm not understanding exactly how jq is working, for instance, I tried the following query
jq ' .[] | select(.runner[].idle == true) | {name: .name}'
The result was
{ "nome": "mars" } { "nome": "mars" } { "nome": "june" } {
"nome": "june" } { "nome": "june" }
(3x true in june and 2x in mars) I can keep parsing the json and get to the result I want, but it doesn't seems right.
Your input data doesn't seem to correspond exactly with your posting. I'll assume you meant:
[{"name":"mars",
"runner":[{"foo":null,"idle":true},{"foo":null,"idle":true},{"foo":null,"idle":false}]},
{"name":"june",
"runner":[{"foo":null,"idle":true},{"foo":null,"idle":true},{"foo":null,"idle":true}]}]
length
A reasonable approach would be to rely on the builtin length function:
map( { name, idle: (.runner | map(select(.idle)) | length)} )
count
A better (e.g. more efficient) solution would be to define a function that can count:
def count(s): reduce s as $i (0; .+1);
Here s is any filter that produces a stream of values. A solution to the problem at hand could then be written as follows:
map( {name, idle: count(.runner[] | select(.idle))} )
Output
The output in both cases:
[
{
"name": "mars",
"idle": 2
},
{
"name": "june",
"idle": 3
}
]

jq get the value of x based on y in a complex json file

jq strikes again. Trying to get the value of DATABASES_DEFAULT based on the name in a json file that has a whole lot of names and I'm completely lost.
My file looks like the following (output of an aws ecs describe-task-definition) only much more complex; I've stripped this to the most basic example I can where the structure is still intact.
{
"taskDefinition": {
"status": "bar",
"family": "bar2",
"volumes": [],
"taskDefinitionArn": "bar3",
"containerDefinitions": [
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo"
}
],
"name": "baz",
"links": []
},
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo2"
}
],
"name": "boo",
"links": []
}
],
"revision": 1
}
}
I need the value of DATABASES_DEFAULT where the name is baz. Note that there are a lot of keypairs with name, I'm specifically talking about the one outside of environment.
I've been tinkering with this but only got this far before realizing that I don't understand how to access nested values.
jq '.[] | select(.name==DATABASES_DEFAULT) | .value'
which is returning
jq: error: DATABASES_DEFAULT/0 is not defined at <top-level>, line 1:
.[] | select(.name==DATABASES_DEFAULT) | .value
jq: 1 compile error
Obviously this a) doesn't work, and b) even if it did, it's independant of the name value. My thought was to return all the db defaults and then identify the one with baz, but I don't know if that's the right approach.
I like to think of it as digging down into the structure, so first you open the outer layers:
.taskDefinition.containerDefinitions[]
Now select the one you want:
select(.name =="baz")
Open the inner structure:
.environment[]
Select the desired object:
select(.name == "DATABASES_DEFAULT")
Choose the key you want:
.value
Taken together:
parse.jq
.taskDefinition.containerDefinitions[] |
select(.name =="baz") |
.environment[] |
select(.name == "DATABASES_DEFAULT") |
.value
Run it like this:
<infile jq -f parse.jq
Output:
"foo"
The following seems to work:
.taskDefinition.containerDefinitions[] |
select(
select(
.environment[] | .name == "DATABASES_DEFAULT"
).name == "baz"
)
The output is the object with the name key mapped to "baz".
$ jq '.taskDefinition.containerDefinitions[] | select(select(.environment[]|.name == "DATABASES_DEFAULT").name=="baz")' tmp.json
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo"
}
],
"name": "baz",
"links": []
}

jq: output values of ids instead of numbers

Here's my input json:
{
"channels": [
{ "id": 1, "name": "Pop"},
{ "id": 2, "name": "Rock"}
],
"links": [
{ "id": 2, "streams": [ {"url": "http://example.com/rock"} ] },
{ "id": 1, "streams": [ {"url": "http://example.com/pop"} ] }
]
}
This is what I want as an output:
"http://example.com/pop"
"Pop"
"http://example.com/rock"
"Rock"
So I need jq to replace .channels[].id with .links[].streams[0].url based on .links[].id
I don't know if it's right, but this is how I managed to output the urls:
(.channels[].id | tostring) as $ids | [.links[]] | map({(.id | tostring): .streams[0].url}) | add as $urls | $urls[$ids]
"http://example.com/pop"
"http://example.com/rock"
The question is, how do I add .channels[].name to it?
You sometimes have to be careful what you ask for, but this will produce the result you said you want:
.channels[] as $channel
| $channel.name,
(.links[] | select(.id == $channel.id) | .streams[0].url)
Output for the given input:
"Pop"
"http://example.com/pop"
"Rock"
"http://example.com/rock"
Here is a solution which uses reduce and setpath to make a $urls lookup table from .links and then scans .channels generating corresponding urls and names.
(
reduce .links[] as $l (
{};
setpath([ $l.id|tostring ]; [$l.streams[].url])
)
) as $urls
| .channels[]
| $urls[ .id|tostring ][], .name
If multiple urls are present in the "streams" attribute this will
print them all before printing the name. e.g. if the input is
{
"channels": [
{ "id": 1, "name": "Pop"},
{ "id": 2, "name": "Rock"}
],
"links": [
{ "id": 2, "streams": [ {"url": "http://example.com/rock"},
{"url": "http://example.com/hardrock"} ] },
{ "id": 1, "streams": [ {"url": "http://example.com/pop"} ] }
]
}
the output will be
"http://example.com/pop"
"Pop"
"http://example.com/rock"
"http://example.com/hardrock"
"Rock"