So I am trying to pull out some JSON with jq.
Here is an example:
{
"limit":10,
"offset":20,
"values": [
{
"id": "abcd"
"type": "file"
"users": {
"total": 2,
"profiles": [
{
"first_name": "John",
"last_name": "Smith"
},
{
"first_name": "Sue",
"last_name": "Johnson"
}
]
}
},
{
"id": "efgh"
"type": "folder"
"users": {
"total": 1,
"profiles": [
{
"first_name": "Steve",
"last_name": "Gold"
}
]
}
},
]
}
I would love to get the following in my result
limit:10
offset:20
id:abcd, type:file, [users.total:2, users.profiles.first_name: [John, Sue], users.profiles.last_name: [Smith, Johnson]],
id:efgh, type:folder, [users.total:1, users.profiles.first_name: [Steve], users.profiles.last_name: [Gold]],
I know I can pipe this to jq, but I don't know how. I can get stuff in an array easily, but I can't seem to figure out how to do it and add in the top level elements. So it's 'limit' and 'offset' that are throwing me fits.
I can get something out
jq ".values[] | .users.total, .users[].first_name, .users[].last_name"
But I cannot seem to figure out how to add in limit and offset.
You can use commas at the top-level to get those fields. Use parentheses to group the expression that gets everything from .values.
jq '.limit, .offset, (.values[] | .id, .type, .users.total, .users.profiles[].first_name, .users.profiles[].last_name)'
DEMO
So it's 'limit' and 'offset' that are throwing me fits.
It looks like to_entries might be what you're looking for. Consider, for example:
to_entries[]
| select(.value | type != "array")
| [.key, .value]
| join(": ")
With your JSON, this produces the lines you are having difficulty with, without any explicit mention of the key names:
limit: 10
offset: 20
You could then (for example) use the comma operator (,) to produce the other lines you want.
Related
I have a json that looks like this:
{
"HostedZones": [
{
"ResourceRecordSetCount": 2,
"CallerReference": "test20150527-2",
"Config": {
"Comment": "test2",
"PrivateZone": true
},
"Id": "/hostedzone/Z119WBBTVP5WFX",
"Name": "dev.devx.company.services."
},
{
"ResourceRecordSetCount": 2,
"CallerReference": "test20150527-1",
"Config": {
"Comment": "test",
"PrivateZone": true
},
"Id": "/hostedzone/Z3P5QSUBK4POTI",
"Name": "test.devx.company.services."
}
],
"IsTruncated": false,
"MaxItems": "100"
}
And my goal is to fetch a specific Name (in my case it's the test.devx.company.services), however the Name field contains an extra "." at the end that I'd like to remove from the output.
This is what I have so far:
jq --raw-output '.HostedZones[] | select(.Name | test("test")?) | (.Name[:-1] | sub("."; ""))'
The problem with that it is removing the first character from the output also.
So the output currently is: est.devx.company.services (JQ play snippet)
Not sure what I'm doing wrong :/
To always remove the last character, if it contains "test":
jq '(.HostedZones[].Name | select(contains("test"))) |= .[:-1]'
To remove it only if it is a dot:
jq '(.HostedZones[].Name | select(contains("test"))) |= sub("[.]$"; "")'
I have this JSON document:
{
"1": {
"a": "G1"
},
"2": {
"a": "GM1"
}
}
My expected result should be:
1,G1
2,GM1
With *.a i get
[
"G1",
"GM1"
]
but I am absolutely stuck for the rest.
Sadly there is not much you can do that would be totally matching your use case and that would scale properly.
This is because JMESPath does not have a way to reference its parent, although this has been requested before, to allow you something like
*.[join(',', [keys($), a])]
You can definitely extract a list of keys and values, thanks to the function keys:
#.{keys: keys(#), values: *.a}
That gives
{
"keys": [
"1",
"2"
],
"values": [
"G1",
"GM1"
]
}
But then you just fall under the same case as this other question, because keys will give you a list of keys.
You can also end with a list of lists:
#.[keys(#), *.a]
Will give you:
[
[
"1",
"2"
],
[
"G1",
"GM1"
]
]
And you can even go further and flatten it if needed:
#.[keys(#), *.a] []
Gives:
[
"1",
"2",
"G1",
"GM1"
]
With all this if you do happen to have a list of exactly two items, then a solution would be to use a combination of join and slice:
#.[join(',',[keys(#),*.a][] | [::2]), join(',',[keys(#),*.a][] | [1::2])]
That would give the expected:
[
"1,G1",
"2,GM1"
]
But, sadly, as soon as you have more than two items to consider you would end up with a buggy:
[
"1,3,G1,GM3",
"2,4,GM1,GM4"
]
With a data set of
{
"1": {
"a": "G1"
},
"2": {
"a": "GM1"
},
"3": {
"a": "GM3"
},
"4": {
"a": "GM4"
}
}
And then, of course, the same can be achieved hardcoding indexes:
#.[join(',', [keys(#)[0], *.a | [0]]), join(',', [keys(#)[1], *.a | [1]])]
That also gives the expected:
[
"1,G1",
"2,GM1"
]
But, sadly, this only works if you know in advance the number of rows that are going to be returned to you.
And if you want a single string, given that were you want to feed the data accepts \n as a new line, you can join he whole array again:
#.[join(',', [keys(#)[0], *.a | [0]]), join(',', [keys(#)[1], *.a | [1]])].join(`\n`,#)
Will give:
"1,G1\n2,GM1"
Finally this expression worked 100% for me:
[{key1:keys(#)[0],a:*.a| [0]},{key1:keys(#)[1],a:*.a| [1]}]
Two of my EC2 instances have 3 IPs each. I managed to successfully grab a list of JSON objects:
aws ec2 describe-instances | jq '.Reservations[] | .Instances[] | (.Tags | { "iname": ( map ( select(.Value | contains("my-vm")))[] | .Value ) } ) + ( { "ip": ( .NetworkInterfaces[].PrivateIpAddress) } )' | jq -s .
Giving me the following result:
[
{
"iname": "my-vm-b",
"ip": "10.11.2.145"
},
{
"iname": "my-vm-b",
"ip": "10.11.1.146"
},
{
"iname": "my-vm-b",
"ip": "10.11.10.144"
},
{
"iname": "my-vm-a",
"ip": "10.11.1.9"
},
{
"iname": "my-vm-a",
"ip": "10.11.10.125"
},
{
"iname": "my-vm-a",
"ip": "10.11.2.85"
}
]
and then I added to the command the following:
... | jq ' group_by(.iname)[] | {(.[0].iname): [.[] | .ip]}' | jq -s .
To finally get the list of objects the way I wanted:
[
{
"my-vm-a": [
"10.11.1.9",
"10.11.10.125",
"10.11.2.85"
]
},
{
"my-vm-b": [
"10.11.2.145",
"10.11.1.146",
"10.11.10.144"
]
}
]
Notice I had to call jq like 4 times. I know I must be doing something wrong so I was wondering if I could do it with a single jq call.
Thanks!
You can easily eliminate the calls to jq -s by wrapping expressions as appropriately in square brackets, or maybe better, using map.
For example, your last pair of jq calls can be simplified to:
jq 'group_by(.iname) | map({(.[0].iname): [.[] | .ip]})'
The following should allow you to reduce the four calls to one:
[.Reservations[]
| .Instances[]
| (.Tags | { "iname": ( map ( select(.Value | contains("my-vm")))[] | .Value ) } )
+ ( { "ip": ( .NetworkInterfaces[].PrivateIpAddress) } ) ]
| group_by(.iname) | map({(.[0].iname): [.[] | .ip]})
However, I would advise against using contains here, unless you fully understand the complications.
Before you go into trying to simplify your jq calls, I think it would be more beneficial to first look at the source data and how it relates to the result you want.
Ignoring a lot of the other details in the data, I think we can agree that your data looks sorta like this:
{
"Reservations": [
{
"Instances": [
{
"NetworkInterfaces": [
{ "PrivateIpAddress": "10.11.2.145" },
{ "PrivateIpAddress": "10.11.1.146" },
{ "PrivateIpAddress": "10.11.10.144" }
],
"Tags": [
{ "Key": "Name", "Value": "my-vm-b" }
]
},
{
"NetworkInterfaces": [
{ "PrivateIpAddress": "10.11.1.9" },
{ "PrivateIpAddress": "10.11.10.125" },
{ "PrivateIpAddress": "10.11.2.85" }
],
"Tags": [
{ "Key": "Name", "Value": "my-vm-a" }
]
}
]
}
]
}
With something that looks like this, your jq query can simply be:
[.Reservations[].Instances[] |
{
(.Tags|from_entries.Name): [.NetworkInterfaces[].PrivateIpAddress]
}
]
No intermediate results needed. Just a few things of note here.
The tags are already an array of key/value pairs, you can easily read values from here converting them to an object first using from_entries
You are selecting instances based on an existence of a tag value containing "my-vm". I'm not sure you even need to do this, I don't know what your data looks like but they are likely in a fixed name so you should just use that name.
I'm trying to set up an alert for when the following JSON object state says anything but started. I'm beginning to play around with conditional jq but I'm unsure how to implement regex into this.
{
"page": 0,
"page_size": 100,
"total_pages": 10,
"total_rows": 929,
"headers": [
"*"
],
"rows": [
{
"id": "168",
"state": "STARTED"
},
{
"id": "169",
"state": "FAILED"
},
{
"id": "170",
"state": "STARTED"
}
]
}
I only want to display the id and state of the failed object, this is what I tried
jq '.rows[] | .id, select(.state | contains("!STARTED"))' test.json
I'd like my output to be something like
{
"id": "169",
"state": "FAILED"
}
If you simply want to print out the objects for which .state is NOT "STARTED", just use negation:
.rows[] | select(.state != "STARTED")
If the "started" state is associated with multiple values, please give further details. There might not be any need to use regular expressions. If you really do need to use regular expressions, then you will probably want to use test.
I have some JSON output I am trying to parse with jq. I read some examples on filtering but I don't really understand it and my output it more complicated than the examples. I have no idea where to even begin beyond jq '.[]' as I don't understand the syntax of jq beyond that and the hierarchy and terminology are challenging as well. My JSON output is below. I want to return the value for Valid where the ItemName equals Item_2. How can I do this?
"1"
[
{
"GroupId": "1569",
"Title": "My_title",
"Logo": "logo.jpg",
"Tags": [
"tag1",
"tag2",
"tag3"
],
"Owner": [
{
"Name": "John Doe",
"Id": "53335"
}
],
"ItemId": "209766",
"Item": [
{
"Id": 47744,
"ItemName": "Item_1",
"Valid": false
},
{
"Id": 47872,
"ItemName": "Item_2",
"Valid": true
},
{
"Id": 47872,
"ItemName": "Item_3",
"Valid": false
}
]
}
]
"Browse"
"8fj9438jgge9hdfv0jj0en34ijnd9nnf"
"v9er84n9ogjuwheofn9gerinneorheoj"
Except for the initial and trailing JSON scalars, you'd simply write:
.[] | .Item[] | select( .ItemName == "Item_2" ) | .Valid
In your particular case, to ensure the top-level JSON scalars are ignored, you could prefix the above with:
arrays |