I've been using jq to parse the output from AWS cli.
The output looks something like this..
{
"Vpcs": [
{
"CidrBlock": "10.29.19.64/26",
"State": "available",
"VpcId": "vpc-0ba51bd29c41d41",
"IsDefault": false,
"Tags": [
{
"Key": "Name",
"Value": "CloudEndure-Europe-Development"
}
]
}
]}
and the script I am using looks like this..
.Vpcs[] | [.VpcId, .CidrBlock, (.Tags[]|select(.Key=="Name")|.Value)]
If I run it under Windows it fails like this.
jq: error: Name/0 is not defined at , line 1:
.Vpcs[] | [.VpcId, .CidrBlock, (.Tags[]|select(.Key==Name)|.Value)]
jq: 1 compile error
But it works fine in jqplay.org.
Any ideas, on Windows Im using jq-1.6.
Thanks
Bruce.
The correct jq program is
.Vpcs[] | [.VpcId, .CidrBlock, ( .Tags[] | select( .Key == "Name" ) | .Value ) ]
You didn't show the command you used, but you provided the following to jq:
.Vpcs[] | [.VpcId, .CidrBlock, ( .Tags[] | select( .Key == Name ) | .Value ) ]
That's incorrect. (Notice the missing quotes.)
Not only did you not provide what command you used, you didn't specify whether it was being provided to the Windows API (CreateProcess), Windows Shell (cmd) or Power Shell.
I'm guessing cmd. In order to provide the above program to jq, you can use the following cmd command:
jq ".Vpcs[] | [.VpcId, .CidrBlock, ( .Tags[] | select( .Key == \"Name\" ) | .Value ) ]" file.json
I'm not agreeing to ikegami about the CMD command that [he/she?] provided because the character used for CMD escaping is ^, not \ like Assembly/C/C++. I hope this will work (I don't want to test this on my potato thing):
jq .Vpcs[] | [.VpcId, .CidrBlock, ( .Tags[] | select( .Key == "Name" ) | .Value ) ] file.json
or this:
jq .Vpcs[] | [.VpcId, .CidrBlock, ( .Tags[] | select( .Key == ^"Name^" ) | .Value ) ] file.json
Related
I got a Json which is basically a array but with a weird format that i can not change.
Is there any way that i can get with JQ the url by searching for the name, like this?
{
"servers": {
"servers[0].name": "abc",
"servers[0].url": "www.abc.test.com",
"servers[1].name": "xyz",
"servers[1].url": "www.xyz.test.com"
}
}
jq -r '.servers | select(.name=="abc") | .url'
Assuming the "=" can be naively changed to ":":
sed 's/ = /: /' | jq '
.servers
| keys_unsorted[] as $k
| select(.[$k] == "abc")
| ($k | sub("[.]name"; ".url")) as $k
| .[$k]
'
If you are looking for a general way to build a JSON array or object from such source, here's one way using reduce and setpath with regexes for splitting up the keys:
def build:
reduce (to_entries[] | .key |= [
splits("(?=\\[\\d+\\])|\\.")
| capture("\\[(?<index>\\d+)\\]|(?<field>.+)")
| (.index | tonumber)? // .field
]) as {$key, $value} (null; setpath($key; $value));
.servers | build.servers[] | select(.name == "abc").url
Demo
From this "test.json" file :
{
"key1": "abc",
"key2": "def",
"key3": "ghi"
}
I can update "key2" value with the following command :
jq '.key2="123"' test.json
But, is there a way to use the key filter with case unsensitive and make this command works :
jq '.KeY2="123"' test.json
And also... to do nothing if the key was not found in the JSON file. The default behavior is to append the searched key.
test(regex; "i") can be used to perform a case-insensitive regular expression match in jq. Thus:
jq --arg key KeY2 --arg newValue 123 '
[to_entries[] |
if .key | test($key; "i") then
.value = $newValue
else . end
] | from_entries' <test.json
That said, being as this is a regex match, you might want to think about how your key name behaves as a regex -- anchoring, etc. An alternative is to convert both versions to lowercase for the comparison only:
jq --arg key KeY2 --arg newValue 123 '
($key | ascii_downcase) as $lower_key |
[to_entries[] |
if (.key | ascii_downcase) == $lower_key then
.value = $newValue
else . end
] | from_entries' <test.json
You can use ascii_downcase to get the lowercase version of the key:
jq --arg k Key1 '(.[$k | ascii_downcase] // empty) = "123"'
The // empty part prevents the creation of the key.
Input json:
{
"food_group": "fruit",
"glycemic_index": "low",
"fruits": {
"fruit_name": "apple",
"size": "large",
"color": "red"
}
}
Below two jq commands work:
# jq -r 'keys_unsorted[] as $key | "\($key), \(.[$key])"' food.json
food_group, fruit
glycemic_index, low
fruits, {"fruit_name":"apple","size":"large","color":"red"}
# jq -r 'keys_unsorted[0:2] as $key | "\($key)"' food.json
["food_group","glycemic_index"]
How to get values for the first two keys using jq in the same manner? I tried below
# jq -r 'keys_unsorted[0:2] as $key | "\($key), \(.[$key])"' food.json
jq: error (at food.json:9): Cannot index object with array
Expected output:
food_group, fruit
glycemic_index, low
To iterate over a hash array , you can use to_entries and that will transform to a array .
After you can use select to filter rows you want to keep .
jq -r 'to_entries[]| select( ( .value | type ) == "string" ) | "\(.key), \(.value)" '
You can use to_entries
to_entries[] | select(.key=="food_group" or .key=="glycemic_index") | "\(.key), \(.value)"
Demo
https://jqplay.org/s/Aqvos4w7bo
I have the following code, which lists all the current aws lambda functions on my account:
aws lambda list-functions --region eu-west-1 | jq -r '.Functions | .[] | .FunctionName' | xargs -L1 -I {} aws logs describe-log-streams --log-group-name /aws/lambda/{} | jq 'select(.logStreams[-1] != null)' | jq -r '.logStreams | .[] | [.arn, .lastEventTimestamp] | #csv'
that returns
aws:logs:eu-west-1:****:log-group:/aws/lambda/admin-devices-block-master:log-stream:2018/01/23/[$LATEST]64965367852942f490305cb8707d81b4",1516717768514
i am only interested in admin-devices-block-master and i want to convert the timestamp 1516717768514 in as strflocaltime("%Y-%m-%d %I:%M%p")
so it should just return:
"admin-devices-block-master",1516717768514
i tried
aws lambda list-functions --region eu-west-1 | jq -r '.Functions | .[] | .FunctionName' | xargs -L1 -I {} aws logs describe-log-streams --log-group-name /aws/lambda/{} | jq 'select(.logStreams[-1] != null)' | jq -r '.logStreams | .[] | [.arn,[.lastEventTimestamp|./1000|strflocaltime("%Y-%m-%d %I:%M%p")]]'
jq: error: strflocaltime/1 is not defined at <top-level>, line 1:
.logStreams | .[] | [.arn,[.lastEventTimestamp|./1000|strflocaltime("%Y-%m-%d %I:%M%p")]]
jq: 1 compile error
^CException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
any advice is much appreciated
strflocaltime need jq version 1.6 , thanks to #oliv to remark it .
This is a very simple example that will replace a EPOCH with milliseconds by a local time .
date -d #1572892409
Mon Nov 4 13:33:29 EST 2019
echo '{ "ts" : 1572892409356 , "id": 2 , "v": "foobar" } ' | \
jq '.ts|=( ./1000|strflocaltime("%Y-%m-%d %I:%M%p")) '
{
"ts": "2019-11-04 01:33PM",
"id": 2,
"v": "foobar"
}
A second version that test if ts exists
(
echo '{ "ts" : 1572892409356 , "id": 2 , "v": "foobar" } ' ;
echo '{ "id":3 }' ;
echo '{ "id": 4 , "v": "barfoo" }'
) | jq 'if .ts != null
then ( .ts|=( ./1000|strflocaltime("%Y-%m-%d %I:%M%p")) )
else .
end '
I've following json,
{
"A": {
"C": {
"D": "T1",
"E": 1
},
"F": {
"D": "T2",
"E": 2
}
},
"B": {
"C": {
"D": "T3",
"E": 3
}
}
}
I want to convert it into csv as follows,
A,C,T1,1
A,F,T2,2
B,C,T3,3
Description of output: The parents keys will be printed until, I've reached the leaf child. Once I reached leaf child, print its value.
I've tried following and couldn't succeed,
cat my.json | jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $rows[] | #csv'
and it throwing me an error.
I can't hardcode the parent keys, as the actual json has too many records. But the structure of the json is similar. What am I missing?
Some of the requirements are unclear, but the following solves one interpretation of the problem:
paths as $path
| {path: $path, value: getpath($path)}
| select(.value|type == "object" )
| select( [.value[]][0] | type != "object")
| .path + ([.value[]])
| #csv
(This program could be optimized but the presentation here is intended to make the separate steps clear.)
Invocation:
jq -r -f leaves-to-csv.jq input.json
Output:
"A","C","T1",1
"A","F","T2",2
"B","C","T3",3
Unquoted strings
To avoid the quotation marks around strings, you could replace the last component of the pipeline above with:
join(",")
Here is a solution using tostream and group_by
[
tostream
| select(length == 2) # e.g. [["A","C","D"],"T1"]
| .[0][:-1] + [.[1]] # ["A","C","T1"]
]
| group_by(.[:-1]) # [[["A","C","T1"],["A","C",1]],...
| .[] # [["A","C","T1"],["A","C",1]]
| .[0][0:2] + map(.[-1]|tostring) # ["A","C","T1","1"]
| join(",") # "A,C,T1,1"