I have the following in a file (which I will call "myfile"):
[{
"id": 123,
"name": "John",
"aux": [{
"abc": "random",
"def": "I want this"
}],
"blah": 23.11
}]
I could parse it if the file did not have the first [ and last ] as follows:
$ cat myfile | jq -r '.aux[] | .def'
I want this
$
but with the [ and ] I get:
$ cat myfile | jq -r '.aux[] | .def'
jq: error: Cannot index array with string
How can I deal with the [ and ] using jq? (I'm sure I could parse them off with a different tool but I want to learn correct usage of jq.
It should be:
jq '.[].aux[].def' file.json
.[] iterates over the outer array, .aux[] then iterates over the the aux array of every node and .def prints their .def property.
This will output:
"I want this"
If you want to get rid of the double quotes pass -r (--raw) to jq:
jq -r '.[].aux[].def' file.json
Output:
I want this
Related
curl http://testhost.test.com:8080/application/app/version | jq '.version' | jq '.[]'
The above command outputs only the values as below:
"madireddy#test.com"
"2323"
"test"
"02-03-2014-13:41"
"application"
How can I get the key names instead like the below:
email
versionID
context
date
versionName
To get the keys in the order they appear in the original JSON use:
jq 'keys_unsorted' file.json
If you want the keys sorted alphanumerically, you can use:
jq 'keys' file.json
Complete example
$ cat file.json
{ "Created-By" : "Apache Maven", "Build-Number" : "", "Archiver-Version" : "Plexus Archiver", "Build-Id" : "", "Build-Tag" : "", "Built-By" : "cporter"}
$ jq 'keys_unsorted' file.json
[
"Created-By",
"Build-Number",
"Archiver-Version",
"Build-Id",
"Build-Tag",
"Built-By"
]
$ jq 'keys' file.json
[
"Archiver-Version",
"Build-Id",
"Build-Number",
"Build-Tag",
"Built-By",
"Created-By"
]
To get the keys on a deeper node in a JSON:
echo '{"data": "1", "user": { "name": 2, "phone": 3 } }' | jq '.user | keys[]'
"name"
"phone"
You need to use jq 'keys[]'. For example:
echo '{"example1" : 1, "example2" : 2, "example3" : 3}' | jq 'keys[]'
Will output a line separated list:
"example1"
"example2"
"example3"
In combination with the above answer, you want to ask jq for raw output, so your last filter should be eg.:
cat input.json | jq -r 'keys'
From jq help:
-r output raw strings, not JSON texts;
To print keys on one line as csv:
echo '{"b":"2","a":"1"}' | jq -r 'keys | [ .[] | tostring ] | #csv'
Output:
"a","b"
For csv completeness ... to print values on one line as csv:
echo '{"b":"2","a":"1"}' | jq -rS . | jq -r '. | [ .[] | tostring ] | #csv'
Output:
"1","2"
If your input is an array of objects,
[
{
"a01" : { "name" : "A", "user" : "B" }
},
{
"a02" : { "name" : "C", "user" : "D" }
}
]
try with:
jq '.[] | keys[]'
Oddly enough, the accepted answer doesn’t actually answer the Q exactly, so for reference, here is a solution that does:
$ jq -r 'keys_unsorted[]' file.json
echo '{"ab": 1, "cd": 2}' | jq -r 'keys[]' prints all keys one key per line without quotes.
ab
cd
Here's another way of getting a Bash array with the example JSON given by #anubhava in his answer:
arr=($(jq --raw-output 'keys_unsorted | #sh' file.json))
echo ${arr[0]} # 'Archiver-Version'
echo ${arr[1]} # 'Build-Id'
echo ${arr[2]} # 'Build-Jdk'
I need a jq command to replace array values with non-json standard values i.e. in the following json input I need to replace the "webOrigins" array values with a non-json value, which is a Jinja2 template variable replacement as per the second json block below.
Input (example.json)
{
"clients": [
{
"clientId": "abc",
"webOrigins": [ "/", "/api" ]
},
{
"clientId": "xyz",
"webOrigins": [ ]
}
]
}
Desired Output
{
"clients": [
{
"clientId": "abc",
"webOrigins": {{clients.abc.webOrigins}}
},
{
"clientId": "xyz",
"webOrigins": {{clients.xyz.webOrigins}}
}
]
}
My current attempt of shell script calling jq to loop through the input json and replace with template variable is this
for clientId in $(jq -r '.clients[] | .clientId' example.json)
do
jq '(.clients[] | select(.clientId == "'${clientId}'") | .webOrigins) |= {{ clients['\'${clientId}\''].webOrigins | default([]) }}' example.json > tmp.j2; mv -f tmp.j2 example.json
done
But fails with the error:
jq: error: syntax error, unexpected '{' (Unix shell quoting issues?) at <top-level>, line 1:
(.clients[] | select(.clientId == "abc") | .webOrigins) |= {{ clients['abc'].webOrigins | default([]) }}
jq: 1 compile error
Of course if I add double quotes around the template variable to make the replacement variable valid json the script works but I need the value to not have quotes.
It would be impractical to implement a jq-only solution, but jq can certainly help, e.g. the following works in your case:
jq '.clients[] |= (.webOrigins = "{{clients.\(.clientId).webOrigins}}")' |
sed '/"webOrigins":/ { s/"[{][{]/{{/; s/[}][}]"$/}}/; }'
With the following input file:
{
"events": [
{
"mydata": {
"id": "123456",
"account": "21234"
}
},
{
"mydata": {
"id": "123457",
"account": "21234"
}
}
]
}
When I run it through this JQ filter,
jq ".events[] | [.mydata.id, .mydata.account]" events.json
I get a set of arrays:
[
"123456",
"21234"
]
[
"123457",
"21234"
]
When I put this output through the #csv filter to create CSV output:
jq ".events[] | [.mydata.id, .mydata.account] | #csv" events.json
I get a CSV file with one record per row:
"\"123456\",\"21234\""
"\"123457\",\"21234\""
I would like CSV file with two records per row, like this:
"123456","21234"
"123457","21234"
What am I doing wrong?
Use the -r flag.
Here is the explanation in the manual:
--raw-output / -r: With this option, if the filter's result is a string then it will be written directly to standard output rather than
being formatted as a JSON string with quotes.
jq -r '.events[] | [.mydata.id, .mydata.account] | #csv'
Yields
"123456","21234"
"123457","21234"
Haven't used jq before but I'm wanting to build a shell script that will get a JSON response and extract just the values. To learn I thought I would try on my blog's WP API but for some reason I'm getting an error of:
jq: error (at :322): Cannot index array with string "slug"
When researching for and testing previous questions:
jq: Cannot index array with string
jq is sed for JSON
JSON array to bash variables using jq
How to use jq in a shell pipeline?
How to extract data from a JSON file
The above reading I've tried to code:
URL="http://foobar.com"
RESPONSE=$(curl -so /dev/null -w "%{http_code}" $URL)
WPAPI="/wp-json/wp/v2"
IDENTIFIER="categories"
if (("$RESPONSE" == 200)); then
curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | jq '.' >> $IDENTIFIER.json
result=$(jq .slug $IDENTIFIER.json)
echo $result
else
echo "Not returned status 200";
fi
An additional attempt changing the jq after the curl:
curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | jq '.' | $IDENTIFIER.json
result=(jq -r '.slug' $IDENTIFIER.json)
echo $result
I can modify the uncompress with the python JSON tool:
result=(curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | python -m json.tool > $IDENTIFIER.json)
I can save the JSON to a file but when I use jq I cannot get just the slug and here are my other trys:
catCalled=$(curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | python -m json.tool | ./jq -r '.slug')
echo $catCalled
My end goal is to try to use jq in a shell script and build a slug array with jq. What am I doing wrong in my jq and can I use jq on a string without creating a file?
Return from curl after uncompress per comment request:
[
{
"id": 4,
"count": 18,
"description": "",
"link": "http://foobar.com/category/foo/",
"name": "Foo",
"slug": "foo",
"taxonomy": "category",
},
{
"id": 8,
"count": 9,
"description": "",
"link": "http://foobar.com/category/bar/",
"name": "Bar",
"slug": "bar",
"taxonomy": "category",
},
{
"id": 5,
"count": 1,
"description": "",
"link": "http://foobar.com/category/mon/",
"name": "Mon",
"slug": "mon",
"taxonomy": "category",
},
{
"id": 11,
"count": 8,
"description": "",
"link": "http://foobar.com/category/fort/",
"name": "Fort",
"slug": "fort",
"taxonomy": "category",
}
]
eventually my goal is trying to get the name of the slug's into an array like:
catArray=('foo','bar','mon', 'fort')
There are 2 issues here:
slug is not a root level element in your example json. The root level element is an array. If you want to access the slug property of each element of the array, you can do so like this:
jq '.[].slug' $IDENTIFIER.json
Your example json has trailing commas after the last property of each array element. Remove the commas after "taxonomy": "category".
If I take your sample json, remove the errant commas, save it to a plain text file called test.json and run the following command:
jq '.[].slug' test.json
I get the following output:
"foo"
"bar"
"mon"
"fort"
Preprocessing
Unfortunately, the JSON-like data shown as having been produced by curl is not strictly JSON. jq does not have a "relaxed JSON" mode, so in order to use jq, you will have to preprocess the JSON-like data, e.g. using hjson (see http://hjson.org/):
$ hjson -j input.qjson > input.json
jq
With the JSON in input.json:
$ jq -c 'map(.slug)' input.json
["foo","bar","mon","fort"]
your string is not json, notice how the last member of your objects ends with a comma,
{foo:"bar",baz:9,}
this is legal in javascript, but it's illegal in json. if you are supposed to be receiving json from that endpoint, then contact the people behind it and tell them to fix the bug (it's breaking the json specs by ending objects's last member with a comma, which is illegal in json.) - until it's fixed, i guess you can patch it with a little regex, but it's a dirty quickfix, and probably not very reliable, but running it through
perl -p -0777 -e 's/\"\,\s*}/\"}/g;' makes it legal json..
curl http://testhost.test.com:8080/application/app/version | jq '.version' | jq '.[]'
The above command outputs only the values as below:
"madireddy#test.com"
"2323"
"test"
"02-03-2014-13:41"
"application"
How can I get the key names instead like the below:
email
versionID
context
date
versionName
To get the keys in the order they appear in the original JSON use:
jq 'keys_unsorted' file.json
If you want the keys sorted alphanumerically, you can use:
jq 'keys' file.json
Complete example
$ cat file.json
{ "Created-By" : "Apache Maven", "Build-Number" : "", "Archiver-Version" : "Plexus Archiver", "Build-Id" : "", "Build-Tag" : "", "Built-By" : "cporter"}
$ jq 'keys_unsorted' file.json
[
"Created-By",
"Build-Number",
"Archiver-Version",
"Build-Id",
"Build-Tag",
"Built-By"
]
$ jq 'keys' file.json
[
"Archiver-Version",
"Build-Id",
"Build-Number",
"Build-Tag",
"Built-By",
"Created-By"
]
To get the keys on a deeper node in a JSON:
echo '{"data": "1", "user": { "name": 2, "phone": 3 } }' | jq '.user | keys[]'
"name"
"phone"
You need to use jq 'keys[]'. For example:
echo '{"example1" : 1, "example2" : 2, "example3" : 3}' | jq 'keys[]'
Will output a line separated list:
"example1"
"example2"
"example3"
In combination with the above answer, you want to ask jq for raw output, so your last filter should be eg.:
cat input.json | jq -r 'keys'
From jq help:
-r output raw strings, not JSON texts;
To print keys on one line as csv:
echo '{"b":"2","a":"1"}' | jq -r 'keys | [ .[] | tostring ] | #csv'
Output:
"a","b"
For csv completeness ... to print values on one line as csv:
echo '{"b":"2","a":"1"}' | jq -rS . | jq -r '. | [ .[] | tostring ] | #csv'
Output:
"1","2"
If your input is an array of objects,
[
{
"a01" : { "name" : "A", "user" : "B" }
},
{
"a02" : { "name" : "C", "user" : "D" }
}
]
try with:
jq '.[] | keys[]'
Oddly enough, the accepted answer doesn’t actually answer the Q exactly, so for reference, here is a solution that does:
$ jq -r 'keys_unsorted[]' file.json
echo '{"ab": 1, "cd": 2}' | jq -r 'keys[]' prints all keys one key per line without quotes.
ab
cd
Here's another way of getting a Bash array with the example JSON given by #anubhava in his answer:
arr=($(jq --raw-output 'keys_unsorted | #sh' file.json))
echo ${arr[0]} # 'Archiver-Version'
echo ${arr[1]} # 'Build-Id'
echo ${arr[2]} # 'Build-Jdk'