Extract a particular field from json output using jq - json

I'm working a bash script to extraxt specific field from json output using jq.
USERNAME=$(echo "$OUTPUT" | jq -r '.[] | .name')
Due to jq it always fails with parse error: Invalid numeric literal at line 1, column 2 error.
My restapi result has the below output.
[
{
"url": "#/systemadm/groups/uuid-d6e4e05",
"options": {},
"group_id": 313,
"owner": "abc-123-mec",
"owner_id": "ad1337884",
"id": "c258d7b330",
"name": "abc-group"
},
{
"options": {},
"id": "global%3Regmebers",
"name": "Udata-123"
},
{
"url": "#/systemadm/groups/uuid-38943000",
"options": {},
"group_id": 910,
"owner": "framework-abcc",
"owner_id": "78d4472b738bc",
"id": "38943000057a",
"name": "def-group"
},
........................
............................
......................................
So what's wrong with this jq response of code to get "name" ?

jq can only process valid JSON.
If the value of OUTPUT is literally "id": "c258d7b330","name": "abc-group", then you could enclose it in curly braces to make it valid JSON. No guarantees though; this depends on the exact format of your input.
OUTPUT='"id": "c258d7b330",
"name": "abc-group"'
USERNAME="$(printf '%s\n' "{$OUTPUT}" | jq -r '.name')"
printf '%s\n' "$USERNAME"; // abc-group
If it cannot be converted to valid JSON, maybe a simple solution using grep+cut or awk would suffice?
OUTPUT='"id": "c258d7b330",
"name": "abc-group"'
USERNAME="$(printf '%s\n' "$OUTPUT" | grep '^"name":' | cut -d'"' -f4)"
awk:
printf '%s\n' "$OUTPUT" | awk -F'"' '/^"name":/{print $4}'
Or even use jq to parse the input as array of strings and then filter for the line in which you are interested:
jq -Rr '(select(startswith("\"name\":")) / "\"")[3]'
All options are really fragile and I recommend to fix your input to be actual, valid JSON

Related

Bash. How to access parameters while looping through json with jq?

i have a details.json file with a lot of entries and a shops.txt file like below. I like to have a little script which compares two values and just return the matching json entries.
[
{
"userName": "Anne",
"email": "anne#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "EFHJKI-KJEFT-DHMNEB",
"prod": "Car",
},
"store": {
"id": "05611a7f-a679-12ad-a3u2-0745e3650a03",
"storeName": "shop-a57ca0a3-120c-1a73-153b-fa4231cab768",
}
},
{
"userName": "Tom",
"email": "tom#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "DFSGSE-FGEAR-GWRTGW",
"prod": "Bike",
},
"store": null
},
]
This is the other file "shops.txt" (can be a lot more of shops inside)
shop-a57ca0a3-120c-1a73-153b-fa4231cab768
The script is looping through the shops, for every shop it loops through the json and should compare the currentShop with the store.shop from json and then echo the user and the shop.
But I can not access the specific parameters inside the json. How can I do this?
#!/bin/bash
shops="shops.txt"
while IFS= read -r line
do
currentShop="$line"
jq -c '.[.userName, .store.storeName]' details.json | while read i; do
if [[ $i.store.storeName == *$currentShop* ]]; then
echo $i.userName
echo $currentShop
fi
done
done < "$shops"
First of all, you might want to 'clean' your json, remove any trailing ,'s etc.
After looping through each line in the file we just need one select() to get the matching object.
The script could look something like:
#!/bin/bash
while read shop; do
echo "Check: $shop"
jq -r --arg storeName "$shop" '.[] | select(.store.storeName == "\($storeName)") | "\(.userName) - \(.store.storeName)"' details.json
done < "shops.txt"
Which will produce
Check: shop-a57ca0a3-120c-1a73-153b-fa4231cab768
Anne - shop-a57ca0a3-120c-1a73-153b-fa4231cab768
I guess this could be combined into a single jq call, but it seems like you want to loop over each entry found
You can test this jq selector on this online JqPlay Demo.
I was able to access the values with the following command:
echo $i | jq -r '.userName'

jq : print key and value for each entry in nested object

This is JSON Object
{
"success": true,
"terms": "https://coinlayer.com/terms",
"privacy": "https://coinlayer.com/privacy",
"timestamp": 1620244806,
"target": "USD",
"rates": {
"611": 0.389165,
"ABC": 59.99,
"ACP": 0.014931,
"ACT": 0.021098,
"ACT*": 0.017178,
"ADA": 1.460965
}
}
I require this type of output:
611,0.389165
ABC,59.99
ACP,0.014931
ACT,0.021098
ACT*,0.017178
ADA,1.460965
Can somebody help me figure out doing it preferably with jq, shell script or command.
You can use #csv to generate CSV output from arrays, and to_entries to break up the object's elements into said arrays:
$ jq -r '.rates | to_entries[] | [ .key, .value ] | #csv' input.json
"611",0.389165
"ABC",59.99
"ACP",0.014931
"ACT",0.021098
"ACT*",0.017178
"ADA",1.460965

Ansible output extract package version using jq

I have below json output from ansible, I want to parse this file using jq. I want to match packages name GeoIP and print version number.
How can I use jq to get this version number for matching package name? looking output like this
"GeoIP","1.5.0"
sample ansible output
{
"ansible_facts.packages": {
"GConf2": [
{
"arch": "x86_64",
"epoch": null,
"name": "GConf2",
"release": "8.el7",
"source": "rpm",
"version": "3.2.6"
}
],
"GeoIP": [
{
"arch": "x86_64",
"epoch": null,
"name": "GeoIP",
"release": "14.el7",
"source": "rpm",
"version": "1.5.0"
}
],
"ImageMagick": [
{
"arch": "x86_64",
"epoch": null,
"name": "ImageMagick",
"release": "18.el7",
"source": "rpm",
"version": "6.7.8.9"
}
],
}
}
First, make the source valid json by removing the comma in the third to last row.
With that done, we can start at the end and work back. The desired output can be produced using jq's #csv format. That, in turn, requires that the output is in an array. (See the section of the manual titled "Format strings and escaping".) So we need to get the data to look like this.
jq -r '["GeoIP","1.5.0"] | #csv'
One way to do that is to put each item in its own array and add the arrays together. (See the section titled "Addition".)
jq -r '["GeoIP"] + [.[] | .GeoIP[].version] | #csv'
Since map(x) is defined as [.[] | x], you can say this instead.
jq -r '["GeoIP"] + map(.GeoIP[].version) | #csv'
You can use a variable to specify the package name you want like this.
jq -r --arg "package" "GeoIP" '[$package] + map(.[$package][].version) | #csv'
Update
My original solution has unnecessary steps. The array can be made like this.
jq -r '[ "GeoIP", .[].GeoIP[].version ] | #csv'
Or, using a variable
jq -r --arg "package" "GeoIP" '[$package,(.[] | .[$package][].version)]| #csv'
Q: "How can I use jq to get the version number for matching package name?"
A: Try the script below
$ cat print-version.sh
#!/bin/sh
cat output.json | jq ".ansible_facts.packages.$1 | .[].version"
For example
$ print-version.sh GeoIP
"1.5.0"
Improved version of the script
Quoting the comment: "Using the shell's string interpolation as done here is risky and generally regarded as an "anti-pattern". – peak"
The improved script below does the job
$ cat print-version.sh
#!/bin/sh
cat output.json | jq ".ansible_facts.packages.$1[].version"
output.json
{
"ansible_facts": {
"packages": {
"GConf2": [
{
"arch": "x86_64",
"epoch": null,
"name": "GConf2",
"release": "8.el7",
"source": "rpm",
"version": "3.2.6"
}
],
"GeoIP": [
{
"arch": "x86_64",
"epoch": null,
"name": "GeoIP",
"release": "14.el7",
"source": "rpm",
"version": "1.5.0"
}
],
"ImageMagick": [
{
"arch": "x86_64",
"epoch": null,
"name": "ImageMagick",
"release": "18.el7",
"source": "rpm",
"version": "6.7.8.9"
}
]
}
}
}
I assume you want to get a loop over all key-values attributes in the ansible_facts.packages, so it's recommended to use a Bash loop to iterate over it. Also the sample contents of the JSON you provided don't compile, I removed the comma 3 rows from the end.
You need one jq to get all the keys of the ansible_facts.packages and then in each iteration to get the version of the given key (package).
To get all the keys:
[http_offline#greenhat-30 tmp]$ cat file.json | jq '."ansible_facts.packages" | keys[]'
"GConf2"
"GeoIP"
"ImageMagick"
[http_offline#greenhat-30 tmp]$
Loop:
for package in `cat file.json | jq '."ansible_facts.packages" | keys[]'`
do
VERSION=`cat file.json | jq ".\"ansible_facts.packages\".$package[0].version"`
echo "$package,$VERSION"
done
Sample run:
[http_offline#greenhat-30 tmp]$ for package in `cat file.json | jq '."ansible_facts.packages" | keys[]'`
> do
> VERSION=`cat file.json | jq ".\"ansible_facts.packages\".$package[0].version"`
> echo "$package,$VERSION"
> done
"GConf2","3.2.6"
"GeoIP","1.5.0"
"ImageMagick","6.7.8.9"
[http_offline#greenhat-30 tmp]$

Why is my jq failing on my JSON?

Haven't used jq before but I'm wanting to build a shell script that will get a JSON response and extract just the values. To learn I thought I would try on my blog's WP API but for some reason I'm getting an error of:
jq: error (at :322): Cannot index array with string "slug"
When researching for and testing previous questions:
jq: Cannot index array with string
jq is sed for JSON
JSON array to bash variables using jq
How to use jq in a shell pipeline?
How to extract data from a JSON file
The above reading I've tried to code:
URL="http://foobar.com"
RESPONSE=$(curl -so /dev/null -w "%{http_code}" $URL)
WPAPI="/wp-json/wp/v2"
IDENTIFIER="categories"
if (("$RESPONSE" == 200)); then
curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | jq '.' >> $IDENTIFIER.json
result=$(jq .slug $IDENTIFIER.json)
echo $result
else
echo "Not returned status 200";
fi
An additional attempt changing the jq after the curl:
curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | jq '.' | $IDENTIFIER.json
result=(jq -r '.slug' $IDENTIFIER.json)
echo $result
I can modify the uncompress with the python JSON tool:
result=(curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | python -m json.tool > $IDENTIFIER.json)
I can save the JSON to a file but when I use jq I cannot get just the slug and here are my other trys:
catCalled=$(curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | python -m json.tool | ./jq -r '.slug')
echo $catCalled
My end goal is to try to use jq in a shell script and build a slug array with jq. What am I doing wrong in my jq and can I use jq on a string without creating a file?
Return from curl after uncompress per comment request:
[
{
"id": 4,
"count": 18,
"description": "",
"link": "http://foobar.com/category/foo/",
"name": "Foo",
"slug": "foo",
"taxonomy": "category",
},
{
"id": 8,
"count": 9,
"description": "",
"link": "http://foobar.com/category/bar/",
"name": "Bar",
"slug": "bar",
"taxonomy": "category",
},
{
"id": 5,
"count": 1,
"description": "",
"link": "http://foobar.com/category/mon/",
"name": "Mon",
"slug": "mon",
"taxonomy": "category",
},
{
"id": 11,
"count": 8,
"description": "",
"link": "http://foobar.com/category/fort/",
"name": "Fort",
"slug": "fort",
"taxonomy": "category",
}
]
eventually my goal is trying to get the name of the slug's into an array like:
catArray=('foo','bar','mon', 'fort')
There are 2 issues here:
slug is not a root level element in your example json. The root level element is an array. If you want to access the slug property of each element of the array, you can do so like this:
jq '.[].slug' $IDENTIFIER.json
Your example json has trailing commas after the last property of each array element. Remove the commas after "taxonomy": "category".
If I take your sample json, remove the errant commas, save it to a plain text file called test.json and run the following command:
jq '.[].slug' test.json
I get the following output:
"foo"
"bar"
"mon"
"fort"
Preprocessing
Unfortunately, the JSON-like data shown as having been produced by curl is not strictly JSON. jq does not have a "relaxed JSON" mode, so in order to use jq, you will have to preprocess the JSON-like data, e.g. using hjson (see http://hjson.org/):
$ hjson -j input.qjson > input.json
jq
With the JSON in input.json:
$ jq -c 'map(.slug)' input.json
["foo","bar","mon","fort"]
your string is not json, notice how the last member of your objects ends with a comma,
{foo:"bar",baz:9,}
this is legal in javascript, but it's illegal in json. if you are supposed to be receiving json from that endpoint, then contact the people behind it and tell them to fix the bug (it's breaking the json specs by ending objects's last member with a comma, which is illegal in json.) - until it's fixed, i guess you can patch it with a little regex, but it's a dirty quickfix, and probably not very reliable, but running it through
perl -p -0777 -e 's/\"\,\s*}/\"}/g;' makes it legal json..

Find the value of key from JSON

I'd like to extract the "id" key from this single line of JSON.
I believe this can be accomplished with grep, but I am not sure on the correct way.
If there is a better way that does not have dependencies, I would be interested.
Here is my example output:
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": ["www.test.domain.com", "test.domain.com"],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
If you have a grep that can do Perl compatible regular expressions (PCRE):
$ grep -Po '"id": *\K"[^"]*"' infile.json
"4dCYd4W9i6gHQHvd"
-P enables PCRE
-o retains nothing but the match
"id": * matches "id" and an arbitrary amount of spaces
\K throws away everything to its left ("variable size positive look-behind")
"[^"]*" matches two quotes and all the non-quotes between them
If your grep can't do that, you an use
$ grep -o '"id": *"[^"]*"' infile.json | grep -o '"[^"]*"$'
"4dCYd4W9i6gHQHvd"
This uses grep twice. The result of the first command is "id": "4dCYd4W9i6gHQHvd"; the second command removes everything but a pair of quotes and the non-quotes between them, anchored at the end of the string ($).
But, as pointed out, you shouldn't use grep for this, but a tool that can parse JSON – for example jq:
$ jq '.data.id' infile.json
"4dCYd4W9i6gHQHvd"
This is just a simple filter for the id key in the data object. To get rid of the double quotes, you can use the -r ("raw output") option:
$ jq -r '.data.id' infile.json
4dCYd4W9i6gHQHvd
jq can also neatly pretty print your JSON:
$ jq . infile.json
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
Just pipe your data to jq and select by keys
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
} | jq '.data.id'
# 4dCYd4W9i6gHQHvd
Tutorial Here
I found myself that the best way is to use python, as it handles JSON natively and is preinstalled on most systems these days, unlike jq:
$ python -c 'import sys, json; print(json.load(sys.stdin)["data"]["id"])' < infile.json
4dCYd4W9i6gHQHvd
No python ,jq, awk, sed just GNU grep:
#!/bin/bash
json='{"data": {"name": "test", "id": "4dCYd4W9i6gHQHvd", "domains": ["www.test.domain.com", "test.domain.com"], "serverid": "bbBdbbHF8PajW221", "ssl": null, "runtime": "php5.6", "sysuserid": "4gm4K3lUerbSPfxz", "datecreated": 1474597357}, "actionid": "WXVAAHQDCSILMYTV"}'
echo $json | grep -o '"id": "[^"]*' | grep -o '[^"]*$'
Tested & working here: https://ideone.com/EG7fv7
source: https://brianchildress.co/parse-json-using-grep
$ grep -oP '"id": *"\K[^"]*' infile.json
4dCYd4W9i6gHQHvd
Hopefully it will work for all. As this will work for me to print without quotes.