Getting output using jq - json

I have the following JSON outp
{
"environment": {
"reg": "abc"
},
"system": {
"svcs": {
"upsvcs": [
{
"name": "monitor",
"tags": [],
"vmnts": [],
"label": "upsvcs",
"credentials": {
"Date": "Feb152018",
"time": "1330"
}
},
{
"name": "application",
"tags": [],
"vmnts": [],
"label": "upsvcs",
"credentials": {
"lastViewed": "2018-02-07"
}
}
]
}
}
and to retrieve Date value (from credentials). I have tried `curl xxx | jq -r '. | select (.Date)'
which is not returning any value. Can someone please let me know what is the correct syntax and any explanation on how to retrieve elements (or any articles that do so).
TIA

The (or at least a) short answer is:
.system.svcs.upsvcs[0].credentials.Date
Without a schema, it is often a bit tricky to get the exact path correctly, so you might want to consider using the jq filter paths. In your case:
$ jq -c paths input.json | grep Date
["system","svcs","upsvcs",0,"credentials","Date"]
You could also use this path (i.e., the above array) directly:
$ jq 'getpath(["system","svcs","upsvcs",0,"credentials","Date"])' input.json
"Feb152018"
and thus you could use a "path-free" query:
$ jq --argjson p $(jq -c paths input.json | grep --max 1 Date) 'getpath($p)' input.json
"Feb152018"
Another approach would be just to retrieve all “truthy” .Date values, no matter where they appear:
$ jq -c '.. | .Date? // empty' input.json
"Feb152018"
select
Since you mentioned select, please note that:
$ jq -c '.. | select(.Date?)' input.json
{"Date":"Feb152018","time":"1330"}
Further information
For further information about jq in general and the topic of retrieval in particular, see the online tutorial, manual, and FAQ:
https://stedolan.github.io/jq/tutorial/
https://stedolan.github.io/jq/manual/v1.5/
https://github.com/stedolan/jq/wiki/FAQ

Related

parse json and get values with same type

In the below json, I'm unable to get the value which have reporter only.
the output should be jhoncena only which should written into a file.
jq -r '.values' response.json | grep reporter
the output for this is
"name": "reporter-jhoncena"
{
"size": 3,
"limit": 25,
"isLastPage": true,
"values": [
{
"name": "hello-world"
},
{
"name": "test-frame"
},
{
"name": "reporter-jhoncena"
}
],
"start": 0
}
You can use capture :
jq -r '.values[].name
| capture("^reporter-(?<name>.*)").name
' response.json
You can use split such as
jq -r '.values[2].name | split("-")[1]' response.json
Demo
Edit : Alternatively you can use
jq -r '.values[].name | select(.|split("-")[0]=="reporter")|split("-")[1]' response.json > outfile.txt
without knowing the order of the name element within the array
Demo
jq -r '.values[]
| select(.name|index("reporter"))
| .name
| sub("reporter-";"")' in.json > out.txt
Of course you might wish to use a different selection criterion, e.g. using startswith or test.

Ansible output extract package version using jq

I have below json output from ansible, I want to parse this file using jq. I want to match packages name GeoIP and print version number.
How can I use jq to get this version number for matching package name? looking output like this
"GeoIP","1.5.0"
sample ansible output
{
"ansible_facts.packages": {
"GConf2": [
{
"arch": "x86_64",
"epoch": null,
"name": "GConf2",
"release": "8.el7",
"source": "rpm",
"version": "3.2.6"
}
],
"GeoIP": [
{
"arch": "x86_64",
"epoch": null,
"name": "GeoIP",
"release": "14.el7",
"source": "rpm",
"version": "1.5.0"
}
],
"ImageMagick": [
{
"arch": "x86_64",
"epoch": null,
"name": "ImageMagick",
"release": "18.el7",
"source": "rpm",
"version": "6.7.8.9"
}
],
}
}
First, make the source valid json by removing the comma in the third to last row.
With that done, we can start at the end and work back. The desired output can be produced using jq's #csv format. That, in turn, requires that the output is in an array. (See the section of the manual titled "Format strings and escaping".) So we need to get the data to look like this.
jq -r '["GeoIP","1.5.0"] | #csv'
One way to do that is to put each item in its own array and add the arrays together. (See the section titled "Addition".)
jq -r '["GeoIP"] + [.[] | .GeoIP[].version] | #csv'
Since map(x) is defined as [.[] | x], you can say this instead.
jq -r '["GeoIP"] + map(.GeoIP[].version) | #csv'
You can use a variable to specify the package name you want like this.
jq -r --arg "package" "GeoIP" '[$package] + map(.[$package][].version) | #csv'
Update
My original solution has unnecessary steps. The array can be made like this.
jq -r '[ "GeoIP", .[].GeoIP[].version ] | #csv'
Or, using a variable
jq -r --arg "package" "GeoIP" '[$package,(.[] | .[$package][].version)]| #csv'
Q: "How can I use jq to get the version number for matching package name?"
A: Try the script below
$ cat print-version.sh
#!/bin/sh
cat output.json | jq ".ansible_facts.packages.$1 | .[].version"
For example
$ print-version.sh GeoIP
"1.5.0"
Improved version of the script
Quoting the comment: "Using the shell's string interpolation as done here is risky and generally regarded as an "anti-pattern". – peak"
The improved script below does the job
$ cat print-version.sh
#!/bin/sh
cat output.json | jq ".ansible_facts.packages.$1[].version"
output.json
{
"ansible_facts": {
"packages": {
"GConf2": [
{
"arch": "x86_64",
"epoch": null,
"name": "GConf2",
"release": "8.el7",
"source": "rpm",
"version": "3.2.6"
}
],
"GeoIP": [
{
"arch": "x86_64",
"epoch": null,
"name": "GeoIP",
"release": "14.el7",
"source": "rpm",
"version": "1.5.0"
}
],
"ImageMagick": [
{
"arch": "x86_64",
"epoch": null,
"name": "ImageMagick",
"release": "18.el7",
"source": "rpm",
"version": "6.7.8.9"
}
]
}
}
}
I assume you want to get a loop over all key-values attributes in the ansible_facts.packages, so it's recommended to use a Bash loop to iterate over it. Also the sample contents of the JSON you provided don't compile, I removed the comma 3 rows from the end.
You need one jq to get all the keys of the ansible_facts.packages and then in each iteration to get the version of the given key (package).
To get all the keys:
[http_offline#greenhat-30 tmp]$ cat file.json | jq '."ansible_facts.packages" | keys[]'
"GConf2"
"GeoIP"
"ImageMagick"
[http_offline#greenhat-30 tmp]$
Loop:
for package in `cat file.json | jq '."ansible_facts.packages" | keys[]'`
do
VERSION=`cat file.json | jq ".\"ansible_facts.packages\".$package[0].version"`
echo "$package,$VERSION"
done
Sample run:
[http_offline#greenhat-30 tmp]$ for package in `cat file.json | jq '."ansible_facts.packages" | keys[]'`
> do
> VERSION=`cat file.json | jq ".\"ansible_facts.packages\".$package[0].version"`
> echo "$package,$VERSION"
> done
"GConf2","3.2.6"
"GeoIP","1.5.0"
"ImageMagick","6.7.8.9"
[http_offline#greenhat-30 tmp]$

jq: convert array to object indexed by filename?

Using jq how can I convert an array into object indexed by filename, or read multiple files into one object indexed by their filename?
e.g.
jq -s 'map(select(.roles[]? | contains ("mysql")))' -C dir/file1.json dir/file2.json
This gives me the data I want, but I need to know which file they came from.
So instead of
[
{ "roles": ["mysql"] },
{ "roles": ["mysql", "php"] }
]
for output, I want:
{
"file1": { "roles": ["mysql"] },
"file2": { "roles": ["mysql", "php"] }
}
I do want the ".json" file extension stripped too if possible, and just the basename (dir excluded).
Example
file1.json
{ "roles": ["mysql"] }
file2.json
{ "roles": ["mysql", "php"] }
file3.json
{ }
My real files obviously have other stuff in them too, but that should be enough for this example. file3 is simply to demonstrate "roles" is sometimes missing.
In other words: I'm trying to find files that contain "mysql" in their list of "roles". I need the filename and contents combined into one JSON object.
To simplify the problem further:
jq 'input_filename' f1 f2
Gives me all the filenames like I want, but I don't know how to combine them into one object or array.
Whereas,
jq -s 'map(input_filename)' f1 f2
Gives me the same filename repeated once for each file. e.g. [ "f1", "f1" ] instead of [ "f1", "f2" ]
If your jq has inputs (as does jq 1.5) then the task can be accomplished with just one invocation of jq.
Also, it might be more efficient to use any than iterating over all the elements of .roles.
The trick is to invoke jq with the -n option, e.g.
jq -n '
[inputs
| select(.roles and any(.roles[]; contains("mysql")))
| {(input_filename | gsub(".*/|\\.json$";"")): .}]
| add' file*.json
jq approach:
jq 'if (.roles[] | contains("mysql")) then {(input_filename | gsub(".*/|\\.json$";"")): .}
else empty end' ./file1.json ./file2.json | jq -s 'add'
The expected output:
{
"file1": {
"roles": [
"mysql"
]
},
"file2": {
"roles": [
"mysql",
"php"
]
}
}

Convert a nested JSON of objects into array into a bash array using jq

I am doing something fundamentally wrong but just can't see what, could some kind person point out my fault with jq or JSON here?
I have the the following child objects contained within an array “entries”
{
"profile": {
"name": "TesterRun1",
"download": {
"entries": [{
"ENTRY_A": "testserver1_place_com",
"store": "A",
"type": "direct"
},
{
"ENTRY_B": "testserver2_anotherplace_com",
"store": "B",
"type": "bypass"
},
{
"ENTRY_B": "testserver2_anotherplace_com",
"store": "A",
"type": "bypass"
}
]
}
}
}
I wish to convert these to an array accessible by bash via the jq function “to_entries” using the below query but so far nothing!
jq 'to_entries|.[]|.profile.download.entries|select(.store=="A")|.[]'
You can see here that nothing is returned on JQ Play - enter link description here
Please help save my sanity, what am I doing wrong
to_entries has nothing whatsoever to do with exposing JQ results to bash. Rather, it takes each entry in a JSON object and emits a {"key": key, "value": value} pair.
That can be useful, if you want to identify and extract arbitrary keys. For example:
#!/usr/bin/env bash
jq_script='
.profile.download.entries[]
| select(.store == "A")
| to_entries[]
| select(.key != "store")
| select(.key != "type")
| [.key, .value]
| #tsv
'
declare -A array=( )
while IFS=$'\t' read -r key value; do
array[$key]=$value
done < <(jq -r "$jq_script")
# print array output
declare -p array
...will, when given your input on stdin, emit (albeit on a single line, without the whitespace changes):
declare -A array=([ENTRY_A]="testserver1_place_com"
[ENTRY_B]="testserver2_anotherplace_com" )
...which I assume, for lack of any better description in the question, is what you actually want.
Here is a slightly different approach (with some cleaned-up data) which captures the output from jq into separate column arrays.
#!/bin/bash
data='{
"profile": {
"name": "TesterRun1",
"download": {
"entries": [
{
"entry": "testserver1_place_com",
"store": "A",
"type": "direct"
},
{
"entry": "testserver2_anotherplace_com",
"store": "B",
"type": "bypass"
},
{
"entry": "testserver2_anotherplace_com",
"store": "A",
"type": "bypass"
}
]
}
}
}'
filter='
.profile.download.entries[]
| select(.store == "A")
| .entry, .store, .type
'
declare -a ENTRY
declare -a STORE
declare -a TYPE
i=0
while read -r entry; read -r store; read -r type; do
ENTRY[$i]="$entry"
STORE[$i]="$store"
TYPE[$i]="$type"
i=$((i + 1))
done < <(jq -Mr "$filter" <<< "$data")
declare -p ENTRY STORE TYPE
Output
declare -a ENTRY='([0]="testserver1_place_com" [1]="testserver2_anotherplace_com")'
declare -a STORE='([0]="A" [1]="A")'
declare -a TYPE='([0]="direct" [1]="bypass")'

Find the value of key from JSON

I'd like to extract the "id" key from this single line of JSON.
I believe this can be accomplished with grep, but I am not sure on the correct way.
If there is a better way that does not have dependencies, I would be interested.
Here is my example output:
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": ["www.test.domain.com", "test.domain.com"],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
If you have a grep that can do Perl compatible regular expressions (PCRE):
$ grep -Po '"id": *\K"[^"]*"' infile.json
"4dCYd4W9i6gHQHvd"
-P enables PCRE
-o retains nothing but the match
"id": * matches "id" and an arbitrary amount of spaces
\K throws away everything to its left ("variable size positive look-behind")
"[^"]*" matches two quotes and all the non-quotes between them
If your grep can't do that, you an use
$ grep -o '"id": *"[^"]*"' infile.json | grep -o '"[^"]*"$'
"4dCYd4W9i6gHQHvd"
This uses grep twice. The result of the first command is "id": "4dCYd4W9i6gHQHvd"; the second command removes everything but a pair of quotes and the non-quotes between them, anchored at the end of the string ($).
But, as pointed out, you shouldn't use grep for this, but a tool that can parse JSON – for example jq:
$ jq '.data.id' infile.json
"4dCYd4W9i6gHQHvd"
This is just a simple filter for the id key in the data object. To get rid of the double quotes, you can use the -r ("raw output") option:
$ jq -r '.data.id' infile.json
4dCYd4W9i6gHQHvd
jq can also neatly pretty print your JSON:
$ jq . infile.json
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
Just pipe your data to jq and select by keys
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
} | jq '.data.id'
# 4dCYd4W9i6gHQHvd
Tutorial Here
I found myself that the best way is to use python, as it handles JSON natively and is preinstalled on most systems these days, unlike jq:
$ python -c 'import sys, json; print(json.load(sys.stdin)["data"]["id"])' < infile.json
4dCYd4W9i6gHQHvd
No python ,jq, awk, sed just GNU grep:
#!/bin/bash
json='{"data": {"name": "test", "id": "4dCYd4W9i6gHQHvd", "domains": ["www.test.domain.com", "test.domain.com"], "serverid": "bbBdbbHF8PajW221", "ssl": null, "runtime": "php5.6", "sysuserid": "4gm4K3lUerbSPfxz", "datecreated": 1474597357}, "actionid": "WXVAAHQDCSILMYTV"}'
echo $json | grep -o '"id": "[^"]*' | grep -o '[^"]*$'
Tested & working here: https://ideone.com/EG7fv7
source: https://brianchildress.co/parse-json-using-grep
$ grep -oP '"id": *"\K[^"]*' infile.json
4dCYd4W9i6gHQHvd
Hopefully it will work for all. As this will work for me to print without quotes.