Bash Jq parse json string - json

I've to call a file and pass a json as parameters in this way
(suppose that my file is called test.sh), from bash I need to do something like this:
./test.sh "[{\"username\":\"user1\",\"password\":\"pwd1\",\"group\":\"usergroup1\"},{\"username\":\"user2\",\"password\":\"pwd2\",\"group\":\"usergroup2\"},{\"username\":\"user3\",\"password\":\"pwd3\",\"group\":\"usergroup3\"}]"
and the content of test.sh is the following
#!/bin/bash
#read the json
system_user="$1"
printf "$system_user"
accounts=($(jq -s ".[]" <<< $system_user))
printf "$accounts"
for account in "${accounts[#]}"
do
printf "\n\n$account\n\n"
done
the output of -> printf "$system_user" is
[{"username":"user1","password":"pwd1","group":"usergroup1"},{"username":"user2","password":"pwd2","group":"usergroup2"},{"username":"user3","password":"pwd3","group":"usergroup3"}]
but the output of -> printf "$accounts" is something like this
[
[
{
"username":
"user1"
etc. etc. one object for each token :-(
and so on, but what I was expecting is an array of three object (like you can test on jqplay.org)
{
"username": "user1",
"password": "pwd1",
"group": "usergroup1"
}
{
"username": "user2",
"password": "pwd2",
"group": "usergroup2"
}
{
"username": "user3",
"password": "pwd3",
"group": "usergroup3"
}
In this way I can make a foreach on ${accounts[#]}
What I'm doing wrong?
Thank you

With the -c option, you can print each JSON object on a single line, making it easier to populate the array you want.
$ readarray -t arr < <(jq -c '.[]' <<< "[{\"username\":\"user1\",\"password\":\"pwd1\",\"group\":\"usergroup1\"},{\"username\":\"user2\",\"password\":\"pwd2\",\"group\":\"usergroup2\"},{\"username\":\"user3\",\"password\":\"pwd3\",\"group\":\"usergroup3\"}]")
$ printf "Object: %s\n" "${arr[#]}"
Object: {"username":"user1","password":"pwd1","group":"usergroup1"}
Object: {"username":"user2","password":"pwd2","group":"usergroup2"}
Object: {"username":"user3","password":"pwd3","group":"usergroup3"}

You are interchanging bash arrays and JSON arrays. When you are creating accounts array, bash splits the elements per each whitespace. That's why you don't get what you expect. You can try the following:
declare -A accounts
while IFS="=" read -r key value
do
accounts[$key]="$value"
done < <(jq -r "to_entries|map(\"\(.key)=\(.value)\")|.[]" <<< $system_user)
for account in "${accounts[#]}"
do
printf "$account\n"
done
(stolen from here: https://stackoverflow.com/a/26717401/328977)
to get the following output:
{"username":"user1","password":"pwd1","group":"usergroup1"}
{"username":"user2","password":"pwd2","group":"usergroup2"}
{"username":"user3","password":"pwd3","group":"usergroup3"}

Related

Create a JSON file with parameters from a shell script

Let's say that with a bash script I want to create a file, so to the command to create the file will be something like this:
myscript hostgroup_one 2
hostgroup_one and the number 2 are parameters.
How can I insert the parameters in the lines below and output all the lines as a file?
{
"bool": {
"should": [
{
"match_phrase": {
"hostgroup.keyword": "$1"
}
}
],
"minimum_should_match": $2
}
}
I'd use jq to build the JSON:
jq -n \
--arg hostgroup "$1" \
--argjson minimum "$2" \
'{bool: {should: [{match_phrase: {"hostgroup.keyword": $hostgroup}}], minimum_should_match: $minimum}}'
will produce your desired output.
While jq is a great tool for manipulating json, as glenn jackman recommends, if you don't have it and your sysadmin won't install it (?!?)...
You can use a "here document"
Your myscript could look something like this:
#!/bin/bash
echo "dollar-1 is '${1}' dollar-2 is '${2}' dollar-3 is '${3}'"
cat <<EOF >"${1}"
{
"bool": {
"should": [
{
"match_phrase": {
"hostgroup.keyword": "${2}"
}
}
],
"minimum_should_match": ${3}
}
}
EOF
echo "done"
I've made the output filename the first parameter, and then your two parameters, so it would be run like:
myscript output.json hostgroup_one 2
You don't need to do that, you could use 2 params and redirect output:
myscript hostgroup_one 2 > output.json
(note you don't have to use EOF as your here-document delimiter, I just like it)
Of course you don't need the echo statements, and you should have error checking (does ${#} equal 3 parameters?)

How to iterate a JSON array of objects with jq and grab multiple variables from each object in each loop

I need to grab variables from JSON properties.
The JSON array looks like this (GitHub API for repository tags), which I obtain from a curl request.
[
{
"name": "my-tag-name",
"zipball_url": "https://api.github.com/repos/path-to-my-tag-name",
"tarball_url": "https://api.github.com/repos/path-to-my-tag-name-tarball",
"commit": {
"sha": "commit-sha",
"url": "https://api.github.com/repos/path-to-my-commit-sha"
},
"node_id": "node-id"
},
{
"name": "another-tag-name",
"zipball_url": "https://api.github.com/repos/path-to-my-tag-name",
"tarball_url": "https://api.github.com/repos/path-to-my-tag-name-tarball",
"commit": {
"sha": "commit-sha",
"url": "https://api.github.com/repos/path-to-my-commit-sha"
},
"node_id": "node-id"
},
]
In my actual JSON there are 100s of objects like these.
While I loop each one of these I need to grab the name and the commit URL, then perform more operations with these two variables before I get to the next object and repeat.
I tried (with and without -r)
tags=$(curl -s -u "${GITHUB_USERNAME}:${GITHUB_TOKEN}" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/path-to-my-repository/tags?per_page=100&page=${page}")
for row in $(jq -r '.[]' <<< "$tags"); do
tag=$(jq -r '.name' <<< "$row")
# I have also tried with the syntax:
url=$(echo "${row}" | jq -r '.commit.url')
# do stuff with $tag and $url...
done
But I get errors like:
parse error: Unfinished JSON term at EOF at line 2, column 0 jq: error
(at :1): Cannot index string with string "name" } parse error:
Unmatched '}' at line 1, column 1
And from the terminal output it appears that it is trying to parse $row in a strange way, trying to grab .name from every substring? Not sure.
I am assuming the output from $(jq '.[]' <<< "$tags") could be valid JSON, from which I could again use jq to grab the object properties I need, but maybe that is not the case? If I output ${row} it does look like valid JSON to me, and I tried pasting the results in a JSON validator, everything seems to check out...
How do I grab the ".name" and ".commit.url" for each of these object before I move onto the next one?
Thanks
It would be better to avoid calling jq more than once. Consider, for example:
while read -r name ; do
read -r url
echo "$name" "$url"
done < <( curl .... | jq -r '.[] | .name, .commit.url' )
where curl .... signifies the relevant invocation of curl.

parse json and put in array

I am trying to parse a JSON file which looks like this and then trying to store each field in an array, then trying to read it. However, its going in an infinite loop, I think. Anyone knows what I am doing wrong?
#!/bin/bash
test(){
local file="/Users/f.json"
if [ -f "$file" ]; then
echo "present"
else
echo "absent"
fi
#jq . f.json
while read rule; do
local idd
local username
idd=$(jq --raw-output '.id' <<< ${rule})
username=$(jq --raw-output '.username' <<< ${rule})
#username=$(jq --raw-output '.username')
done
for (( i=0; i<${#idd[#]}; i++ )); do
echo "${idd[i]}"
done
}
test
Here is json:
{
"id": 5679162,
"username": "ryderw1"
}
{
"id": 5679163,
"username": "ryderw3"
}
{
"id": 5679164,
"username": "ryderw4"
}
My desired o/p should be:
5679162
5679163
5679164
I suggest this to read output from jq to an array.
mapfile -t idd < <(jq '.id' /Users/f.json)
declare -p idd
Output:
declare -a idd=([0]="5679162" [1]="5679163" [2]="5679164")

How to parse json in shell script using jq add store into array in shell script [duplicate]

I'm parsing a JSON response with a tool called jq.
The output from jq will give me a list of full names in my command line.
I have the variable getNames which contains JSON, for example:
{
"count": 49,
"user": [{
"username": "jamesbrown",
"name": "James Brown",
"id": 1
}, {
"username": "matthewthompson",
"name": "Matthew Thompson",
"id": 2
}]
}
I pass this through JQ to filter the json using the following command:
echo $getNames | jq -r .user[].name
Which gives me a list like this:
James Brown
Matthew Thompson
I want to put each one of these entries into a bash array, so I enter the following commands:
declare -a myArray
myArray=( `echo $getNames | jq -r .user[].name` )
However, when I try to print the array using:
printf '%s\n' "${myArray[#]}"
I get the following:
James
Brown
Matthew
Thompson
How do I ensure that a new index is created after a new line and not a space? Why are the names being separated?
Thanks.
A simple script in bash to feed each line of the output into the array myArray.
#!/bin/bash
myArray=()
while IFS= read -r line; do
[[ $line ]] || break # break if line is empty
myArray+=("$line")
done < <(jq -r .user[].name <<< "$getNames")
# To print the array
printf '%s\n' "${myArray[#]}"
Just use mapfile command to read multiple lines into an array like this:
mapfile -t myArray < <(jq -r .user[].name <<< "$getNames")

Bash store json response in another variable

I'm curling an endpoint:
#!/bin/bash
instance_info=$(curl -sk https://internal.admin.com/app/instance)
which gives a json response:
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}
I'm trying to avoid curling more than once to get the variables I need from the json using JQ.
Using bash I'd really appreciate if someone can show me how to store the response as another var and then use this to variablize name: server: url: http_port:
The following seems to run the curl twice:
#!/bin/bash
instance_info=$(curl -sk https://internal.admin.com/app/instance)
server_name=$(echo instance_info | /usr/bin/jq --raw-output '.server')
url=$(echo instance_info | /usr/bin/jq --raw-output '.url')
You are calling curl once and this suffices. Then you have the content in a variable, so you can access it without calling curl again.
Regarding your code, your approach is fine but you are missing $ when you are echoing the variable:
server_name=$(echo $instance_info | /usr/bin/jq --raw-output '.server')
# ^
See a sample. Here I hardcode the JSON:
your_json='
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}'
for the server:
$ echo "$your_json" | jq --raw-output '.server'
webserver1
For the url you need to indicate the block where it lies on. That is, metadata:
$ echo "$your_json" | jq --raw-output '.metadata.url'
www.google.com
To store into a variable, say:
your_field=$(echo "$your_json" | jq --raw-output 'XXXX')
# ^^^^
Here is a script which demonstrates how to use jq's #sh formatting directive along with bash eval to set bash variables using the output of a filter. In this case we hardcode the json which presumably would have come from curl.
#!/bin/bash
instance_info='
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}'
eval "$(jq -M -r '#sh "server_name=\(.server) url=\(.metadata.url)"' <<< "$instance_info")"
echo $server_name
echo $url
When run this produces the output
webserver1
www.google.com
The Extract data and set shell variables section of the JQ Cookbook has more examples of #sh.
sed -rn 's/[ ]*"([^"]*)"[^"]*("[^"]*").*/\1=\2/gp'' file.json
O/P:
basePath="/install"
deployed_artifact="app01"
docker="True"
http_port="7471"
url="www.google.com"
name="app-01"
server="webserver1"
eval this output you can get the variable
eval $(sed -rn 's/[ ]*"([^"]*)"[^"]*("[^"]*").*/\1=\2/gp' file.json )
echo $name $server
O/p:
app-01 webserver1
Read the values with one jq pass then there's no need to store the json:
read name server url http_port < <(
curl -sk https://internal.admin.com/app/instance | \
jq --raw-output '[.name, .server, .metadata.url, .metadata.http_port] | join(" ")'
)
echo -e "\$name: $name\n\$server: $server\n\$url: $url\n\$http_port $http_port"
$name: app-01
$server: webserver1
$url: www.google.com
$http_port 7471