Bash store json response in another variable - json

I'm curling an endpoint:
#!/bin/bash
instance_info=$(curl -sk https://internal.admin.com/app/instance)
which gives a json response:
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}
I'm trying to avoid curling more than once to get the variables I need from the json using JQ.
Using bash I'd really appreciate if someone can show me how to store the response as another var and then use this to variablize name: server: url: http_port:
The following seems to run the curl twice:
#!/bin/bash
instance_info=$(curl -sk https://internal.admin.com/app/instance)
server_name=$(echo instance_info | /usr/bin/jq --raw-output '.server')
url=$(echo instance_info | /usr/bin/jq --raw-output '.url')

You are calling curl once and this suffices. Then you have the content in a variable, so you can access it without calling curl again.
Regarding your code, your approach is fine but you are missing $ when you are echoing the variable:
server_name=$(echo $instance_info | /usr/bin/jq --raw-output '.server')
# ^
See a sample. Here I hardcode the JSON:
your_json='
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}'
for the server:
$ echo "$your_json" | jq --raw-output '.server'
webserver1
For the url you need to indicate the block where it lies on. That is, metadata:
$ echo "$your_json" | jq --raw-output '.metadata.url'
www.google.com
To store into a variable, say:
your_field=$(echo "$your_json" | jq --raw-output 'XXXX')
# ^^^^

Here is a script which demonstrates how to use jq's #sh formatting directive along with bash eval to set bash variables using the output of a filter. In this case we hardcode the json which presumably would have come from curl.
#!/bin/bash
instance_info='
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}'
eval "$(jq -M -r '#sh "server_name=\(.server) url=\(.metadata.url)"' <<< "$instance_info")"
echo $server_name
echo $url
When run this produces the output
webserver1
www.google.com
The Extract data and set shell variables section of the JQ Cookbook has more examples of #sh.

sed -rn 's/[ ]*"([^"]*)"[^"]*("[^"]*").*/\1=\2/gp'' file.json
O/P:
basePath="/install"
deployed_artifact="app01"
docker="True"
http_port="7471"
url="www.google.com"
name="app-01"
server="webserver1"
eval this output you can get the variable
eval $(sed -rn 's/[ ]*"([^"]*)"[^"]*("[^"]*").*/\1=\2/gp' file.json )
echo $name $server
O/p:
app-01 webserver1

Read the values with one jq pass then there's no need to store the json:
read name server url http_port < <(
curl -sk https://internal.admin.com/app/instance | \
jq --raw-output '[.name, .server, .metadata.url, .metadata.http_port] | join(" ")'
)
echo -e "\$name: $name\n\$server: $server\n\$url: $url\n\$http_port $http_port"
$name: app-01
$server: webserver1
$url: www.google.com
$http_port 7471

Related

Using jq with an unknown amount of arguments from shell script?

I'm about to lose my mind working with jq for the first time today. I've tried every ever so dirty way to make this work somehow. Let's get down to business:
I'm trying to further develop the pretty barebones API for Unifi Controllers for some use cases:
https://dl.ui.com/unifi/6.0.41/unifi_sh_api
Here's some mock up data (the real json data is thousands of lines):
{
"meta": {
"rc": "ok"
},
"data": [
{
"_id": "61a0da77f730e404af0edc3c",
"ip": "10.19.31.120",
"mac": "ff:ec:ff:89:ff:58",
"name": "Test-Accesspoint"
}
]
}
Here is my advancedapi.sh
#!/bin/bash
source unifiapi.sh
unifi_login 1>dev/null
if [ "$1" ];
then
command=$1
shift
else
echo "Please enter command"
exit
fi
#jq -r '.data[] | "\(.name),\(.mac)"'
assembleKeys(){
c="0"
seperator=";"
keys=$(for i in "$#"
do
c=$(expr $c + 1)
if [ $c -gt 1 ];
then
echo -n "${seperator}\(.${i})"
else
echo -n "\(.${i})"
fi
done
echo)
}
assembleJQ(){
c=0
jq=$(echo -en 'jq -r -j \x27.data[] | '
for i in "$#"
do
if [ "$c" != "0" ];
then
echo -en ",\x22 \x22,"
fi
c=1
echo -en ".$i"
done
echo -en ",\x22\\\n\x22\x27")
echo "$jq"
}
getDeviceKeys(){
assembleJQ "$#"
#unifi_list_devices | jq -r '.data[]' | jq -r "${keys}"
#export keys=mac
#export second=name
#unifi_list_devices | jq -r -j --arg KEYS "$keys" --arg SECOND "$second" '.data[] | .[$KEYS]," ",.[$SECOND],"\n"'
unifi_list_devices | $jq
#unifi_list_devices | jq -r -j '.data[] | .mac," ",.name,"\n"'
}
"$command" $#
Users should be able to call any function from the API with as many arguments as they want.
So basically this:
#export keys=mac
#export second=name
#unifi_list_devices | jq -r -j --arg KEYS "$keys" --arg SECOND "$second" '.data[] | .[$KEYS]," ",.[$SECOND],"\n"'
which works, but only with a limited number of arguments. I want to pass $# to jq.
I'm trying to get the name and mac-address of a device by calling:
./advancedapi.sh getDeviceKeys mac name
this should return the mac-address and name of the device. But it only gives me this:
jq -r -j '.data[] | .mac," ",.name,"\n"'
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'.data[]
jq: 1 compile error
The top-most line being the jq command, which works perfectly fine when called manually.
The assembleKeys function was my attempt at generating a string that looks like this:
jq -r '.data[] | "\(.name),\(.mac)"'
Can anyone explain to me, preferably in-depth, how to do this? I'm going insane here!
Thanks!
I want to pass $# to jq.
There are actually many ways to do this, but the simplest might be using the --args command-line option, as illustrated here:
File: check
#/bin/bash
echo "${#}"
jq -n '$ARGS.positional' --args "${#}"
Example run:
./check 1 2 3
1 2 3
[
"1",
"2",
"3"
]
Projection
Here's an example that might be closer to what you're looking for.
Using your sample JSON:
File: check2
#/bin/bash
jq -r '
def project($array):
. as $in | reduce $array[] as $k ([]; . + [$in[$k]]);
$ARGS.positional,
(.data[] | project($ARGS.positional))
| #csv
' input.json --args "${#}"
./check2 name mac
"name","mac"
"Test-Accesspoint","ff:ec:ff:89:ff:58"
There is error in the json (red highlighted commas)(that commas are not needed)
$
$ cat j.json | jq -r '.data[] | "\(.name)█\(.mac)█\(.ip)█\(._id)"'
Test-Accesspoint█ff:ec:ff:89:ff:58█10.19.31.120█61a0da77f730e404af0edc3c
$ cat j.json
{
"meta": {
"rc": "ok"
},
"data": [
{
"_id": "61a0da77f730e404af0edc3c",
"ip": "10.19.31.120",
"mac": "ff:ec:ff:89:ff:58",
"name": "Test-Accesspoint"
}
]
}
$
If the api have given this json you should probably edit it, before piping it into jq [maybe with sed or something like it]

How to iterate a JSON array of objects with jq and grab multiple variables from each object in each loop

I need to grab variables from JSON properties.
The JSON array looks like this (GitHub API for repository tags), which I obtain from a curl request.
[
{
"name": "my-tag-name",
"zipball_url": "https://api.github.com/repos/path-to-my-tag-name",
"tarball_url": "https://api.github.com/repos/path-to-my-tag-name-tarball",
"commit": {
"sha": "commit-sha",
"url": "https://api.github.com/repos/path-to-my-commit-sha"
},
"node_id": "node-id"
},
{
"name": "another-tag-name",
"zipball_url": "https://api.github.com/repos/path-to-my-tag-name",
"tarball_url": "https://api.github.com/repos/path-to-my-tag-name-tarball",
"commit": {
"sha": "commit-sha",
"url": "https://api.github.com/repos/path-to-my-commit-sha"
},
"node_id": "node-id"
},
]
In my actual JSON there are 100s of objects like these.
While I loop each one of these I need to grab the name and the commit URL, then perform more operations with these two variables before I get to the next object and repeat.
I tried (with and without -r)
tags=$(curl -s -u "${GITHUB_USERNAME}:${GITHUB_TOKEN}" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/path-to-my-repository/tags?per_page=100&page=${page}")
for row in $(jq -r '.[]' <<< "$tags"); do
tag=$(jq -r '.name' <<< "$row")
# I have also tried with the syntax:
url=$(echo "${row}" | jq -r '.commit.url')
# do stuff with $tag and $url...
done
But I get errors like:
parse error: Unfinished JSON term at EOF at line 2, column 0 jq: error
(at :1): Cannot index string with string "name" } parse error:
Unmatched '}' at line 1, column 1
And from the terminal output it appears that it is trying to parse $row in a strange way, trying to grab .name from every substring? Not sure.
I am assuming the output from $(jq '.[]' <<< "$tags") could be valid JSON, from which I could again use jq to grab the object properties I need, but maybe that is not the case? If I output ${row} it does look like valid JSON to me, and I tried pasting the results in a JSON validator, everything seems to check out...
How do I grab the ".name" and ".commit.url" for each of these object before I move onto the next one?
Thanks
It would be better to avoid calling jq more than once. Consider, for example:
while read -r name ; do
read -r url
echo "$name" "$url"
done < <( curl .... | jq -r '.[] | .name, .commit.url' )
where curl .... signifies the relevant invocation of curl.

How to loop a json keys result from bash script

input.json:-
{
"menu": {
"id": "file",
"value": "File",
"user": {
"address": "USA",
"email": "user#gmail.com"
}
}
}
Command:-
result=$(cat input.json | jq -r '.menu | keys[]')
Result:-
id
value
user
Loop through result:-
for type in "${result[#]}"
do
echo "--$type--"
done
Output:-
--id
value
user--
I want to do process the keys values in a loop. When I do the above, It result as a single string.
How can I do a loop with json keys result in bash script?
The canonical way :
file='input.json'
cat "$file" | jq -r '.menu | keys[]' |
while IFS= read -r value; do
echo "$value"
done
bash faq #1
But you seems to want an array, so the syntax is (missing parentheses) :
file='input.json'
result=( $(cat "$file" | jq -r '.menu | keys[]') )
for type in "${result[#]}"; do
echo "--$type--"
done
Output:
--id--
--value--
--user--
Using bash to just print an object keys from JSON data is redundant.
Jq is able to handle it by itself. Use the following simple jq solution:
jq -r '.menu | keys_unsorted[] | "--"+ . +"--"' input.json
The output:
--id--
--value--
--user--

Parsing curl json output using awk

Using this command in a loop:
curl -s -X POST -F "filedata=#inputfile" -F "containerid=documentLibrary" -F "destination=filelocation" http://user:pass#server:host/service/api/upload;
I am able to get the following:
{
"nodeRef": "123",
"fileName": "filename.pdf",
"status":
{
"code": 200,
"name": "OK",
"description": "File uploaded successfully"
}
}
this snippet will appear 100s, 1000s of times. I need to extract just the nodeRef value (123) and put into a csv using awk or any other parsing tool that does not require installation like jq.
lacking specialized json tools, perhaps grep
... | grep -oP '(?<="nodeRef": ")\w+'
still assumes something about the format...
Few awk approach:-
<curl command> | awk -F'"' '/nodeRef/{ print $(NF-1) }'
<curl command> | awk '/nodeRef/{X=$NF;gsub(/[",]/,Y,X);print X}'

Bash Jq parse json string

I've to call a file and pass a json as parameters in this way
(suppose that my file is called test.sh), from bash I need to do something like this:
./test.sh "[{\"username\":\"user1\",\"password\":\"pwd1\",\"group\":\"usergroup1\"},{\"username\":\"user2\",\"password\":\"pwd2\",\"group\":\"usergroup2\"},{\"username\":\"user3\",\"password\":\"pwd3\",\"group\":\"usergroup3\"}]"
and the content of test.sh is the following
#!/bin/bash
#read the json
system_user="$1"
printf "$system_user"
accounts=($(jq -s ".[]" <<< $system_user))
printf "$accounts"
for account in "${accounts[#]}"
do
printf "\n\n$account\n\n"
done
the output of -> printf "$system_user" is
[{"username":"user1","password":"pwd1","group":"usergroup1"},{"username":"user2","password":"pwd2","group":"usergroup2"},{"username":"user3","password":"pwd3","group":"usergroup3"}]
but the output of -> printf "$accounts" is something like this
[
[
{
"username":
"user1"
etc. etc. one object for each token :-(
and so on, but what I was expecting is an array of three object (like you can test on jqplay.org)
{
"username": "user1",
"password": "pwd1",
"group": "usergroup1"
}
{
"username": "user2",
"password": "pwd2",
"group": "usergroup2"
}
{
"username": "user3",
"password": "pwd3",
"group": "usergroup3"
}
In this way I can make a foreach on ${accounts[#]}
What I'm doing wrong?
Thank you
With the -c option, you can print each JSON object on a single line, making it easier to populate the array you want.
$ readarray -t arr < <(jq -c '.[]' <<< "[{\"username\":\"user1\",\"password\":\"pwd1\",\"group\":\"usergroup1\"},{\"username\":\"user2\",\"password\":\"pwd2\",\"group\":\"usergroup2\"},{\"username\":\"user3\",\"password\":\"pwd3\",\"group\":\"usergroup3\"}]")
$ printf "Object: %s\n" "${arr[#]}"
Object: {"username":"user1","password":"pwd1","group":"usergroup1"}
Object: {"username":"user2","password":"pwd2","group":"usergroup2"}
Object: {"username":"user3","password":"pwd3","group":"usergroup3"}
You are interchanging bash arrays and JSON arrays. When you are creating accounts array, bash splits the elements per each whitespace. That's why you don't get what you expect. You can try the following:
declare -A accounts
while IFS="=" read -r key value
do
accounts[$key]="$value"
done < <(jq -r "to_entries|map(\"\(.key)=\(.value)\")|.[]" <<< $system_user)
for account in "${accounts[#]}"
do
printf "$account\n"
done
(stolen from here: https://stackoverflow.com/a/26717401/328977)
to get the following output:
{"username":"user1","password":"pwd1","group":"usergroup1"}
{"username":"user2","password":"pwd2","group":"usergroup2"}
{"username":"user3","password":"pwd3","group":"usergroup3"}