Read array of json from a file in shell scripting [duplicate] - json

This question already has answers here:
Increment variable value by 1 (shell programming)
(4 answers)
Closed 2 years ago.
I have a JSON file which is an array of Object. I want to parse the file and read the data from it using shell scripting.
file.json:-
[
{
"serviceName": "CreateAssociationService",
"targetJarFile": "../../create-association-service/target/create-association-service-1.0.0-aws.jar",
"function_name": "CreateAssociationService",
"handler": "org.qmetech.aws.handler.CreateAssociationFunctionHandler",
"method": "POST"
},
{
"serviceName": "CreateAssociationService1",
"targetJarFile": "../../create-association-service/target/create-association1-service-1.0.0-aws.jar",
"function_name": "CreateAssociationService",
"handler": "org.qmetech.aws.handler.DataHandler",
"method": "GET"
}
]
Does anyone know how can I do it?
As suggested :-
I am using the below command to read the file.
config_file="./file.json"
length=`jq '. | length' $config_file`
i=1
while (( $i <= $length ))
do
serviceName=`cat $config_file | jq '.[$i].serviceName'`
targetJarFile=`cat $config_file | jq '.[$i].targetJarFile'`
function_name=`cat $config_file | jq '.[$i].targetJarFile'`
handler= `cat $config_file | jq '.[$i].handler'`
method= `cat $config_file | jq '.[$i].method'`
echo "-service Name" "$serviceName"
echo "-targetJarFile" "$targetJarFile"
i++
done
But this code goes in an infinite loop with the below error:-
./deploy.sh: line 34: i++: command not found
jq: error: $i is not defined at <top-level>, line 1:
.[$i].serviceName
jq: 1 compile error
jq: error: $i is not defined at <top-level>, line 1:
.[$i].targetJarFile
jq: 1 compile error
jq: error: $i is not defined at <top-level>, line 1:
.[$i].targetJarFile
jq: 1 compile error
and the value of $i is also not evaluated properly.

How about using jq tool
cat file.json | jq '.[0]'
{
"serviceName": "CreateAssociationService",
"targetJarFile": "../../create-association-service/target/create-association-service-1.0.0-aws.jar",
"function_name": "CreateAssociationService",
"handler": "org.qmetech.aws.handler.CreateAssociationFunctionHandler",
"method": "POST"
}
cat file.json | jq '.[].serviceName'
"CreateAssociationService"
"CreateAssociationService1"

Given file.json has the values following script I tested and can get the values. Can you check this please. Should work.
count=$(jq -r '. | length' file.json)
for sequence in $(seq $count)
do
let indx=$sequence-1
serviceName=$(jq -r ".[$indx].serviceName" file.json)
targetJarFile=$(jq -r ".[$indx].targetJarFile" file.json)
# Display variables
echo $serviceName
echo $targetJarFile
echo "---"
done
output
CreateAssociationService
../../create-association-service/target/create-association-service-1.0.0-aws.jar
---
CreateAssociationService1
../../create-association-service/target/create-association1-service-1.0.0-aws.jar
---

Related

How to append json array using jq+shell in a loop?

I can create a json object with an array using jq like this
echo '{"input":{"names":[]}}' | jq --arg val "alice" '.input.names[0] += $val'| jq --arg val "bob" '.input.names[1] += $val'
which gives
{
"input": {
"names": [
"alice",
"bob"
]
}
}
Now I want to recreate this in a shell script where "names" come from a shell array
#!/bin/sh
names=( alice bob )
ITER=0
for i in "${names[#]}"
do
echo '{"input":{"names":[]}}' | jq --arg val "$i" '.input.names[$ITER] += $val'
echo ${I} ${ITER}
ITER=$(expr $ITER + 1)
done
but I run into
jq: error: $ITER is not defined at <top-level>, line 1:
.input.names[$ITER] += $val
jq: 1 compile error
0
jq: error: $ITER is not defined at <top-level>, line 1:
.input.names[$ITER] += $val
jq: 1 compile error
1
You could get rid of the shell loop and use one jq call:
names=( alice bob )
jq -n --arg names "${names[*]}" '{"input": {"names": ($names / " ") } }'
or
names=( alice bob )
printf '%s\0' "${names[#]}" |
jq -sR '(. / "\u0000") as $names | { "input": { "names": $names } }'
You don't capture the output of an iteration step, so you lose the output from the jq call and start all over again in the next iteration step.
Here's a bash solution which captures the output using $(...) and provides it for the next using <<<. (Note that there are better ways to solve this problem, e.g. without looping but by proving jq all elements to be added at once.)
json='{"input":{"names":[]}}'
for i in alice bob
do json="$(jq --arg val "$i" '.input.names += [$val]' <<< "$json")"
done
echo "$json"
{
"input": {
"names": [
"alice",
"bob"
]
}
}

Using jq with an unknown amount of arguments from shell script?

I'm about to lose my mind working with jq for the first time today. I've tried every ever so dirty way to make this work somehow. Let's get down to business:
I'm trying to further develop the pretty barebones API for Unifi Controllers for some use cases:
https://dl.ui.com/unifi/6.0.41/unifi_sh_api
Here's some mock up data (the real json data is thousands of lines):
{
"meta": {
"rc": "ok"
},
"data": [
{
"_id": "61a0da77f730e404af0edc3c",
"ip": "10.19.31.120",
"mac": "ff:ec:ff:89:ff:58",
"name": "Test-Accesspoint"
}
]
}
Here is my advancedapi.sh
#!/bin/bash
source unifiapi.sh
unifi_login 1>dev/null
if [ "$1" ];
then
command=$1
shift
else
echo "Please enter command"
exit
fi
#jq -r '.data[] | "\(.name),\(.mac)"'
assembleKeys(){
c="0"
seperator=";"
keys=$(for i in "$#"
do
c=$(expr $c + 1)
if [ $c -gt 1 ];
then
echo -n "${seperator}\(.${i})"
else
echo -n "\(.${i})"
fi
done
echo)
}
assembleJQ(){
c=0
jq=$(echo -en 'jq -r -j \x27.data[] | '
for i in "$#"
do
if [ "$c" != "0" ];
then
echo -en ",\x22 \x22,"
fi
c=1
echo -en ".$i"
done
echo -en ",\x22\\\n\x22\x27")
echo "$jq"
}
getDeviceKeys(){
assembleJQ "$#"
#unifi_list_devices | jq -r '.data[]' | jq -r "${keys}"
#export keys=mac
#export second=name
#unifi_list_devices | jq -r -j --arg KEYS "$keys" --arg SECOND "$second" '.data[] | .[$KEYS]," ",.[$SECOND],"\n"'
unifi_list_devices | $jq
#unifi_list_devices | jq -r -j '.data[] | .mac," ",.name,"\n"'
}
"$command" $#
Users should be able to call any function from the API with as many arguments as they want.
So basically this:
#export keys=mac
#export second=name
#unifi_list_devices | jq -r -j --arg KEYS "$keys" --arg SECOND "$second" '.data[] | .[$KEYS]," ",.[$SECOND],"\n"'
which works, but only with a limited number of arguments. I want to pass $# to jq.
I'm trying to get the name and mac-address of a device by calling:
./advancedapi.sh getDeviceKeys mac name
this should return the mac-address and name of the device. But it only gives me this:
jq -r -j '.data[] | .mac," ",.name,"\n"'
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'.data[]
jq: 1 compile error
The top-most line being the jq command, which works perfectly fine when called manually.
The assembleKeys function was my attempt at generating a string that looks like this:
jq -r '.data[] | "\(.name),\(.mac)"'
Can anyone explain to me, preferably in-depth, how to do this? I'm going insane here!
Thanks!
I want to pass $# to jq.
There are actually many ways to do this, but the simplest might be using the --args command-line option, as illustrated here:
File: check
#/bin/bash
echo "${#}"
jq -n '$ARGS.positional' --args "${#}"
Example run:
./check 1 2 3
1 2 3
[
"1",
"2",
"3"
]
Projection
Here's an example that might be closer to what you're looking for.
Using your sample JSON:
File: check2
#/bin/bash
jq -r '
def project($array):
. as $in | reduce $array[] as $k ([]; . + [$in[$k]]);
$ARGS.positional,
(.data[] | project($ARGS.positional))
| #csv
' input.json --args "${#}"
./check2 name mac
"name","mac"
"Test-Accesspoint","ff:ec:ff:89:ff:58"
There is error in the json (red highlighted commas)(that commas are not needed)
$
$ cat j.json | jq -r '.data[] | "\(.name)█\(.mac)█\(.ip)█\(._id)"'
Test-Accesspoint█ff:ec:ff:89:ff:58█10.19.31.120█61a0da77f730e404af0edc3c
$ cat j.json
{
"meta": {
"rc": "ok"
},
"data": [
{
"_id": "61a0da77f730e404af0edc3c",
"ip": "10.19.31.120",
"mac": "ff:ec:ff:89:ff:58",
"name": "Test-Accesspoint"
}
]
}
$
If the api have given this json you should probably edit it, before piping it into jq [maybe with sed or something like it]

How to make a new array and add json value in bash

for region in $(jq '.data | keys | .[]' <<< "$data"); do
value=$(jq -r ".data[$region]" <<< "$data");
deliveryRegionId=$(jq -r '.deliveryRegionId' <<< "$value");
json_template='{}';
json_data=$(jq --argjson deliveryRegionId "$deliveryRegionId" --arg deliverableDistance 5000 '.deliveryRegionId=$deliveryRegionId | .deliverableDistance=5000' <<<"$json_template"); echo $json_data;
requestArray=$(jq '. += [$json_data]' <<< $requestArray)
done;
As in the code above, I'm going to create a json value called json_data and add it to the array.
What should I do to make this work?
jq: error: $json_data is not defined at <top-level>, line 1:
. += [$json_data]
jq: 1 compile error
this is error
There's no jq variable named $json_data.
There is a shell variable named that, but you can't access another program's variables.
Provide the value via the environment
json_data="$json_data" jq '. += [ env.json_data ]' <<<"$requestArray"
Provide the value via the environment
export json_data
jq '. += [ env.json_data ]' <<<"$requestArray"
Provide the value as an argument
jq --arg json_data "$json_data" '. += [ $json_data ]' <<<"$requestArray"
There's no reason to use jq so many times! Your entire program can be replaced with this:
requestArray="$(
jq '.data | map( { deliveryRegionId, deliverableDistance: 5000 } )' \
<<<"$data"
)"
Demo on jqplay

how to use jq and bash to inject a files json array contents while appending

I have a .json file which I want to read, take all the contents.. and append it to a json string that's in a bash variable
input.json
[{
"maven": {
"coordinates": "somelib",
"repo": "somerepo"
}
},
{
"maven": {
"coordinates": "anotherlib",
"exclusions": ["exclude:this", "*:and-all-that"]
}
}]
OUR_BIG_JSON variable
{
"name": "",
"myarray": [
{
"generic": {
"package": "somepymodule"
}
},
{
"rcann": {
"package": "anothermodule==3.0.0"
}
}
],
and a json I want to append to.. it's residing in a variable so we must echo it as we use jq
command attempt
echo "starting..."
inputs_json=`cat input.json`
echo "$inputs_json"
echo "$OUR_BIG_JSON"
OUR_BIG_JSON=$(echo "${OUR_BIG_JSON}" |./jq '.myarray += [{"date":"today"}]') # This worked..
next line fails
OUR_BIG_JSON=$(echo "${OUR_BIG_JSON}" |./jq '.myarray += ' $inputs_json)
out comes errors when trying to take the $inputs_json contents and just pipe it into the variable.. basically append to the JSON in memory..
jq: error: syntax error, unexpected INVALID_CHARACTER (Windows cmd shell quoting issues?) at <top-level>, line 1: .myarray += \"$inputs_json\" jq: 1 compile error
Using the --argjson option to read in the contents of your shell variable enables you to use it as variable inside the jq filter, too. Your input file can normally be read in via parameter.
jq --argjson v "${OUR_BIG_JSON}" '…your filter here using $v…' input.json
For example:
jq --argjson v "${OUR_BIG_JSON}" '$v + {myarray: ($v.myarray + .)}' input.json
Or by making the input too a variable inside the jq filter
jq --argjson v "${OUR_BIG_JSON}" '. as $in | $v | .myarray += $in' input.json
With the -n option you could also reference the input using input:
jq --argjson v "${OUR_BIG_JSON}" -n '$v | .myarray += input' input.json
And there's also the options --argfile and --slurpfile to read in the input file as variable directly...
As you have tagged bash you could also do it the other way around and use a herestring <<< to read in the bash variable as input and then using either --argfile or --slurpfile to reference the file content through a variable. For instance:
jq --argfile in input.json '.myarray += $in' <<< "${OUR_BIG_JSON}"

JQ error: is not defined at <top-level> when trying to add values to jq template

I have a .jq template that I want to update values under the samples list, formatted as:
{
"test": "abc",
"p": "1",
"v": "1.0.0",
"samples": [
{
"uptime": $uptime,
"curr_connections": $curr_connections,
"listen_disabled_num": $listen_disabled_num,
"conn_yields": $conn_yields,
"cmd_get": $cmd_get,
"cmd_set": $cmd_set,
"bytes_read": $bytes_read,
"bytes_written": $bytes_writtem,
"get_hits": $get_hits,
"rejected_connections": $rejected_connections,
"limit_maxbytes": $limit_maxbytes,
"cmd_flush": $cmd_flush
}
]
}
My shell script to do this is below, I am basically running a command to pull some memcached output stats and want to insert some of the results into the jq template as key/values.
JQ=`cat template.jq`
SAMPLES=(uptime curr_connections listen_disabled_num conn_yields cmd_get cmd_set cmd_flush bytes_read bytes_written get_hits rejected_connections limit_maxbytes)
for metric in ${SAMPLES[*]}
do
KEY=$(echo stats | nc $HOST $PORT | grep $metric | awk '{print $2}')
VALUE=$(echo stats | nc $HOST $PORT | grep $metric | awk '{print $3}')
echo "Using KEY: $KEY with value: $VALUE"
jq -n --argjson $KEY $VALUE -f template.jq
done
Not sure if this is the best way to handle this scenario, but I am getting a ton of errors such as:
jq: error: conn_yields/0 is not defined at <top-level>, line 12:
"conn_yields": $conn_yields,
jq: error: cmd_get/0 is not defined at <top-level>, line 13:
"cmd_get": $cmd_get,
jq: error: cmd_set/0 is not defined at <top-level>, line 14:
"cmd_set": $cmd_set,
If you are going to invoke jq using -f template.jq, then each of the $-variables in template.jq will have to be set separately on the command-line, one by one. In your case, this does not look like a very happy option.
If you are stuck with template.jq as it is, then it will be hard slogging, though there are alternatives besides setting the $-variables on the command line.
Please see https://github.com/stedolan/jq/wiki/Cookbook#using-jq-as-a-template-engine in the jq Cookbook for an alternative to using $-variables. Consider for example the implications of this illustration of "destructuring":
jq -nc '{a:1,b:2} as {a: $a, b:$b} | [$a,$b]'
[1,2]
Another alternative
In your particular case, you could replace all the "$" characters in template.jq with ".", and then pass in a JSON object with the appropriate keys; e.g. change $uptime to .uptime, and then include a key/value pair for uptime.