I want to create a JSON from the list of servers that are stored inside a variable but I get a parse error.
How can I make this work? I think quoting each server might work but I can't do that since the data is retrieved from a CSV using curl.
server_list=$(curl http://localhost/api/server_list.csv | cut -d ',' -f2);
echo $server_list
host001 host002 host003 host004
echo $server_list | jq '.'
parse error: Invalid numeric literal at line 1, column 15
if that had worked then I could run the following command to create json.
echo $server_list | jq -r '.' | jq -s '{hosts:map({"hostid":.})}')
One invocation of jq is sufficient:
jq -R 'split(" ") | {hosts:map({"hostid":.})}' <<< "$server_list"
Figured it out.
echo $server_list | jq -R 'split (" ")' | jq '.[]' | jq -s '{hosts:map({"hostid":.})}'
If you use echo $server_list | jq -R '.' | jq -s '{hosts:map({"hostid":.})}' will produce:
{
"hosts": [
{
"hostid": "host001 host002 host003 host004"
}
]
}
Not sure if that is exactly what you are expecting for output, but hopefully helps.
With a little bit of additional manipulation within jq you can easily expand the output result.
echo $server_list | jq -s -R '{hosts:split(" ")|map(split("\n"))|map({"hostid": .[0]})}'
{
"hosts": [
{
"hostid": "host001"
},
{
"hostid": "host002"
},
{
"hostid": "host003"
},
{
"hostid": "host004"
}
]
}
Related
I'm about to lose my mind working with jq for the first time today. I've tried every ever so dirty way to make this work somehow. Let's get down to business:
I'm trying to further develop the pretty barebones API for Unifi Controllers for some use cases:
https://dl.ui.com/unifi/6.0.41/unifi_sh_api
Here's some mock up data (the real json data is thousands of lines):
{
"meta": {
"rc": "ok"
},
"data": [
{
"_id": "61a0da77f730e404af0edc3c",
"ip": "10.19.31.120",
"mac": "ff:ec:ff:89:ff:58",
"name": "Test-Accesspoint"
}
]
}
Here is my advancedapi.sh
#!/bin/bash
source unifiapi.sh
unifi_login 1>dev/null
if [ "$1" ];
then
command=$1
shift
else
echo "Please enter command"
exit
fi
#jq -r '.data[] | "\(.name),\(.mac)"'
assembleKeys(){
c="0"
seperator=";"
keys=$(for i in "$#"
do
c=$(expr $c + 1)
if [ $c -gt 1 ];
then
echo -n "${seperator}\(.${i})"
else
echo -n "\(.${i})"
fi
done
echo)
}
assembleJQ(){
c=0
jq=$(echo -en 'jq -r -j \x27.data[] | '
for i in "$#"
do
if [ "$c" != "0" ];
then
echo -en ",\x22 \x22,"
fi
c=1
echo -en ".$i"
done
echo -en ",\x22\\\n\x22\x27")
echo "$jq"
}
getDeviceKeys(){
assembleJQ "$#"
#unifi_list_devices | jq -r '.data[]' | jq -r "${keys}"
#export keys=mac
#export second=name
#unifi_list_devices | jq -r -j --arg KEYS "$keys" --arg SECOND "$second" '.data[] | .[$KEYS]," ",.[$SECOND],"\n"'
unifi_list_devices | $jq
#unifi_list_devices | jq -r -j '.data[] | .mac," ",.name,"\n"'
}
"$command" $#
Users should be able to call any function from the API with as many arguments as they want.
So basically this:
#export keys=mac
#export second=name
#unifi_list_devices | jq -r -j --arg KEYS "$keys" --arg SECOND "$second" '.data[] | .[$KEYS]," ",.[$SECOND],"\n"'
which works, but only with a limited number of arguments. I want to pass $# to jq.
I'm trying to get the name and mac-address of a device by calling:
./advancedapi.sh getDeviceKeys mac name
this should return the mac-address and name of the device. But it only gives me this:
jq -r -j '.data[] | .mac," ",.name,"\n"'
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'.data[]
jq: 1 compile error
The top-most line being the jq command, which works perfectly fine when called manually.
The assembleKeys function was my attempt at generating a string that looks like this:
jq -r '.data[] | "\(.name),\(.mac)"'
Can anyone explain to me, preferably in-depth, how to do this? I'm going insane here!
Thanks!
I want to pass $# to jq.
There are actually many ways to do this, but the simplest might be using the --args command-line option, as illustrated here:
File: check
#/bin/bash
echo "${#}"
jq -n '$ARGS.positional' --args "${#}"
Example run:
./check 1 2 3
1 2 3
[
"1",
"2",
"3"
]
Projection
Here's an example that might be closer to what you're looking for.
Using your sample JSON:
File: check2
#/bin/bash
jq -r '
def project($array):
. as $in | reduce $array[] as $k ([]; . + [$in[$k]]);
$ARGS.positional,
(.data[] | project($ARGS.positional))
| #csv
' input.json --args "${#}"
./check2 name mac
"name","mac"
"Test-Accesspoint","ff:ec:ff:89:ff:58"
There is error in the json (red highlighted commas)(that commas are not needed)
$
$ cat j.json | jq -r '.data[] | "\(.name)█\(.mac)█\(.ip)█\(._id)"'
Test-Accesspoint█ff:ec:ff:89:ff:58█10.19.31.120█61a0da77f730e404af0edc3c
$ cat j.json
{
"meta": {
"rc": "ok"
},
"data": [
{
"_id": "61a0da77f730e404af0edc3c",
"ip": "10.19.31.120",
"mac": "ff:ec:ff:89:ff:58",
"name": "Test-Accesspoint"
}
]
}
$
If the api have given this json you should probably edit it, before piping it into jq [maybe with sed or something like it]
I would like to print each path and value of a json file with included key values line by line. I would like the output to be comma delimited or at least very easy to cut and sort using Linux command line tools. Given the following json and jq, I have been given jq code which seems to do this for the test JSON, but I am not sure it works in all cases or is the proper approach.
Is there a function in jq which does this automatically? If not, is there a "most concise best way" to do it?
My wish would be something like:
$ cat short.json | jq -doit '.'
Reservations,0,Instances,0,ImageId,ami-a
Reservations,0,Instances,0,InstanceId,i-a
Reservations,0,Instances,0,InstanceType,t2.micro
Reservations,0,Instances,0,KeyName,ubuntu
Test JSON:
$ cat short.json | jq '.'
{
"Reservations": [
{
"Groups": [],
"Instances": [
{
"ImageId": "ami-a",
"InstanceId": "i-a",
"InstanceType": "t2.micro",
"KeyName": "ubuntu"
}
]
}
]
}
Code Recommended:
https://unix.stackexchange.com/questions/561460/how-to-print-path-and-key-values-of-json-file
Supporting:
https://unix.stackexchange.com/questions/515573/convert-json-file-to-a-key-path-with-the-resulting-value-at-the-end-of-each-k
JQ Code Too long and complicated!
jq -r '
paths(scalars) as $p
| [ ( [ $p[] | tostring ] | join(".") )
, ( getpath($p) | tojson )
]
| join(": ")
' short.json
Result:
Reservations.0.Instances.0.ImageId: "ami-a"
Reservations.0.Instances.0.InstanceId: "i-a"
Reservations.0.Instances.0.InstanceType: "t2.micro"
Reservations.0.Instances.0.KeyName: "ubuntu"
A simple jq query to achieve the requested format:
paths(scalars) as $p
| $p + [getpath($p)]
| join(",")
If your jq is ancient and you cannot upgrade, insert | map(tostring) before the last line above.
Output with the -r option
Reservations,0,Instances,0,ImageId,ami-a
Reservations,0,Instances,0,InstanceId,i-a
Reservations,0,Instances,0,InstanceType,t2.micro
Reservations,0,Instances,0,KeyName,ubuntu
Caveat
If a key or atomic value contains "," then of course using a comma may be inadvisable. For this reason, it might be preferable to use a character such as TAB that cannot appear in a JSON key or atomic value. Consider therefore using #tsv:
paths(scalars) as $p
| $p + [getpath($p)]
| #tsv
(The comment above about ancient versions of jq applies here too.)
Read it as a stream.
$ jq --stream -r 'select(.[1]|scalars!=null) | "\(.[0]|join(".")): \(.[1]|tojson)"' short.json
Use -c paths as follows:
cat short.json | jq -c paths | tr -d '[' | tr -d ']'
I am using jq-1.5-1-a5b5cbe
I would like to below code to output
KNOCK_KNOCK="Who is there?"
TEST_FILE='{
"KNOCK_KNOCK": "Who is there?"
}'
for s in $(echo $TEST_FILE | jq -r "to_entries|map(\"\
(.key)=\(.value|tostring)\")|.[]" ); do
echo $s
done
I got the loop from this post: Exporting JSON to environment variables
and cannot figure out how to modify to get my expected output. The problem seems to be the spaces in the .value
For the following test case I get the expected results:
TEST_FILE='{
"KNOCK_KNOCK": "Whoisthere?",
"POSTGRES_URI": "postgress://user:pass#testdb.com",
"WILL": "I.AM"
}'
for s in $(echo $TEST_FILE | jq -r "to_entries|map(\"\(.key)=\
(.value|tostring)\")|.[]" ); do
echo $s
done
KNOCK_KNOCK=Whoisthere?
POSTGRES_URI=postgress://user:pass#testdb.com
WILL=I.AM
I went with the following solution, which works for me, but the accepted answer is ok.
TEST_FILE='{
"KNOCK_KNOCK": "Who is there?"
}'
echo $TEST_FILE | sed 's_[{}]__'| sed 's_: _=_' | sed 's_ _export _'
One of many possibilities:
jq -r 'to_entries[] | [.key,.value] | join("=")' <<< "$TEST_FILE"
It would probably be advisable to use #sh as well:
$ echo $'{"KNOCK_KNOCK": "Who \'is\' there?"}'
| jq -r 'to_entries[] | [.key,(.value|#sh)] | join("=")'
KNOCK_KNOCK='Who '\''is'\'' there?'
$ KNOCK_KNOCK='Who '\''is'\'' there?'
$ echo "$KNOCK_KNOCK"
Who 'is' there?
$
Note also that if you are going to use $TEST_FILE, the double-quotes might be necessary.
You almost got it but you can do this without a loop with this line:
$ echo '{"KNOCK_KNOCK": "Who is there?"}' | jq -r 'to_entries[] | .key + "=\"" + (.value|tostring) + "\""'
KNOCK_KNOCK="Who is there?"
Explanation
to_entries transforms your object to an array of objects like this:
$ echo '{"KNOCK_KNOCK": "Who is there?"}' | jq -r 'to_entries'
[
{
"key": "KNOCK_KNOCK",
"value": "Who is there?"
}
]
You can then take each array element with the [] and use the key and value field to do simple string concatenation using the + operator.
input.json:-
{
"menu": {
"id": "file",
"value": "File",
"user": {
"address": "USA",
"email": "user#gmail.com"
}
}
}
Command:-
result=$(cat input.json | jq -r '.menu | keys[]')
Result:-
id
value
user
Loop through result:-
for type in "${result[#]}"
do
echo "--$type--"
done
Output:-
--id
value
user--
I want to do process the keys values in a loop. When I do the above, It result as a single string.
How can I do a loop with json keys result in bash script?
The canonical way :
file='input.json'
cat "$file" | jq -r '.menu | keys[]' |
while IFS= read -r value; do
echo "$value"
done
bash faq #1
But you seems to want an array, so the syntax is (missing parentheses) :
file='input.json'
result=( $(cat "$file" | jq -r '.menu | keys[]') )
for type in "${result[#]}"; do
echo "--$type--"
done
Output:
--id--
--value--
--user--
Using bash to just print an object keys from JSON data is redundant.
Jq is able to handle it by itself. Use the following simple jq solution:
jq -r '.menu | keys_unsorted[] | "--"+ . +"--"' input.json
The output:
--id--
--value--
--user--
I'm looking for a solution where I'm building out a JSON record and need to generate some text in JQ but pipe this text to an MD5 sum function and use it as a value for a key.
echo '{"first": "John", "last": "Big"}' | jq '. | { id: (.first + .last) | md5 }'
From looking at the manual and the GH issues I can't figure out how to do this since a function can't call out to a shell and there is not built in that provides a unique hash like functionality.
Edit
A better example what I'm looking for is this:
echo '{"first": "John", "last": "Big"}' | jq '. | {first, last, id: (.first + .last | md5) }'
to output:
{
"first": "John",
"last": "Big",
"id": "cda5c2dd89a0ab28a598a6b22e5b88ce"
}
Edit2
and a little more context. I'm creating NDJson files for use with esbulk. I need to generate a unique key for each record. Initially, I thought piping out to the shell would be the simplest solution so I could either use sha1sum or some other hash function easily, but that is looking more challenging than I thought.
A better example what I'm looking for is this:
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' | jq -c '.[] | {first, last, id: (.first + .last | md5) }'
to output:
{"first":"John","last":"Big","id":"cda5c2dd89a0ab28a598a6b22e5b88ce"}
{"first":"Justin","last":"Frozen","id":"af97f1bd8468e013c432208c32272668"}
Using tee allows a pipeline to be used, e.g.:
echo '{"first": "John", "last": "Big"}' |
tee >( jq -r '.first + .last' | md5 | jq -R '{id: .}') |
jq -s add
Output:
{
"first": "John",
"last": "Big",
"id": "cda5c2dd89a0ab28a598a6b22e5b88ce"
}
Edit2:
The following uses a while loop to iterate through the elements of the array, but it calls jq twice at each iteration. For a solution that does not call jq at all within the loop, see elsewhere on this page.
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' |
jq -c .[] |
while read -r line ; do
jq -r '[.[]]|add' <<< "$line" | md5 |
jq --argjson line "$line" -R '$line + {id: .}'
done
Looking around a little farther I ended up finding this: jq json parser hash the field value which was helpful in getting to my answer of:
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' > /tmp/testfile
jsonfile="/tmp/testfile"
jq -c .[] "$jsonfile" | while read -r jsonline ;
do
# quickly parse the JSON line and build the pre-ID out to get md5sum'd and then store that in a variable
id="$(jq -s -j -r '.[] | .first + .last' <<<"$jsonline" | md5sum | cut -d ' ' -f1)"
# using the stored md5sum'd ID we can use that as an argument for adding it to the existing jsonline
jq --arg id "$id" -s -c '.[] | .id = "\($id)"' <<<"$jsonline"
done
output
{"first":"John","last":"Big","id":"467ffeee8fea6aef01a6ffdcaf747782"}
{"first":"Justin","last":"Frozen","id":"fda76523d5259c0b586441dae7c2db85"}
jq + md5sum trick:
json_data='{"first": "John", "last": "Big"}'
jq -r '.first + .last| #sh' <<<"$json_data" | md5sum | cut -d' ' -f1 \
| jq -R --argjson data "$json_data" '$data + {id: .}'
Sample output:
{
"first": "John",
"last": "Big",
"id": "f9e1e448a766870605b863e23d3fdbd8"
}
Here is an efficient solution to the restated problem. There are altogether just two calls to jq, no matter the length of the array:
json='[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]'
echo "$json" |
jq -c '.[] | [.[]] | add' |
while read -r line ; do echo "$line" | md5 ; done |
jq -s -R --argjson json "$json" 'split("\n")
| map(select(length>0))
| . as $in
| reduce range(0;length) as $i ($json; .[$i].id = $in[$i])'
This produces an array. Just tack on |.[] at the end to produce a stream of the elements.
Or a bit more tersely, with the goal of emitting one object per line without calling jq within the loop:
jq -c --slurpfile md5 <(jq -c '.[] | [.[]] | add' <<< "$json" |
while read -r line ; do printf '"%s"' $(md5 <<< "$line" ) ; done) \
'[., $md5] | transpose[] | .[0] + {id: .[1]}' <<< "$json"
Distinct Digest for each Record
I need to generate a unique key for each record.
It would therefore make sense to compute the digest based on each entire JSON object (or more generally, the entire JSON value), i.e. use jq -c ‘.[]’
I adapted accepted answer's script to my case, posting it here, it could be useful to someone.
input.json:
{"date":100,"text":"some text","name":"april"}
{"date":200,"text":"a b c","name":"may"}
{"date":300,"text":"some text","name":"april"}
output.json:
{"date":100,"text":"some text","name":"april","id":"4d93d51945b88325c213640ef59fc50b"}
{"date":200,"text":"a b c","name":"may","id":"3da904d79fb03e6e3936ff2127039b1a"}
{"date":300,"text":"some text","name":"april","id":"4d93d51945b88325c213640ef59fc50b"}
The bash script to generate output.json:
cat input.json |
while read -r line ; do
jq -r '.text' <<< "$line" | md5 |
jq -c --argjson line "$line" -R '$line + {id: .}' \
>> output.json
done