input.json:-
{
"menu": {
"id": "file",
"value": "File",
"user": {
"address": "USA",
"email": "user#gmail.com"
}
}
}
Command:-
result=$(cat input.json | jq -r '.menu | keys[]')
Result:-
id
value
user
Loop through result:-
for type in "${result[#]}"
do
echo "--$type--"
done
Output:-
--id
value
user--
I want to do process the keys values in a loop. When I do the above, It result as a single string.
How can I do a loop with json keys result in bash script?
The canonical way :
file='input.json'
cat "$file" | jq -r '.menu | keys[]' |
while IFS= read -r value; do
echo "$value"
done
bash faq #1
But you seems to want an array, so the syntax is (missing parentheses) :
file='input.json'
result=( $(cat "$file" | jq -r '.menu | keys[]') )
for type in "${result[#]}"; do
echo "--$type--"
done
Output:
--id--
--value--
--user--
Using bash to just print an object keys from JSON data is redundant.
Jq is able to handle it by itself. Use the following simple jq solution:
jq -r '.menu | keys_unsorted[] | "--"+ . +"--"' input.json
The output:
--id--
--value--
--user--
Related
I am trying to get certain value in the json with some for each and if condition.
code looks like the following
for i in "$(jq -r '.value[]' test.json)"; do
result=$(echo $i | jq -r .result)
if [ "$result" == "succeeded" ]
then
name=$(echo $i | jq -r .agent.name)
echo "$name"
fi
done
json file - test.json
{
"count":3,
"value":[
{
"location":"CA",
"result":"failed",
"agent":{
"id":97833,
"name":"Brad"
},
"priority":0
},
{
"location":"TX",
"result":"failed",
"agent":{i
"id":15232,
"name":"Tom"
},
"priority":0
},
{
"location":"CO",
"result":"succeeded",
"agent":{
"id":13412,
"name":"John"
},
"priority":0
}
]
}
I am trying to loop through the json file using jq to get the name of the agent if the result was "succeeded". but the result i get from result=$(echo $i | jq -r .result) seems like it's returning string array #("failed","failed","succeeded").
========================update==============================
jq ' # get the name of the agent if the result was "succeeded".
.value[]
| select(.result == "succeeded")
| var1=.agent.name
| echo $var1
' test.json
if I want to save the .agent.name into a variable and use it with a different command, what do I have to do?
There is no need to use a shell loop.
With your test.json,
jq ' # get the name of the agent if the result was "succeeded".
.value[]
| select(.result == "succeeded")
| .agent.name
' test.json
produces:
"John"
I want to create a JSON from the list of servers that are stored inside a variable but I get a parse error.
How can I make this work? I think quoting each server might work but I can't do that since the data is retrieved from a CSV using curl.
server_list=$(curl http://localhost/api/server_list.csv | cut -d ',' -f2);
echo $server_list
host001 host002 host003 host004
echo $server_list | jq '.'
parse error: Invalid numeric literal at line 1, column 15
if that had worked then I could run the following command to create json.
echo $server_list | jq -r '.' | jq -s '{hosts:map({"hostid":.})}')
One invocation of jq is sufficient:
jq -R 'split(" ") | {hosts:map({"hostid":.})}' <<< "$server_list"
Figured it out.
echo $server_list | jq -R 'split (" ")' | jq '.[]' | jq -s '{hosts:map({"hostid":.})}'
If you use echo $server_list | jq -R '.' | jq -s '{hosts:map({"hostid":.})}' will produce:
{
"hosts": [
{
"hostid": "host001 host002 host003 host004"
}
]
}
Not sure if that is exactly what you are expecting for output, but hopefully helps.
With a little bit of additional manipulation within jq you can easily expand the output result.
echo $server_list | jq -s -R '{hosts:split(" ")|map(split("\n"))|map({"hostid": .[0]})}'
{
"hosts": [
{
"hostid": "host001"
},
{
"hostid": "host002"
},
{
"hostid": "host003"
},
{
"hostid": "host004"
}
]
}
I am using jq like below to parse a json file -
The json file looks something like -
{
"values": [
{
"email": "user1#domain.com",
"id": "USER1_ID"
},{
"email": "user2#domain.com",
"id": "USER2_ID"
},
.
]
}
I am able to print/ iterate through the ids like below
for k in $(cat input.json | jq .values | jq .[].id); do
echo $k
done
This prints each individual id as expected.
However, what I want is to access both the email and the id in the loop.
I tried to assign values to SHELL variables like below -
emails=$(cat input.json | jq .values | jq .[].email)
ids=$(cat input.json | jq .values | jq .[].id)
This could work for the most part but ids can have spaces too which is breaking this.
I could essentially have to 2 for loops one for email and the other for id and assign values to arrays in the loop
i=0
for k in $(cat input.json | jq .values | jq .[].id); do
ids[$i]=$k
i=$(($i +1))
done
and
i=0
for k in $(cat input.json | jq .values | jq .[].email); do
emails[$i]=$k
i=$(($i +1))
done
Once I have both the values in arrays, I could parallely traverse both of them.
I am not a shell expert so I wanted to know if there is any slick way of doing this with fewer loops/ lines of code.
Thanks for any help!
You can output each email-ID pair as a comma separated list from JQ, and use read them into variables in a while loop like so:
while IFS=',' read -r email id; do
echo "$email"
echo "$id"
done <<EOF
$(jq -r '.values[] | "\(.email),\(.id)"' file)
EOF
Assuming the variables don't have embedded carriage returns, you can save yourself a lot of IFS grief by having separate read commands for each variable, e.g.:
jq -r '.values[] | (.email, .id)' input.json |
while IFS= read -r email ; do
IFS= read -r id
echo "email=$email and id=$id"
done
I would like to below code to output
KNOCK_KNOCK="Who is there?"
TEST_FILE='{
"KNOCK_KNOCK": "Who is there?"
}'
for s in $(echo $TEST_FILE | jq -r "to_entries|map(\"\
(.key)=\(.value|tostring)\")|.[]" ); do
echo $s
done
I got the loop from this post: Exporting JSON to environment variables
and cannot figure out how to modify to get my expected output. The problem seems to be the spaces in the .value
For the following test case I get the expected results:
TEST_FILE='{
"KNOCK_KNOCK": "Whoisthere?",
"POSTGRES_URI": "postgress://user:pass#testdb.com",
"WILL": "I.AM"
}'
for s in $(echo $TEST_FILE | jq -r "to_entries|map(\"\(.key)=\
(.value|tostring)\")|.[]" ); do
echo $s
done
KNOCK_KNOCK=Whoisthere?
POSTGRES_URI=postgress://user:pass#testdb.com
WILL=I.AM
I went with the following solution, which works for me, but the accepted answer is ok.
TEST_FILE='{
"KNOCK_KNOCK": "Who is there?"
}'
echo $TEST_FILE | sed 's_[{}]__'| sed 's_: _=_' | sed 's_ _export _'
One of many possibilities:
jq -r 'to_entries[] | [.key,.value] | join("=")' <<< "$TEST_FILE"
It would probably be advisable to use #sh as well:
$ echo $'{"KNOCK_KNOCK": "Who \'is\' there?"}'
| jq -r 'to_entries[] | [.key,(.value|#sh)] | join("=")'
KNOCK_KNOCK='Who '\''is'\'' there?'
$ KNOCK_KNOCK='Who '\''is'\'' there?'
$ echo "$KNOCK_KNOCK"
Who 'is' there?
$
Note also that if you are going to use $TEST_FILE, the double-quotes might be necessary.
You almost got it but you can do this without a loop with this line:
$ echo '{"KNOCK_KNOCK": "Who is there?"}' | jq -r 'to_entries[] | .key + "=\"" + (.value|tostring) + "\""'
KNOCK_KNOCK="Who is there?"
Explanation
to_entries transforms your object to an array of objects like this:
$ echo '{"KNOCK_KNOCK": "Who is there?"}' | jq -r 'to_entries'
[
{
"key": "KNOCK_KNOCK",
"value": "Who is there?"
}
]
You can then take each array element with the [] and use the key and value field to do simple string concatenation using the + operator.
I'm looking for a solution where I'm building out a JSON record and need to generate some text in JQ but pipe this text to an MD5 sum function and use it as a value for a key.
echo '{"first": "John", "last": "Big"}' | jq '. | { id: (.first + .last) | md5 }'
From looking at the manual and the GH issues I can't figure out how to do this since a function can't call out to a shell and there is not built in that provides a unique hash like functionality.
Edit
A better example what I'm looking for is this:
echo '{"first": "John", "last": "Big"}' | jq '. | {first, last, id: (.first + .last | md5) }'
to output:
{
"first": "John",
"last": "Big",
"id": "cda5c2dd89a0ab28a598a6b22e5b88ce"
}
Edit2
and a little more context. I'm creating NDJson files for use with esbulk. I need to generate a unique key for each record. Initially, I thought piping out to the shell would be the simplest solution so I could either use sha1sum or some other hash function easily, but that is looking more challenging than I thought.
A better example what I'm looking for is this:
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' | jq -c '.[] | {first, last, id: (.first + .last | md5) }'
to output:
{"first":"John","last":"Big","id":"cda5c2dd89a0ab28a598a6b22e5b88ce"}
{"first":"Justin","last":"Frozen","id":"af97f1bd8468e013c432208c32272668"}
Using tee allows a pipeline to be used, e.g.:
echo '{"first": "John", "last": "Big"}' |
tee >( jq -r '.first + .last' | md5 | jq -R '{id: .}') |
jq -s add
Output:
{
"first": "John",
"last": "Big",
"id": "cda5c2dd89a0ab28a598a6b22e5b88ce"
}
Edit2:
The following uses a while loop to iterate through the elements of the array, but it calls jq twice at each iteration. For a solution that does not call jq at all within the loop, see elsewhere on this page.
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' |
jq -c .[] |
while read -r line ; do
jq -r '[.[]]|add' <<< "$line" | md5 |
jq --argjson line "$line" -R '$line + {id: .}'
done
Looking around a little farther I ended up finding this: jq json parser hash the field value which was helpful in getting to my answer of:
echo '[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]' > /tmp/testfile
jsonfile="/tmp/testfile"
jq -c .[] "$jsonfile" | while read -r jsonline ;
do
# quickly parse the JSON line and build the pre-ID out to get md5sum'd and then store that in a variable
id="$(jq -s -j -r '.[] | .first + .last' <<<"$jsonline" | md5sum | cut -d ' ' -f1)"
# using the stored md5sum'd ID we can use that as an argument for adding it to the existing jsonline
jq --arg id "$id" -s -c '.[] | .id = "\($id)"' <<<"$jsonline"
done
output
{"first":"John","last":"Big","id":"467ffeee8fea6aef01a6ffdcaf747782"}
{"first":"Justin","last":"Frozen","id":"fda76523d5259c0b586441dae7c2db85"}
jq + md5sum trick:
json_data='{"first": "John", "last": "Big"}'
jq -r '.first + .last| #sh' <<<"$json_data" | md5sum | cut -d' ' -f1 \
| jq -R --argjson data "$json_data" '$data + {id: .}'
Sample output:
{
"first": "John",
"last": "Big",
"id": "f9e1e448a766870605b863e23d3fdbd8"
}
Here is an efficient solution to the restated problem. There are altogether just two calls to jq, no matter the length of the array:
json='[{"first": "John", "last": "Big"}, {"first": "Justin", "last": "Frozen"}]'
echo "$json" |
jq -c '.[] | [.[]] | add' |
while read -r line ; do echo "$line" | md5 ; done |
jq -s -R --argjson json "$json" 'split("\n")
| map(select(length>0))
| . as $in
| reduce range(0;length) as $i ($json; .[$i].id = $in[$i])'
This produces an array. Just tack on |.[] at the end to produce a stream of the elements.
Or a bit more tersely, with the goal of emitting one object per line without calling jq within the loop:
jq -c --slurpfile md5 <(jq -c '.[] | [.[]] | add' <<< "$json" |
while read -r line ; do printf '"%s"' $(md5 <<< "$line" ) ; done) \
'[., $md5] | transpose[] | .[0] + {id: .[1]}' <<< "$json"
Distinct Digest for each Record
I need to generate a unique key for each record.
It would therefore make sense to compute the digest based on each entire JSON object (or more generally, the entire JSON value), i.e. use jq -c ‘.[]’
I adapted accepted answer's script to my case, posting it here, it could be useful to someone.
input.json:
{"date":100,"text":"some text","name":"april"}
{"date":200,"text":"a b c","name":"may"}
{"date":300,"text":"some text","name":"april"}
output.json:
{"date":100,"text":"some text","name":"april","id":"4d93d51945b88325c213640ef59fc50b"}
{"date":200,"text":"a b c","name":"may","id":"3da904d79fb03e6e3936ff2127039b1a"}
{"date":300,"text":"some text","name":"april","id":"4d93d51945b88325c213640ef59fc50b"}
The bash script to generate output.json:
cat input.json |
while read -r line ; do
jq -r '.text' <<< "$line" | md5 |
jq -c --argjson line "$line" -R '$line + {id: .}' \
>> output.json
done