I got a Json string as:
{
"id": 3397,
"title": "title_1"
}
{
"id": 3396,
"title": "title_2"
}
what I want to do is get every id in a loop,
I use the following code :
for (( i = 0; i < requestCount; i++ )); do
requestId=$(echo $jsonString[$i] | jq '.id')
echo requestId;
done
but it doesn't work, I think the way I use variable is wrong, I can't find anything useful here jq.
Let jq do the iterating. (That is, let jq do the iterating through the input stream of JSON objects.) For example:
$ jq .id <<< "$json" | while read id ; do echo "hello $id"; done
Output:
hello 3397
hello 3396
This way, you don't have to know how many JSON objects are in the input. You might want to use "read -r", or "IFS= read -r".
The alternatives are ugly and inefficient, e.g.:
$ for ((i=0;i<2;i++)) ; do jq -s --argjson i "$i" '.[$i].id' <<< "$json" ; done
Related
I do not seem to be able to find an answer, but have seen enough to know there is likely a better way of doing what I want to do.
Problem: I have a bash array. For each element in the bash array, I want to update a JSON array.
The JSON looks like the below. I am wanting to update the fruit array.
"foods": {
"perishable": {
"fruit": []
I'll get an array of length n, for example:
fruit_array=("banana" "orange")
It should look something like this:
"foods": {
"perishable": {
"fruit": [
{
"001": {
"002": "banana"
}
},
{
"001": {
"002": "orange"
}
}
]
Is there a nice way of doing this? At the moment I am trying the below:
#!/bin/bash
fruit_array=("banana" "orange")
for fruit in "${fruit_array[#]}"; do
jq \
--arg fruit $fruit \
'.foods.perishables.fruit += [{"001": {"002": $fruit}}]' \
template.json > template_with_fruit.json
done
This doesn't work for the obvious reason that the template is being re-read, but I have messed around to get it consuming the output of the previous iteration and nothing comes out at the end. I am only able to update the template once.
However, I know this seems a little dodgy and suspect there is a cleaner, more jq way.
A previous - aborted - attempt went something like this:
jq \
--argjson fruit "$(printf '{"001": {"002": "%s"}}\n' \"${fruit_array[#]}\" | jq -nR '[inputs]')" \
'.foods.perishables.fruit += $fruit' \
Which produced a escaped string which I couldn't do anything with, but at least hinted that there might be a neater solution to the standard bash loop.
I am missing something.
Any help would, as always, be appreciated.
JQ can do all that on its own; you don't need a loop or anything.
jq '.foods.perishable.fruit += (
$ARGS.positional
| map({"001": {"002": .}})
)' template.json --args "${fruit_array[#]}" >template_with_fruit.json
If you pass your array as a space delimited string, you can use JQ like so:
jq --arg fruits "$fruit_array" \
'.foods.perishable.fruit |= ($fruits | split(" ") | map({ "001": { "002": . } }))' input
{
"foods": {
"perishable": {
"fruit": [
{
"001": {
"002": "banana"
}
},
{
"001": {
"002": "orange"
}
}
]
}
}
}
Create whole json string at once with printf:
fruit_array=("banana" "orange")
printf -v fruits '{"001": {"002": %s}},' "${fruit_array[#]}"
$ echo $fruits
{"001": {"002": banana}},{"001": {"002": orange}},
Then add it to your template removing last comma:
$ echo ${fruits%,}
{"001": {"002": banana}},{"001": {"002": orange}}
"Numbers" could also be set like this:
fruit_array=("1 2 banana" "3 4 orange")
printf '{"%.3d": {"%.3d": %s}},' ${fruit_array[#]}
{"001": {"002": banana}},{"003": {"004": orange}},
I have a javascript file which prints a JSON array of objects:
// myfile.js output
[
{ "id": 1, "name": "blah blah", ... },
{ "id": 2, "name": "xxx", ... },
...
]
In my bash script, I want to iterate through each object.
I've tried following, but it doesn't work.
#!/bin/bash
output=$(myfile.js)
for row in $(echo ${output} | jq -c '.[]'); do
echo $row
done
You are trying to invoke myfile.js as a command. You need this:
output=$(cat myfile.js)
instead of this:
output=$(myfile.js)
But even then, your current approach isn't going to work well if the data has whitespace in it (which it does, based on the sample you posted). I suggest the following alternative:
jq -c '.[]' < myfile.js |
while read -r row
do
echo "$row"
done
Output:
{"id":1,"name":"blah blah"}
{"id":2,"name":"xxx"}
Edit:
If your data is arising from a previous process invocation, such as mongo in your case, you can pipe it directly to jq (to remain portable), like this:
mongo myfile.js |
jq -c '.[]' |
while read -r row
do
echo "$row"
done
How can I make jq -c '.[]' < (mongo myfile.js) work?
In a bash shell, you would write an expression along the following lines:
while read -r line ; do .... done < <(mongo myfile.js | jq -c .[])
Note that there are two occurrences of "<" in the above expression.
Also, the above assumes mongo is emitting valid JSON. If it emits //-style comments, those would have somehow to be removed.
Comparison with piping into while
If you use the idiom:
... | while read -r line ; do .... done
then the bindings of any variables in .... will be lost.
My json knowledge is shaky, so pardon me if I use the wrong terminology.
I have input.txt which can be simplified down to this:
[
{
"foo1": "bar1",
"baz1": "fizz1"
},
{
"foo2": "bar2",
"baz2": "fizz2"
}
]
I want to iterate through each object via a loop, so I'm essentially hoping to tackle just the 1's first, then loop through the 2's, etc.
I thought it was something like:
jq 'keys[]' input.json | while read key ; do
echo "loop --$(jq "[$key]" input.json)"
done
but that's giving me
loop 0
loop 1
where I would expect to see (spacing here is optional, not sure how jq would parse it):
loop { "foo1": "bar1", "baz1": "fizz1" }
loop { "foo2": "bar2", "baz2": "fizz2" }
What am I missing?
No need to use bash, you can do this in jq itself:
jq -r 'keys[] as $k | "loop: \(.[$k])"' file.json
loop: {"foo1":"bar1","baz1":"fizz1"}
loop: {"foo2":"bar2","baz2":"fizz2"}
What about using the -c option:
$ jq -c '.[]' file | sed 's/^/loop /'
loop {"foo1":"bar1","baz1":"fizz1"}
loop {"foo2":"bar2","baz2":"fizz2"}
Assuming response is a variable containing your data :
echo "$response" | jq --raw-output '.[] | "loop " + tostring'
loop {"foo1":"bar1","baz1":"fizz1"}
loop {"foo2":"bar2","baz2":"fizz2"}
Hope it helps!
I would like to convert an associative array in bash to a JSON hash/dict. I would prefer to use JQ to do this as it is already a dependency and I can rely on it to produce well formed json. Could someone demonstrate how to achieve this?
#!/bin/bash
declare -A dict=()
dict["foo"]=1
dict["bar"]=2
dict["baz"]=3
for i in "${!dict[#]}"
do
echo "key : $i"
echo "value: ${dict[$i]}"
done
echo 'desired output using jq: { "foo": 1, "bar": 2, "baz": 3 }'
There are many possibilities, but given that you already have written a bash for loop, you might like to begin with this variation of your script:
#!/bin/bash
# Requires bash with associative arrays
declare -A dict
dict["foo"]=1
dict["bar"]=2
dict["baz"]=3
for i in "${!dict[#]}"
do
echo "$i"
echo "${dict[$i]}"
done |
jq -n -R 'reduce inputs as $i ({}; . + { ($i): (input|(tonumber? // .)) })'
The result reflects the ordering of keys produced by the bash for loop:
{
"bar": 2,
"baz": 3,
"foo": 1
}
In general, the approach based on feeding jq the key-value pairs, with one key on a line followed by the corresponding value on the next line, has much to recommend it. A generic solution following this general scheme, but using NUL as the "line-end" character, is given below.
Keys and Values as JSON Entities
To make the above more generic, it would be better to present the keys and values as JSON entities. In the present case, we could write:
for i in "${!dict[#]}"
do
echo "\"$i\""
echo "${dict[$i]}"
done |
jq -n 'reduce inputs as $i ({}; . + { ($i): input })'
Other Variations
JSON keys must be JSON strings, so it may take some work to ensure that the desired mapping from bash keys to JSON keys is implemented. Similar remarks apply to the mapping from bash array values to JSON values. One way to handle arbitrary bash keys would be to let jq do the conversion:
printf "%s" "$i" | jq -Rs .
You could of course do the same thing with the bash array values, and let jq check whether the value can be converted to a number or to some other JSON type as desired (e.g. using fromjson? // .).
A Generic Solution
Here is a generic solution along the lines mentioned in the jq FAQ and advocated by #CharlesDuffy. It uses NUL as the delimiter when passing the bash keys and values to jq, and has the advantage of only requiring one call to jq. If desired, the filter fromjson? // . can be omitted or replaced by another one.
declare -A dict=( [$'foo\naha']=$'a\nb' [bar]=2 [baz]=$'{"x":0}' )
for key in "${!dict[#]}"; do
printf '%s\0%s\0' "$key" "${dict[$key]}"
done |
jq -Rs '
split("\u0000")
| . as $a
| reduce range(0; length/2) as $i
({}; . + {($a[2*$i]): ($a[2*$i + 1]|fromjson? // .)})'
Output:
{
"foo\naha": "a\nb",
"bar": 2,
"baz": {
"x": 0
}
}
This answer is from nico103 on freenode #jq:
#!/bin/bash
declare -A dict=()
dict["foo"]=1
dict["bar"]=2
dict["baz"]=3
assoc2json() {
declare -n v=$1
printf '%s\0' "${!v[#]}" "${v[#]}" |
jq -Rs 'split("\u0000") | . as $v | (length / 2) as $n | reduce range($n) as $idx ({}; .[$v[$idx]]=$v[$idx+$n])'
}
assoc2json dict
You can initialize a variable to an empty object {} and add the key/values {($key):$value} for each iteration, re-injecting the result in the same variable :
#!/bin/bash
declare -A dict=()
dict["foo"]=1
dict["bar"]=2
dict["baz"]=3
data='{}'
for i in "${!dict[#]}"
do
data=$(jq -n --arg data "$data" \
--arg key "$i" \
--arg value "${dict[$i]}" \
'$data | fromjson + { ($key) : ($value | tonumber) }')
done
echo "$data"
This has been posted, and credited to nico103 on IRC, which is to say, me.
The thing that scares me, naturally, is that these associative array keys and values need quoting. Here's a start that requires some additional work to dequote keys and values:
function assoc2json {
typeset -n v=$1
printf '%q\n' "${!v[#]}" "${v[#]}" |
jq -Rcn '[inputs] |
. as $v |
(length / 2) as $n |
reduce range($n) as $idx ({}; .[$v[$idx]]=$v[$idx+$n])'
}
$ assoc2json a
{"foo\\ bar":"1","b":"bar\\ baz\\\"\\{\\}\\[\\]","c":"$'a\\nb'","d":"1"}
$
So now all that's needed is a jq function that removes the quotes, which come in several flavors:
if the string starts with a single-quote (ksh) then it ends with a single quote and those need to be removed
if the string starts with a dollar sign and a single-quote and ends in a double-quote, then those need to be removed and internal backslash escapes need to be unescaped
else leave as-is
I leave this last iterm as an exercise for the reader.
I should note that I'm using printf here as the iterator!
bash 5.2 introduces the #k parameter transformation which, makes this much easier. Like:
$ declare -A dict=([foo]=1 [bar]=2 [baz]=3)
$ jq -n '[$ARGS.positional | _nwise(2) | {(.[0]): .[1]}] | add' --args "${dict[#]#k}"
{
"foo": "1",
"bar": "2",
"baz": "3"
}
I have the following file
[
{
"id": 1,
"name": "Arthur",
"age": "21"
},
{
"id": 2,
"name": "Richard",
"age": "32"
}
]
To display login and id together, I am using the following command
$ jq '.[] | .name' test
"Arthur"
"Richard"
But when I put it in a shell script and try to assign it to a variable then the whole output is displayed on a single line like below
#!/bin/bash
names=$(jq '.[] | .name' test)
echo $names
$ ./script.sh
"Arthur" "Richard"
I want to break at every iteration similar to how it works on the command line.
Couple of issues in the information you have provided. The jq filter .[] | .login, .id will not produce the output as you claimed on jq-1.5. For your original JSON
{
"login":"dmaxfield",
"id":7449977
}
{
"login":"stackfield",
"id":2342323
}
It will produce four lines of output as,
jq -r '.login, .id' < json
dmaxfield
7449977
stackfield
2342323
If you are interested in storing them side by side, you need to do variable interpolation as
jq -r '"\(.login), \(.id)"' < json
dmaxfield, 7449977
stackfield, 2342323
And if you feel your output stored in a variable is not working. It is probably because of lack of double-quotes when you tried to print the variable in the shell.
jqOutput=$(jq -r '"\(.login), \(.id)"' < json)
printf "%s\n" "$jqOutput"
dmaxfield, 7449977
stackfield, 2342323
This way the embedded new lines in the command output are not swallowed by the shell.
For you updated JSON (totally new one compared to old one), all you need to do is
jqOutput=$(jq -r '.[] | .name' < json)
printf "%s\n" "$jqOutput"
Arthur
Richard
In case the .login or .id contains embedded spaces or other characters that might cause problems, a more robust approach is to ensure each JSON value is on a separate line. Consider, for example:
jq -c .login,.id input.json | while read login ; do read id; echo login="$login" and id="$id" ; done
login="dmaxfield" and id=7449977
login="stackfield" and id=2342323