My json knowledge is shaky, so pardon me if I use the wrong terminology.
I have input.txt which can be simplified down to this:
[
{
"foo1": "bar1",
"baz1": "fizz1"
},
{
"foo2": "bar2",
"baz2": "fizz2"
}
]
I want to iterate through each object via a loop, so I'm essentially hoping to tackle just the 1's first, then loop through the 2's, etc.
I thought it was something like:
jq 'keys[]' input.json | while read key ; do
echo "loop --$(jq "[$key]" input.json)"
done
but that's giving me
loop 0
loop 1
where I would expect to see (spacing here is optional, not sure how jq would parse it):
loop { "foo1": "bar1", "baz1": "fizz1" }
loop { "foo2": "bar2", "baz2": "fizz2" }
What am I missing?
No need to use bash, you can do this in jq itself:
jq -r 'keys[] as $k | "loop: \(.[$k])"' file.json
loop: {"foo1":"bar1","baz1":"fizz1"}
loop: {"foo2":"bar2","baz2":"fizz2"}
What about using the -c option:
$ jq -c '.[]' file | sed 's/^/loop /'
loop {"foo1":"bar1","baz1":"fizz1"}
loop {"foo2":"bar2","baz2":"fizz2"}
Assuming response is a variable containing your data :
echo "$response" | jq --raw-output '.[] | "loop " + tostring'
loop {"foo1":"bar1","baz1":"fizz1"}
loop {"foo2":"bar2","baz2":"fizz2"}
Hope it helps!
Related
I have a json file I am parsing with jq. This is a sample of the file
[{
"key1":{...},
"key2":{...}
}]
[{
"key1":{...},
"key2":{...}
}]
...
each line is a list containing a json (which I know is not technically a json format but jq still works on such a file)
The below jq command works:
cat file.json | jq -r '.[] | [.key1,.key2]'
The above correctly shows:
[
<value_of_key1>,<value_of_key2>
]
[
<value_of_key1>,<value_of_key2>
]
However, I want .key1,.key2 to be dynamic since these keys can change. So I want to pass a variable to jq. Something like:
$KEYS=.key1,.key2
cat file.json | jq -r --arg var "$KEYS" '.[] | [$var]'
But the above is returning the keys themselves:
[
".key1,.key2"
]
[
".key1,.key2"
]
why is this happening? what is the correct command to make this happen?
This answer does not help me. I am not getting any errors as the OP in that question.
Fetching the value of a jq variable doesn't cause it to be executed as jq code.
Furthermore, jq lacks the facility to take a string, compile it as jq code, and evaluate the result. (This is commonly known as eval.)
So, short of a writing a jq parser and evaluator in jq, you will need to impose limits and/or accept a different format.
For example,
keys='[ [ "key1", "childkey" ], [ "key2", "childkey2" ] ]' # JSON
jq --argjson keys "$keys" '.[] | [ getpath( $keys[] ) ]' file.json
or
keys='key1.childkey,key2.childkey2'
jq --arg keys "$keys" '
( ( $keys / "," ) | map( . / "." ) ) as $keys |
.[] | [ getpath( $keys[] ) ]
' file.json
Suppose you have:
cat file
[{
"key1":1,
"key2":2
}]
[{
"key1":1,
"key2":2
}]
You can use a jq command like so:
jq '.[] | [.key1,.key2]' file
[
1,
2
]
[
1,
2
]
You can use -f to execute a filter from a file and nothing keeps you from creating the file separately from the shell variables.
Example:
keys=".key1"
echo ".[] | [${keys}]" >jqf
jq -f jqf file
[
1
]
[
1
]
Or just build the string directly into jq:
# note double " causing string interpolation
jq ".[] | [${keys}]" file
You can use --argjson option and destructuring.
file.json
[{"key1":{"a":1},"key2":{"b":2}}]
[{"key1":{"c":1},"key2":{"d":2}}]
$ in='["key1","key2"]' jq -c --argjson keys "$in" '$keys as [$key1,$key2] | .[] | [.[$key1,$key2]]' file.json
output:
[{"a":1},{"b":2}]
[{"c":1},{"d":2}]
Elaborating on ikegami's answer.
To start with here's my version of the answer:
$ in='key1.a,key2.b'; jq -c --arg keys "$in" '($keys/","|map(./".")) as $paths | .[] | [getpath($paths[])]' <<<$'[{"key1":{"a":1},"key2":{"b":2}}] [{"key1":{"a":3},"key2":{"b":4}}]'
This gives output
[1,2]
[3,4]
Let's try it.
We have input
[{"key1":{"a":1},"key2":{"b":2}}]
[{"key1":{"a":3},"key2":{"b":4}}]
And we want to construct array
[["key1","a"],["key2","b"]]
then use it on getpath(PATHS) builtin to extract values out of our input.
To start with we are given in shell variable with string value key1.a,key2.b. Let's call this $keys.
Then $keys/"," gives
["key1.a","key2.b"]
["key1.a","key2.b"]
After that $keys/","|map(./".") gives what we want.
[["key1","a"],["key2","b"]]
[["key1","a"],["key2","b"]]
Let's call this $paths.
Now if we do .[]|[getpath($paths[])] we get the values from our input equivalent to
[.[] | .key1.a, .key2.b]
which is
[1,2]
[3,4]
I am using jq like below to parse a json file -
The json file looks something like -
{
"values": [
{
"email": "user1#domain.com",
"id": "USER1_ID"
},{
"email": "user2#domain.com",
"id": "USER2_ID"
},
.
]
}
I am able to print/ iterate through the ids like below
for k in $(cat input.json | jq .values | jq .[].id); do
echo $k
done
This prints each individual id as expected.
However, what I want is to access both the email and the id in the loop.
I tried to assign values to SHELL variables like below -
emails=$(cat input.json | jq .values | jq .[].email)
ids=$(cat input.json | jq .values | jq .[].id)
This could work for the most part but ids can have spaces too which is breaking this.
I could essentially have to 2 for loops one for email and the other for id and assign values to arrays in the loop
i=0
for k in $(cat input.json | jq .values | jq .[].id); do
ids[$i]=$k
i=$(($i +1))
done
and
i=0
for k in $(cat input.json | jq .values | jq .[].email); do
emails[$i]=$k
i=$(($i +1))
done
Once I have both the values in arrays, I could parallely traverse both of them.
I am not a shell expert so I wanted to know if there is any slick way of doing this with fewer loops/ lines of code.
Thanks for any help!
You can output each email-ID pair as a comma separated list from JQ, and use read them into variables in a while loop like so:
while IFS=',' read -r email id; do
echo "$email"
echo "$id"
done <<EOF
$(jq -r '.values[] | "\(.email),\(.id)"' file)
EOF
Assuming the variables don't have embedded carriage returns, you can save yourself a lot of IFS grief by having separate read commands for each variable, e.g.:
jq -r '.values[] | (.email, .id)' input.json |
while IFS= read -r email ; do
IFS= read -r id
echo "email=$email and id=$id"
done
I have the following file
[
{
"id": 1,
"name": "Arthur",
"age": "21"
},
{
"id": 2,
"name": "Richard",
"age": "32"
}
]
To display login and id together, I am using the following command
$ jq '.[] | .name' test
"Arthur"
"Richard"
But when I put it in a shell script and try to assign it to a variable then the whole output is displayed on a single line like below
#!/bin/bash
names=$(jq '.[] | .name' test)
echo $names
$ ./script.sh
"Arthur" "Richard"
I want to break at every iteration similar to how it works on the command line.
Couple of issues in the information you have provided. The jq filter .[] | .login, .id will not produce the output as you claimed on jq-1.5. For your original JSON
{
"login":"dmaxfield",
"id":7449977
}
{
"login":"stackfield",
"id":2342323
}
It will produce four lines of output as,
jq -r '.login, .id' < json
dmaxfield
7449977
stackfield
2342323
If you are interested in storing them side by side, you need to do variable interpolation as
jq -r '"\(.login), \(.id)"' < json
dmaxfield, 7449977
stackfield, 2342323
And if you feel your output stored in a variable is not working. It is probably because of lack of double-quotes when you tried to print the variable in the shell.
jqOutput=$(jq -r '"\(.login), \(.id)"' < json)
printf "%s\n" "$jqOutput"
dmaxfield, 7449977
stackfield, 2342323
This way the embedded new lines in the command output are not swallowed by the shell.
For you updated JSON (totally new one compared to old one), all you need to do is
jqOutput=$(jq -r '.[] | .name' < json)
printf "%s\n" "$jqOutput"
Arthur
Richard
In case the .login or .id contains embedded spaces or other characters that might cause problems, a more robust approach is to ensure each JSON value is on a separate line. Consider, for example:
jq -c .login,.id input.json | while read login ; do read id; echo login="$login" and id="$id" ; done
login="dmaxfield" and id=7449977
login="stackfield" and id=2342323
I got a Json string as:
{
"id": 3397,
"title": "title_1"
}
{
"id": 3396,
"title": "title_2"
}
what I want to do is get every id in a loop,
I use the following code :
for (( i = 0; i < requestCount; i++ )); do
requestId=$(echo $jsonString[$i] | jq '.id')
echo requestId;
done
but it doesn't work, I think the way I use variable is wrong, I can't find anything useful here jq.
Let jq do the iterating. (That is, let jq do the iterating through the input stream of JSON objects.) For example:
$ jq .id <<< "$json" | while read id ; do echo "hello $id"; done
Output:
hello 3397
hello 3396
This way, you don't have to know how many JSON objects are in the input. You might want to use "read -r", or "IFS= read -r".
The alternatives are ugly and inefficient, e.g.:
$ for ((i=0;i<2;i++)) ; do jq -s --argjson i "$i" '.[$i].id' <<< "$json" ; done
I have a large JSON file that is an object of objects, which I would like to split into separate files name after object keys. Is it possible to achieve this using jq or any other off-the-shelf tools?
The original JSON is in the following format
{ "item1": {...}, "item2": {...}, ...}
Given this input I would like to produce files item1.json, item2.json etc.
This should give you a start:
for f in `cat input.json | jq -r 'keys[]'` ; do
cat input.json | jq ".$f" > $f.json
done
or when you insist on more bashy syntax like some seem to prefer:
for f in $(jq -r 'keys[]') ; do
jq ".[\"$f\"]" < input.json > "$f.json"
done < input.json
Here's a solution that requires only one call to jq:
jq -cr 'keys[] as $k | "\($k)\n\(.[$k])"' input.json |
while read -r key ; do
read -r item
printf "%s\n" "$item" > "/tmp/$key.json"
done
It might be faster to pipe the output of the jq command to awk, e.g.:
jq -cr 'keys[] as $k | "\($k)\t\(.[$k])"' input.json |
awk -F\\t '{ print $2 > "/tmp/" $1 ".json" }'
Of course, these approaches will need to be modified if the key names contain characters that cannot be used in filenames.
Is it possible to achieve this using jq or any other off-the-shelf tools?
It is. xidel can also do this very efficiently.
Let's assume 'input.json' :
{
"item1": {
"a": 1
},
"item2": {
"b": 2
},
"item3": {
"c": 3
}
}
Inefficient Bash method:
for f in $(xidel -s input.json -e '$json()'); do
xidel -s input.json -e '$json("'$f'")' > $f.json
done
For every object key another instance of xidel is called to parse the object. Especially when you have a very large JSON this is pretty slow.
Efficient file:write() method:
xidel -s input.json -e '
$json() ! file:write(
.||".json",
$json(.),
{"method":"json"}
)
'
One xidel call creates 'item{1,2,3}.json'. Their content is a compact/minified object, like {"a": 1} for 'item1.json'.
xidel -s input.json -e '
for $x in $json() return
file:write(
concat($x,".json"),
$json($x),
{
"method":"json",
"indent":true()
}
)
'
One xidel call creates 'item{1,2,3}.json'. Their content is a prettified object (because of {"indent":true()}), like...
{
"a": 1
}
...for 'item1.json'. Different query (for-loop), same result.
This method is multitudes faster!