JQ Loop over Bash array add elements - json

I do not seem to be able to find an answer, but have seen enough to know there is likely a better way of doing what I want to do.
Problem: I have a bash array. For each element in the bash array, I want to update a JSON array.
The JSON looks like the below. I am wanting to update the fruit array.
"foods": {
"perishable": {
"fruit": []
I'll get an array of length n, for example:
fruit_array=("banana" "orange")
It should look something like this:
"foods": {
"perishable": {
"fruit": [
{
"001": {
"002": "banana"
}
},
{
"001": {
"002": "orange"
}
}
]
Is there a nice way of doing this? At the moment I am trying the below:
#!/bin/bash
fruit_array=("banana" "orange")
for fruit in "${fruit_array[#]}"; do
jq \
--arg fruit $fruit \
'.foods.perishables.fruit += [{"001": {"002": $fruit}}]' \
template.json > template_with_fruit.json
done
This doesn't work for the obvious reason that the template is being re-read, but I have messed around to get it consuming the output of the previous iteration and nothing comes out at the end. I am only able to update the template once.
However, I know this seems a little dodgy and suspect there is a cleaner, more jq way.
A previous - aborted - attempt went something like this:
jq \
--argjson fruit "$(printf '{"001": {"002": "%s"}}\n' \"${fruit_array[#]}\" | jq -nR '[inputs]')" \
'.foods.perishables.fruit += $fruit' \
Which produced a escaped string which I couldn't do anything with, but at least hinted that there might be a neater solution to the standard bash loop.
I am missing something.
Any help would, as always, be appreciated.

JQ can do all that on its own; you don't need a loop or anything.
jq '.foods.perishable.fruit += (
$ARGS.positional
| map({"001": {"002": .}})
)' template.json --args "${fruit_array[#]}" >template_with_fruit.json

If you pass your array as a space delimited string, you can use JQ like so:
jq --arg fruits "$fruit_array" \
'.foods.perishable.fruit |= ($fruits | split(" ") | map({ "001": { "002": . } }))' input
{
"foods": {
"perishable": {
"fruit": [
{
"001": {
"002": "banana"
}
},
{
"001": {
"002": "orange"
}
}
]
}
}
}

Create whole json string at once with printf:
fruit_array=("banana" "orange")
printf -v fruits '{"001": {"002": %s}},' "${fruit_array[#]}"
$ echo $fruits
{"001": {"002": banana}},{"001": {"002": orange}},
Then add it to your template removing last comma:
$ echo ${fruits%,}
{"001": {"002": banana}},{"001": {"002": orange}}
"Numbers" could also be set like this:
fruit_array=("1 2 banana" "3 4 orange")
printf '{"%.3d": {"%.3d": %s}},' ${fruit_array[#]}
{"001": {"002": banana}},{"003": {"004": orange}},

Related

jq - iterate through dictionaries

My json knowledge is shaky, so pardon me if I use the wrong terminology.
I have input.txt which can be simplified down to this:
[
{
"foo1": "bar1",
"baz1": "fizz1"
},
{
"foo2": "bar2",
"baz2": "fizz2"
}
]
I want to iterate through each object via a loop, so I'm essentially hoping to tackle just the 1's first, then loop through the 2's, etc.
I thought it was something like:
jq 'keys[]' input.json | while read key ; do
echo "loop --$(jq "[$key]" input.json)"
done
but that's giving me
loop 0
loop 1
where I would expect to see (spacing here is optional, not sure how jq would parse it):
loop { "foo1": "bar1", "baz1": "fizz1" }
loop { "foo2": "bar2", "baz2": "fizz2" }
What am I missing?
No need to use bash, you can do this in jq itself:
jq -r 'keys[] as $k | "loop: \(.[$k])"' file.json
loop: {"foo1":"bar1","baz1":"fizz1"}
loop: {"foo2":"bar2","baz2":"fizz2"}
What about using the -c option:
$ jq -c '.[]' file | sed 's/^/loop /'
loop {"foo1":"bar1","baz1":"fizz1"}
loop {"foo2":"bar2","baz2":"fizz2"}
Assuming response is a variable containing your data :
echo "$response" | jq --raw-output '.[] | "loop " + tostring'
loop {"foo1":"bar1","baz1":"fizz1"}
loop {"foo2":"bar2","baz2":"fizz2"}
Hope it helps!

Building new JSON with JQ and bash

I am trying to create JSON from scratch using bash.
The final structure needs to be like:
{
"hosts": {
"a_hostname" : {
"ips" : [
1,
2,
3
]
},
{...}
}
}
First I'm creating an input file with the format:
hostname ["1.1.1.1","2.2.2.2"]
host-name2 ["3.3.3.3","4.4.4.4"]
This is being created by:
for host in $( ansible -i hosts all --list-hosts ) ; \
do echo -n "${host} " ; \
ansible -i hosts $host -m setup | sed '1c {' | jq -r -c '.ansible_facts.ansible_all_ipv4_addresses' ; \
done > hosts.txt
The key point here is that the IP list/array, is coming from a JSON file and being extracted by jq. This extraction outputs an already valid / quoted JSON array, but as a string in a txt file.
Next I'm using jq to parse the whole text file into the desired JSON:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]):
{ips:.[1]}
}
] | add }
' < ~/hosts.txt
This is almost correct, everything except for the IPs value which is treated as a string and quoted leading to:
{
"hosts": {
"hostname1": {
"ips": "[\"1.1.1.1\",\"2.2.2.2\"]"
},
"host-name2": {
"ips": "[\"3.3.3.3\",\"4.4.4.4\"]"
}
}
}
I'm now stuck at this final hurdle - how to insert the IPs without causing them to be quoted again.
Edit - quoting solved by using {ips: .[1] | fromjson }} instead of {ips:.[1]}.
However this was completely negated by #CharlesDuffy's help suggesting converting to TSV.
Original Q body:
So far I've got to
jq -n {hosts:{}} | \
for host in $( ansible -i hosts all --list-hosts ) ; \
do jq ".hosts += {$host:{}}" | \
jq ".hosts.$host += {ips:[1,2,3]}" ; \
done ;
([1,2,3] is actually coming from a subshell but including it seemed unnecessary as that part works, and made it harder to read)
This sort of works, but there seems to be 2 problems.
1) Final output only has a single host in it containg data from the first host in the list (this persists even if the second problem is bypassed):
{
"hosts": {
"host_1": {
"ips": [
1,
2,
3
]
}
}
}
2) One of the hostnames has a - in it, which causes syntax and compiler errors from jq. I'm stuck going around quote hell trying to get it to be interpreted but also quoted. Help!
Thanks for any input.
Let's say your input format is:
host_1 1 2 3
host_2 2 3 4
host-with-dashes 3 4 5
host-with-no-addresses
...re: edit specifying a different format: Add #tsv onto the JQ command producing the existing format to generate this one instead.
If you want to transform that to the format in question, it might look like:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]): .[1:]}
] | add
}' <input.txt
Which yields as output:
{
"hosts": {
"host_1": [
"1",
"2",
"3"
],
"host_2": [
"2",
"3",
"4"
],
"host-with-dashes": [
"3",
"4",
"5"
],
"host-with-no-addresses": []
}
}

how can i use variable in jq?(MacOS,shell)

I got a Json string as:
{
"id": 3397,
"title": "title_1"
}
{
"id": 3396,
"title": "title_2"
}
what I want to do is get every id in a loop,
I use the following code :
for (( i = 0; i < requestCount; i++ )); do
requestId=$(echo $jsonString[$i] | jq '.id')
echo requestId;
done
but it doesn't work, I think the way I use variable is wrong, I can't find anything useful here jq.
Let jq do the iterating. (That is, let jq do the iterating through the input stream of JSON objects.) For example:
$ jq .id <<< "$json" | while read id ; do echo "hello $id"; done
Output:
hello 3397
hello 3396
This way, you don't have to know how many JSON objects are in the input. You might want to use "read -r", or "IFS= read -r".
The alternatives are ugly and inefficient, e.g.:
$ for ((i=0;i<2;i++)) ; do jq -s --argjson i "$i" '.[$i].id' <<< "$json" ; done

Getting the contents of an unknown key

I am trying to get the value of the 'url' name which sits underneath a name that I do not know up front. e.g. it's not 'name' or 'size' - just a string that another tool generates - example "x1234" is not known to me by name:
"foo": {
"bar": {
"x1234": {
"url": "http://example.com"
}
}
}
so jq ".foo.bar" returns the "x1234" fragment but what I need is the "url" value underneath it. I've tried many things after reading the docs but I wasn't able to figure out the right syntax.
Can anyone tell me where I'm going wrong?
One approach is to use ... For example, provided the input is valid JSON:
$ jq '.. | .url? | select(.)' input.json
"http://example.com"
Or equivalently (and easier to type):
$ jq '.. | .url? // empty' input.json
Assuming foo and bar are known, you could just do
.foo.bar[].url
Let's say if bar is also unknown, then do the following
.foo[][].url
Here is a solution which uses tostream. If filter.jq contains
tostream
| select(length==2) as [$p,$v]
| if $p[-1] == "url" and ($v|endswith(".zip")) then $v else empty end
and if data.json contains (note outer { } added to make the example legal JSON and a second entry added to demonstrate excluding values not ending in .zip as asked in follow-up comment to peak's answer)
{
"foo": {
"bar": {
"x1234": {
"url": "http://example.com"
},
"x1234xxx": {
"url": "http://example.com/file.zip"
}
}
}
}
then the command
jq -M -f filter.jq data.json
produces
"http://example.com/file.zip"

How can I use jq to find all paths in a JSON object under which a value matches a given criteria?

Is there a way I could use jq to find all paths that hold a value that matches a given criteria?
For example, given the following JSON, I'd like to return all paths where the value of "age" is >35, regardless of the depth of the structure containing that field:
{
"springfield":{
"marge":{
"age":30
},
"homer":{
"age":40,
"job":"xyz"
}
},
"shelbyville":{
"zone1":{
"john":{
"age":10
}
},
"zone2":{
"mark":{
"age":50
}
}
},
"homeless1":{
"age":25
},
"homeless2":{
"age":60
}
}
So the execution would yield something like:
[
["springfield", "homer"],
["shelbyville", "zone2", "mark"],
["homeless2"]
]
This should do the trick:
[paths(.age?>35)]
The following has been tested with jq 1.4 and jq 1.5:
$ jq -M -c '(paths | select(.[length-1] =="age")) as $path
| if (getpath($path) > 35) then $path else empty end' ages.json
["springfield","homer","age"]
["shelbyville","zone2","mark","age"]
["homeless2","age"]
The following requires jq 1.5:
$ jq -c --stream 'select(length == 2 and .[0][-1] == "age"
and .[1] > 35)' ages.json
[["springfield","homer","age"],40]
[["shelbyville","zone2","mark","age"],50]
[["homeless2","age"],60]
These can both easily be adapted to produce similar output.