I am trying to get the value of the 'url' name which sits underneath a name that I do not know up front. e.g. it's not 'name' or 'size' - just a string that another tool generates - example "x1234" is not known to me by name:
"foo": {
"bar": {
"x1234": {
"url": "http://example.com"
}
}
}
so jq ".foo.bar" returns the "x1234" fragment but what I need is the "url" value underneath it. I've tried many things after reading the docs but I wasn't able to figure out the right syntax.
Can anyone tell me where I'm going wrong?
One approach is to use ... For example, provided the input is valid JSON:
$ jq '.. | .url? | select(.)' input.json
"http://example.com"
Or equivalently (and easier to type):
$ jq '.. | .url? // empty' input.json
Assuming foo and bar are known, you could just do
.foo.bar[].url
Let's say if bar is also unknown, then do the following
.foo[][].url
Here is a solution which uses tostream. If filter.jq contains
tostream
| select(length==2) as [$p,$v]
| if $p[-1] == "url" and ($v|endswith(".zip")) then $v else empty end
and if data.json contains (note outer { } added to make the example legal JSON and a second entry added to demonstrate excluding values not ending in .zip as asked in follow-up comment to peak's answer)
{
"foo": {
"bar": {
"x1234": {
"url": "http://example.com"
},
"x1234xxx": {
"url": "http://example.com/file.zip"
}
}
}
}
then the command
jq -M -f filter.jq data.json
produces
"http://example.com/file.zip"
Related
I do not seem to be able to find an answer, but have seen enough to know there is likely a better way of doing what I want to do.
Problem: I have a bash array. For each element in the bash array, I want to update a JSON array.
The JSON looks like the below. I am wanting to update the fruit array.
"foods": {
"perishable": {
"fruit": []
I'll get an array of length n, for example:
fruit_array=("banana" "orange")
It should look something like this:
"foods": {
"perishable": {
"fruit": [
{
"001": {
"002": "banana"
}
},
{
"001": {
"002": "orange"
}
}
]
Is there a nice way of doing this? At the moment I am trying the below:
#!/bin/bash
fruit_array=("banana" "orange")
for fruit in "${fruit_array[#]}"; do
jq \
--arg fruit $fruit \
'.foods.perishables.fruit += [{"001": {"002": $fruit}}]' \
template.json > template_with_fruit.json
done
This doesn't work for the obvious reason that the template is being re-read, but I have messed around to get it consuming the output of the previous iteration and nothing comes out at the end. I am only able to update the template once.
However, I know this seems a little dodgy and suspect there is a cleaner, more jq way.
A previous - aborted - attempt went something like this:
jq \
--argjson fruit "$(printf '{"001": {"002": "%s"}}\n' \"${fruit_array[#]}\" | jq -nR '[inputs]')" \
'.foods.perishables.fruit += $fruit' \
Which produced a escaped string which I couldn't do anything with, but at least hinted that there might be a neater solution to the standard bash loop.
I am missing something.
Any help would, as always, be appreciated.
JQ can do all that on its own; you don't need a loop or anything.
jq '.foods.perishable.fruit += (
$ARGS.positional
| map({"001": {"002": .}})
)' template.json --args "${fruit_array[#]}" >template_with_fruit.json
If you pass your array as a space delimited string, you can use JQ like so:
jq --arg fruits "$fruit_array" \
'.foods.perishable.fruit |= ($fruits | split(" ") | map({ "001": { "002": . } }))' input
{
"foods": {
"perishable": {
"fruit": [
{
"001": {
"002": "banana"
}
},
{
"001": {
"002": "orange"
}
}
]
}
}
}
Create whole json string at once with printf:
fruit_array=("banana" "orange")
printf -v fruits '{"001": {"002": %s}},' "${fruit_array[#]}"
$ echo $fruits
{"001": {"002": banana}},{"001": {"002": orange}},
Then add it to your template removing last comma:
$ echo ${fruits%,}
{"001": {"002": banana}},{"001": {"002": orange}}
"Numbers" could also be set like this:
fruit_array=("1 2 banana" "3 4 orange")
printf '{"%.3d": {"%.3d": %s}},' ${fruit_array[#]}
{"001": {"002": banana}},{"003": {"004": orange}},
I'm trying to identify object elements which a key starting with #t. My goal is to delete them from the object all together.
Example Input
{
"process_state": {
"#user_id": "john smith",
"#t39ee396f50": 1,
"#t375b0311e8": 1,
"#t12dd92bf45": 1
}
}
Expected Output
{
"process_state": {
"#user_id": "john smith",
}
}
I've tried using jq and jtc to accomplish this and both seem to struggle with the leading # symbol. I'm assuming it's a format issue with my code. Can I use wildcards? I've tried a couple methods with no luck.
JQ
jq '. |= map(select(. | contains("#t") | not))'
Error: and string ("#t") cannot have their containment checked
JTC
<file jtc -w'<process_state.#t*>l:'
No error but #t* fields still exist in json object.
Any help is much appreciated.
We're going to use test("^#t") instead of contains("#t") since we want to check for a leading #t. But that's not why you are getting that error. You have the wrong object on the left-hand side of |=, and map doesn't do what you think for objects.
Two solutions:
Using filtering (like you were attempting):
.process_state |= with_entries(
select( .key | test("^#t") | not )
)
Demo on jqplay
Using deletion:
.process_state |= del(
.[
keys_unsorted[] |
select( test("^#t") )
]
)
Demo on jqplay
This is correct jtc usage:
bash $ <file jtc -w'<^#t>L:' -p
{
"process_state": {
"#user_id": "john smith"
}
}
bash $
I am trying to create JSON from scratch using bash.
The final structure needs to be like:
{
"hosts": {
"a_hostname" : {
"ips" : [
1,
2,
3
]
},
{...}
}
}
First I'm creating an input file with the format:
hostname ["1.1.1.1","2.2.2.2"]
host-name2 ["3.3.3.3","4.4.4.4"]
This is being created by:
for host in $( ansible -i hosts all --list-hosts ) ; \
do echo -n "${host} " ; \
ansible -i hosts $host -m setup | sed '1c {' | jq -r -c '.ansible_facts.ansible_all_ipv4_addresses' ; \
done > hosts.txt
The key point here is that the IP list/array, is coming from a JSON file and being extracted by jq. This extraction outputs an already valid / quoted JSON array, but as a string in a txt file.
Next I'm using jq to parse the whole text file into the desired JSON:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]):
{ips:.[1]}
}
] | add }
' < ~/hosts.txt
This is almost correct, everything except for the IPs value which is treated as a string and quoted leading to:
{
"hosts": {
"hostname1": {
"ips": "[\"1.1.1.1\",\"2.2.2.2\"]"
},
"host-name2": {
"ips": "[\"3.3.3.3\",\"4.4.4.4\"]"
}
}
}
I'm now stuck at this final hurdle - how to insert the IPs without causing them to be quoted again.
Edit - quoting solved by using {ips: .[1] | fromjson }} instead of {ips:.[1]}.
However this was completely negated by #CharlesDuffy's help suggesting converting to TSV.
Original Q body:
So far I've got to
jq -n {hosts:{}} | \
for host in $( ansible -i hosts all --list-hosts ) ; \
do jq ".hosts += {$host:{}}" | \
jq ".hosts.$host += {ips:[1,2,3]}" ; \
done ;
([1,2,3] is actually coming from a subshell but including it seemed unnecessary as that part works, and made it harder to read)
This sort of works, but there seems to be 2 problems.
1) Final output only has a single host in it containg data from the first host in the list (this persists even if the second problem is bypassed):
{
"hosts": {
"host_1": {
"ips": [
1,
2,
3
]
}
}
}
2) One of the hostnames has a - in it, which causes syntax and compiler errors from jq. I'm stuck going around quote hell trying to get it to be interpreted but also quoted. Help!
Thanks for any input.
Let's say your input format is:
host_1 1 2 3
host_2 2 3 4
host-with-dashes 3 4 5
host-with-no-addresses
...re: edit specifying a different format: Add #tsv onto the JQ command producing the existing format to generate this one instead.
If you want to transform that to the format in question, it might look like:
jq -Rn '
{ "hosts": [inputs |
split("\\s+"; "g") |
select(length > 0 and .[0] != "") |
{(.[0]): .[1:]}
] | add
}' <input.txt
Which yields as output:
{
"hosts": {
"host_1": [
"1",
"2",
"3"
],
"host_2": [
"2",
"3",
"4"
],
"host-with-dashes": [
"3",
"4",
"5"
],
"host-with-no-addresses": []
}
}
I have a json file in the form of a key-value map. For example:
{
"users":[
{
"key1":"user1",
"key2":"user2"
}
]
}
I have another json file. The values in the second file has to be replaced based on the keys in first file.
For example 2nd file is:
{
"info":
{
"users":["key1","key2","key3","key4"]
}
}
This second file should be replaced with
{
"info":
{
"users":["user1","user2","key3","key4"]
}
}
Because the value of key1 in first file is user1. this could be done with any python program, but I am learning jq and would like to try if it is possible with jq itself. I tried different combinations with reading file using slurpfile, then select & walk etc. But couldn't arrive at the required solution.
Any suggestions for the same will be appreciated.
Since .users[0] is a JSON dictionary, it would make sense to use it as such (e.g. for efficiency):
Invocation:
jq -c --slurpfile users users.json -f program.jq input.json
program.jq:
$users[0].users[0] as $dict
| .info.users |= map($dict[.] // .)
Output:
{"info":{"users":["user1","user2","key3","key4"]}}
Note: the above assumes that the dictionary contains no null or false values, or rather that any such values in the dictionary should be ignored. This avoids the double lookup that would otherwise be required. If this assumption is invalid, then a solution using has or in (e.g. as provided by RomanPerekhrest) would be appropriate.
Solution to supplemental problem
(See "comments".)
$users[0].users[0] as $dict
| second
| .info.users |= (map($dict[.] | select(. != null)))
sponge
It is highly inadvisable to use redirection to overwrite an input file.
If you have or can install sponge, then it would be far better to use it. For further details, see e.g. "What is jq's equivalent of sed -i?" in the jq FAQ.
jq solution:
jq --slurpfile users 1st.json '$users[0].users[0] as $users
| .info.users |= map(if in($users) then $users[.] else . end)' 2nd.json
The output:
{
"info": {
"users": [
"user1",
"user2",
"key3",
"key4"
]
}
}
Is there a way I could use jq to find all paths that hold a value that matches a given criteria?
For example, given the following JSON, I'd like to return all paths where the value of "age" is >35, regardless of the depth of the structure containing that field:
{
"springfield":{
"marge":{
"age":30
},
"homer":{
"age":40,
"job":"xyz"
}
},
"shelbyville":{
"zone1":{
"john":{
"age":10
}
},
"zone2":{
"mark":{
"age":50
}
}
},
"homeless1":{
"age":25
},
"homeless2":{
"age":60
}
}
So the execution would yield something like:
[
["springfield", "homer"],
["shelbyville", "zone2", "mark"],
["homeless2"]
]
This should do the trick:
[paths(.age?>35)]
The following has been tested with jq 1.4 and jq 1.5:
$ jq -M -c '(paths | select(.[length-1] =="age")) as $path
| if (getpath($path) > 35) then $path else empty end' ages.json
["springfield","homer","age"]
["shelbyville","zone2","mark","age"]
["homeless2","age"]
The following requires jq 1.5:
$ jq -c --stream 'select(length == 2 and .[0][-1] == "age"
and .[1] > 35)' ages.json
[["springfield","homer","age"],40]
[["shelbyville","zone2","mark","age"],50]
[["homeless2","age"],60]
These can both easily be adapted to produce similar output.