How or if is it possible to extract data from JSON structure specifying path as an argument to the command. I got this simple snippet taken from some larger script just for simplicity and have problems working it out:
#!/bin/bash
DATA='{
"level1": {
"level2": {
"level3": {
"foo": "bar",
"bar": "baz",
"baz": "bar"
}
}
}
}'
field="level1.level2.level3"
# does not work
jq -r --arg f ${field} '.[$f] | to_entries | .[] | "\"" + .key + "\"=\"" + .value + "\""' <<< ${DATA}
# works
jq -r --arg f ${field} '.level1.level2.level3 | to_entries | .[] | "\"" + .key + "\"=\"" + .value + "\""' <<< ${DATA}
# also works
field2="level3"
jq -r --arg f ${field2} '.level1.level2 | .[$f] | to_entries | .[] | "\"" + .key + "\"=\"" + .value + "\""' <<< ${DATA}
Gives the following output:
user#astra:~/test$ ./jqtest
jq: error (at <stdin>:12): null (null) has no keys
"foo"="bar"
"bar"="baz"
"baz"="bar"
"foo"="bar"
"bar"="baz"
"baz"="bar"
What am I doing wrong?
In this case .[$f] means "return the value associated with the key named level1.level2.level3". See:
$ jq --arg f 'level1.level2.level3' '.[$f]' <<< '{ "level1.level2.level3": "foo" }'
"foo"
Unless any of the path components contain a dot, splitting $f by dots and using the result as argument to getpath should work.
getpath($f / ".")
jq seems picky about paths in input arguments.
One solution could be to provide the path as a json array, and then use getpath to convert that to a path:
field='["level1", "level2", "level3"]'
jq -r --argjson f "$field" 'getpath($f)' <<< ${DATA}
Or for your specific question:
field='["level1", "level2", "level3"]'
jq -r --argjson f "${field}" 'getpath($f) | to_entries | .[] | "\"" + .key + "\"=\"" + .value + "\""' <<< ${DATA}
Related
I have this json stored in a variable foo
{
"app": "myApp",
"env": "test",
"tier": "frontend"
}
I have converted it into key=value pair using the following jq command:
jq -r 'to_entries[] | (.key) + "=" + .value' <(echo "$foo")
Output:
app = myApp
env = test
tier= frontend
I need to transform this to a comma seperated string of following string format and store in a variable:
Expected Output:
app=myApp,env=test,tier=frontend
What I tried:
jq -r 'to_entries[] | (.key) + "=" + .value| join(",")' <(echo "$foo")
But got error jq: error (at <stdin>:4): Cannot iterate over string ("app=myApp")
join() works on an array, but you're returning single items.
Replace [] with map() so we keep the array, then join will work just fine:
to_entries | map(.key + "=" + .value) | join(",")
JqPlay Demo
for region in $(jq '.data | keys | .[]' <<< "$data"); do
value=$(jq -r ".data[$region]" <<< "$data");
deliveryRegionId=$(jq -r '.deliveryRegionId' <<< "$value");
json_template='{}';
json_data=$(jq --argjson deliveryRegionId "$deliveryRegionId" --arg deliverableDistance 5000 '.deliveryRegionId=$deliveryRegionId | .deliverableDistance=5000' <<<"$json_template"); echo $json_data;
requestArray=$(jq '. += [$json_data]' <<< $requestArray)
done;
As in the code above, I'm going to create a json value called json_data and add it to the array.
What should I do to make this work?
jq: error: $json_data is not defined at <top-level>, line 1:
. += [$json_data]
jq: 1 compile error
this is error
There's no jq variable named $json_data.
There is a shell variable named that, but you can't access another program's variables.
Provide the value via the environment
json_data="$json_data" jq '. += [ env.json_data ]' <<<"$requestArray"
Provide the value via the environment
export json_data
jq '. += [ env.json_data ]' <<<"$requestArray"
Provide the value as an argument
jq --arg json_data "$json_data" '. += [ $json_data ]' <<<"$requestArray"
There's no reason to use jq so many times! Your entire program can be replaced with this:
requestArray="$(
jq '.data | map( { deliveryRegionId, deliverableDistance: 5000 } )' \
<<<"$data"
)"
Demo on jqplay
I want to use jq (or anything else when it's the wrong tool) to concatenate a json object like this:
{
"https://github.com": {
"user-one": {
"repository-one": "version-one",
"repository-two": "version-two"
},
"user-two": {
"repository-three": "version-three",
"repository-four": "version-four"
}
},
"https://gitlab.com": {
"user-three": {
"repository-five": "version-five",
"repository-six": "version-six"
},
"user-four": {
"repository-seven": "version-seven",
"repository-eight": "version-eight"
}
}
}
recursively to a bash string array like this:
(
"https://github.com/user-one/repository-one/archive/refs/heads/version-one.tar.gz"
"https://github.com/user-one/repository-two/archive/refs/heads/version-two.tar.gz"
"https://github.com/user-two/repository-three/archive/refs/heads/version-three.tar.gz"
"https://github.com/user-two/repository-four/archive/refs/heads/version-four.tar.gz"
"https://gitlab.com/user-three/repository-five/-/archive/version-five/repository-five-version-five.tar.gz"
"https://gitlab.com/user-three/repository-six/-/archive/version-six/repository-six-version-six.tar.gz"
"https://gitlab.com/user-four/repository-seven/-/archive/version-seven/repository-seven-version-seven.tar.gz"
"https://gitlab.com/user-four/repository-eight/-/archive/version-eight/repository-eight-version-eight.tar.gz"
)
for subsequent use in a loop.
for i in "${arr[#]}"
do
echo "$i"
done
Have no idea how to do that.
As you can see, the values must be handled differently depending on the object name.
"https://github.com" + "/" + $user_name + "/" + $repository_name + "/archive/refs/heads/" + $version + ".tar.gz"
"https://gitlab.com" + "/" + $user_name + "/" + $repository_name + "/-/archive/" + $version + "/" + $repository_name + "-" + $version + ".tar.gz"
Could anyone help?
Easily done.
First, let's focus on the jq code alone:
to_entries[] # split items into keys and values
| .key as $site # store first key in $site
| .value # code below deals with the value
| to_entries[] # split that value into keys and values
| .key as $user # store the key in $user
| .value # code below deals with the value
| to_entries[] # split that value into keys and values
| .key as $repository_name # store the key in $repository_name
| .value as $version # store the value in $version
| if $site == "https://github.com" then
"\($site)/\($user)/\($repository_name)/archive/refs/heads/\($version).tar.gz"
else
"\($site)/\($user)/\($repository_name)/-/archive/\($version)/\($repository_name)-\($version).tar.gz"
end
That generates a list of lines. Reading lines into a bash array looks like readarray -t arrayname < ...datasource...
Thus, using a process substitution to redirect jq's stdout as if it were a file:
readarray -t uris < <(jq -r '
to_entries[]
| .key as $site
| .value
| to_entries[]
| .key as $user
| .value
| to_entries[]
| .key as $repository_name
| .value as $version
| if $site == "https://github.com" then
"\($site)/\($user)/\($repository_name)/archive/refs/heads/\($version).tar.gz"
else
"\($site)/\($user)/\($repository_name)/-/archive/\($version)/\($repository_name)-\($version).tar.gz"
end
' <config.json
)
The basic task of generating the strings can be done efficiently and generically (i.e., without any limits on the depths of the basenames) using the jq filter:
paths(strings) as $p | $p + [getpath($p)] | join("/")
There are several ways to populate a bash array accordingly, but if you merely wish to iterate through the values, you could use a bash while loop, like so:
< input.json jq -r '
paths(strings) as $p | $p + [getpath($p)] | join("/")' |
while read -r line ; do
echo "$line"
done
You might also wish to consider using jq's #sh or #uri filter. For a jq urlencode function, see e.g.
https://rosettacode.org/wiki/URL_encoding#jq
(If the strings contain newlines or tabs, then the above would need to be tweaked accordingly.)
I have a json file A:
{
"remove" : ["foo", "bar"]
}
and a json file B:
{
"someDynamicKey" : {
"foo" : 1,
"xyz" : 2,
"bar" : "x"
}
}
I want to remove all keys in file B that match in "remove" section in file A.
The problem is that I don't know which keys would be in file A.
Expect:
{
"someDynamicKey" : {
"xyz" : 2
}
}
I was trying
jq --slurpfile a A.json '. as $b | reduce $a[] as $key ($b; . = del($b.$key))' B.json
and got error :
jq: error: syntax error, unexpected '$', expecting FORMAT or QQSTRING_START (Unix shell quoting issues?) at <top-level>, line 1:
. as $b | reduce $a[] as $key ($b; . = del($b.$key))
I am not sure how to do next or is it possible to achieve using jq? I appreciate any help!!
Keeping it simple:
jq --argfile A A.json '
$A.remove as $keys
| .someDynamicKey
|= with_entries( .key as $k
| if $keys | index($k)
then empty
else . end)' B.json
Or if you want a one-liner and don't like deprecated features and don't mind not:
jq --slurpfile A A.json '$A[0].remove as $keys | .someDynamicKey |= with_entries(select( .key as $k | $keys | index($k) | not))' B.json
I have two files
file1.json
{
"a": "x1",
"b": "y1",
"c": "z1"
}
file2.json
{
"a": "x2",
"b": "y2"
}
Since a & b already exists in file2, I want to output a new object that contains only c. The values don't really matter.
{
"c": "z1"
}
I tried
jq -s '.[0] | to_entries | map(select(.key | in(.[1]) | not)) | from_entries' temp1.json temp2.json
But I am getting the following error:
jq: error (at temp2.json:4): Cannot index string with number
The funny thing, when I try:
jq -s '.[0] | to_entries | map(select(.key | in({"a": "x2", "b": "y2"}) | not)) | from_entries' file1.json file2.json
I get the right output. So it just seems like jq is treating .[1] as an int? and not as a json object.
This "hack" is what did it for me so far. I am curious if there is anything better.
jq -s '. as $input | $input[0] | to_entries | map(select(.key | in($input[1]) | not)) | from_entries' temp1.json temp2.json
Here's a straightforward approach using reduce:
jq -n --argfile one file1.json --argfile two file2.json '
reduce ($two|keys_unsorted)[] as $k ($one; delpaths([[$k]]))'
And here is a more efficient filter that still uses delpaths:
$one | delpaths($two|keys_unsorted|map([.]))
Here's a generic function for performing object subtraction:
# remove from $x all the keys that occur in $y
def minus($x; $y):
((($x + $y)|to_entries) - ($y|to_entries))
| from_entries;
This can be used to solve the problem in a variety of ways, e.g. with the invocation:
jq -n -f program.jq file1.json file2.json
where program.jq contains the above def followed by:
minus(input; input)