I am trying to build a json with jq with --arg arguments however I'd like for the json not to be able to have a condition if the variable is empty.
An example, if I run the following command
jq -n --arg myvar "${SOMEVAR}" '{ $myvar}'
I'd like the json in that case to be {} if myvar happens to be empty (Because the variable ${SOMEVAR} does not exist) and not { "myvar": "" } which is what I get by just running the command above.
Is there any way to achieve this through some sort of condition?
UPDATE:
Some more details about the use case
I want to build a json based on several environment variables but only include the variables that have a value.
Something like
{"varA": "value", "varB": "value"}
But only include varA if its value is defined and so on. The issue now is that if value is not defined, the property varA will still exist with an empty value and because of the multiple argument/variable nature, using an if/else to build the entire json as suggested will lead to a huge amount of conditions to cover for every possible combination of variables not existing
Suppose you have a template of variable names, in the form of an object as you have suggested you want:
{a, b, c}
Suppose also (for the sake of illustration) that you want to pull in the corresponding values from *ix environment variables. Then you just need to adjust the template, which can be done using this filter:
def adjust: with_entries( env[.key] as $v | select($v != null) | .value = $v );
Example:
Assuming the above filter, together with the following line, is in a file named adjust.jq:
{a,b,c} | adjust
then:
$ export a=123
$ jq -n -f -c adjust.jq
{"a":"123"}
You can use an if/else construct:
jq -n --arg myvar "${SOMEVAR}" 'if ($myvar|length > 0) then {$myvar} else {} end'
It's still not clear where the variable-value pairs are coming from, so maybe it would be simplest to construct the object containing the mapping before invoking jq, and then passing it in using the --argjson or --argfile option?
Related
I have the following JSON structure:
{
"host1": "$PROJECT1",
"host2": "$PROJECT2",
"host3" : "xyz",
"host4" : "$PROJECT4"
}
And the following environment variables in the shell:
PROJECT1="randomtext1"
PROJECT2="randomtext2"
PROJECT4="randomtext3"
I want to check the values for each key, if they have a "$" character in them, replace them with their respective environment variable(which is already present in the shell) so that my JSON template is rendered with the correct environment variables.
I can use the --args option of jq but there are quite a lot of variables in my actual JSON template that I want to render.
I have been trying the following:
jq 'with_entries(.values as v | env.$v)
Basically making each value as a variable, then updating its value with the variable from the env object but seems like I am missing out on some understanding. Is there a straightforward way of doing this?
EDIT
Thanks to the answers on this question, I was able to achieve my larger goal for a part of which this question was asked
iterating over each value in an object,
checking its value,
if it's a string and starts with the character "$"
use the value to update it with an environment variable of the same name .
if it's an array
use the value to retrieve an environment variable of the same name
split the string with "," as delimiter, which returns an array of strings
Update the value with the array of strings
jq 'with_entries(.value |= (if (type=="array") then (env[.[0][1:]] | split(",")) elif (type=="string" and startswith("$")) then (env[.[1:]]) else . end))'
You need to export the Bash variables to be seen by jq:
export PROJECT1="randomtext1"
export PROJECT2="randomtext2"
export PROJECT4="randomtext3"
Then you can go with:
jq -n 'with_entries((.value | select(startswith("$"))) |= env[.[1:]])'
and get:
{
"host1": "randomtext1",
"host2": "randomtext2",
"host3": "xyz",
"host4": "randomtext3"
}
Exporting a large number of shell variables might not be such a good idea and does not address the problem of array-valued variables. It might therefore be a good idea to think along the lines of printing the variable=value details to a file, and then combining that file with the template. It’s easy to do and examples on the internet abound and probably here on SO as well. You could, for example, use printf like so:
printf "%s\t" ${BASH_VERSINFO[#]}
3 2 57 1
You might also find declare -p helpful.
See also https://github.com/stedolan/jq/wiki/Cookbook#arbitrary-strings-as-template-variables
I am trying to parse a very large file which consists of JSON objects like this:
{"id": "100000002", "title": "some_title", "year": 1988}
Now I also have a very big list of ID's that I want to extract from the file, if they are there.
Now I know that I can do this:
jq '[ .[map(.id)|indices("1", "2")[]] ]' 0.txt > p0.json
Which produces the result I want, namely fills p0.json with only the objects that have "id" 1 and "2". Now comes the problem: my list of id's is very long too (100k or so). So I have a Python programm that outputs the relevant id's. My line of thought was, to first assign that to a variable:
REL_IDS=`echo python3 rel_ids.py`
And then do:
jq --arg ids "$REL_IDS" '[ .[map(.id)|indices($ids)[]] ]' 0.txt > p0.json
I tried both with brackets [$ids] and without brackets, but no luck so far.
My question is, given a big amount of arguments for the filter, how would I proceed with putting them into my jq command?
Thanks a lot in advance!
Since the list of ids is long, the trick is NOT to use --arg. However, the details will depend on the details regarding the "long list of ids".
In general, though, you'd want to present the list of ids to jq as a file so that you could use --rawfile or --slurpfile or some such.
If for some reason you don't want to bother with an actual file, then provided your shell allows it, you could use these file-oriented options with process substitution: <( ... )
Example
Assuming ids.json contains a lising of the ids as JSON strings:
"1"
"2"
"3"
then one could write:
< objects.json jq -c -n --slurpfile ids ids.json '
inputs | . as $in | select( $ids | index($in.id))'
Notice the use of the -n command-line option.
I have a JSON file and I am extracting data from it using jq. One simple use case is pulling out any JSON Object that contains an Id which is provided as an argument.
I use the following simple script to do so:
[.[] | select(.id == $ID)]
The script is stored in a separate file (by_id.jq) which I pass in using the -f argument.
The full command looks something like this:
cat ./my_json_file.json | jq -sf --arg ID "8df993c1-57d5-46b3-a8a3-d95066934e5b" ./by_id.jq
Is there a way by only using jq that a comma separated list of values could be passed as an argument to the jq script and iterate through the ids and check them against the value of .id in the the JSON file with the result being the objects that have that id?
For example if I wanted to pull out three objects by their ids I would want to structure the command in this way:
cat ./my_json_file.json | jq -sf --arg ID "8df993c1-57d5-46b3-a8a3-d95066934e5b,1d5441ca-5758-474d-a9fc-40d0f68aa538,23cc618a-8ad4-4141-bc1c-0251y0663963" ./by_id.jq
Sure. Though you'll need to parse (split) that list of ids to something that jq can work with, such as an array of ids. Then your problem becomes, given an array of keys, select objects that have any of these ids. Which you could use approaches found here.
$ jq --arg ID '8df993c1-57d5-46b3-a8a3-d95066934e5b,1d5441ca-5758-474d-a9fc-40d0f68aa538,23cc618a-8ad4-4141-bc1c-0251y0663963' '
select(.id | IN($ID|split(",")[]))
' ./my_json_file.json
I'm not sure what your input looks like but judging by your use of slurping then filtering the slurped input, it's a stream of objects. The slurping is not necessary here.
Here is an approach that focuses on efficiency.
Your Q indicates that in fact you have a stream of objects, so the first step towards efficiency is to avoid the -s option, and use -n with inputs instead.
The second step it to avoid splitting your comma-separated string of values more than once.
So your script might look like this:
INDEX($ids | splits(","); .) as $dict
| inputs
| select($dict[.id])
And the invocation would look like this:
jq -n --args a,b,c -f by_id.jq
This of course assumes that simply splitting the string of ids on "," will suffice. You might need to trim the values and take care of other potential anomalies.
For efficiency, it would be better to split $ID just once.
So if you have to use the -s option, you could use the following jq program:
INDEX($ID | splits(","); .) as $dict
| .[]
| select($dict[.id])
I would like to write a simple jq file that allows me to count items grouped by a specified key.
I expect the script contents to be something similar too:
group_by($group) | map({group: $group, cnt: length})
and to invoke it something like
cat my.json | jq --from-file count_by.jq --args group .header.messageType
Whatever I've tried the argument always ends up as a string and is not usable as a key.
Since you have not followed the minimal complete verifiable example
guidelines, it's a bit difficult to know what the best approach to your problem will be, but whatever approach you take, it is important to bear in mind that --arg always passes in a JSON string. It cannot be used to pass in a jq program fragment unless the fragment is a JSON string.
So let's consider one option: passing in a JSON object representing a path that you can use in your program.
So the invocation could be:
jq -f count_by.jq --argjson group '["header", "messageType"]'
and the program would begin with:
group_by(getpath($group)) | ...
Having your cake ...
If you really want to pass in arguments such as .header.messageType, there is a way: convert the string $group into a jq path:
($group|split(".")|map(select(length>0))) as $path
So your jq filter would look like this:
($group|split(".")|map(select(length>0))) as $path
| group_by(getpath($path)) | map({group: $group, cnt: length})
Shell string interpolation
If you want a quick bash solution that comes with many caveats:
group=".header.messageType"
jq 'group_by('"$group"') | map({group: "'"$group"'", cnt: length}'
I have to curl to a site (statuscake.com) that sends multiple items back in a JSON, each line of which contains multiple items. I want to extract from each line two of them, WebsiteName and TestID, so I can check if WebsiteName matches the one I'm interested in, get the TestID out and pass this to a second curl statement to delete the test.
Although it's more complex, the JSON that comes back is essentially of the form
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName": "SomeSite2"}]
I can't seem to find a magic jq command to do it all in one - if there is one, I'd be really happy to see it.
I've got
cat $data | jq '[.[] | .WebsiteName]'
to get an array of the website names (and a very similar one for the TestIDs, but I think I've done something daft. data is the information coming back from the curl to get the JSON and that's populated OK.
I want to be able to assign these to two arrays, names and ids, then search names for the index of the relevant name, grab the id from ids and pass that to the curl. Unless there's a better way.
Any advice please?
My Xidel can do it all at once by selecting the JSON with a XPath-like query:
E.g. return all ids where the WebsiteName contains "site2" from an array of objects:
xidel /tmp/x.json -e '$json()[contains((.).WebsiteName, "site2")]/TestID'
Or e.g. to download the original JSON and then make the HTTP request with the ids:
xidel http://statuscake.com/your-url... -f '$json()[contains((.).WebsiteName, "site2")]/TestID!x"/your-delete-url{.}..."'
If I'm getting your question right, it sounds like what you want is to, for each element, select those where .WebsiteName == "needle", and then get .TestID from it. You can do just that:
.[] | select(.WebsiteName == "needle") | .TestID
If you want an array as the result, you can wrap the above script in square brackets.
The jq filters startswith and endswith may be of interest to you. If you're going to pass the result back to cURL, you may also be interested in the #sh formatting filter and the -r command-line flag.
Assuming you have a bash 4+ and assuming the json is valid (does not contain newlines in strings, etc.) this works:
$ echo "$data"
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName":
"SomeSite2"}, {"TestID": 555, "WebsiteName": "foo*ba#r blah[54]quux{4,5,6}"}]
$ declare -A arr
$ while IFS= read -r line; do
eval "$line"
done < <(jq -M -r '.[] | #sh "arr[\(.WebsiteName)]+=\(.TestID)"' <<<"$data")
$ declare -p arr
declare -A arr='(["foo*ba#r blah[54]quux{4,5,6}"]="555" [SomeSite2]="1234" [SomeSite1]="123" )'
Here is a solution using only jq primitives.
.[]
| if .WebsiteName == "SomeSite1" then .TestID else empty end
This is essentially the same as Santiago's answer but if you are new to jq it may be informative because select/1 is defined as
def select(f): if f then . else empty end;