How to replace values in a JSON dictionary with their respective shell variables in jq? - json

I have the following JSON structure:
{
"host1": "$PROJECT1",
"host2": "$PROJECT2",
"host3" : "xyz",
"host4" : "$PROJECT4"
}
And the following environment variables in the shell:
PROJECT1="randomtext1"
PROJECT2="randomtext2"
PROJECT4="randomtext3"
I want to check the values for each key, if they have a "$" character in them, replace them with their respective environment variable(which is already present in the shell) so that my JSON template is rendered with the correct environment variables.
I can use the --args option of jq but there are quite a lot of variables in my actual JSON template that I want to render.
I have been trying the following:
jq 'with_entries(.values as v | env.$v)
Basically making each value as a variable, then updating its value with the variable from the env object but seems like I am missing out on some understanding. Is there a straightforward way of doing this?
EDIT
Thanks to the answers on this question, I was able to achieve my larger goal for a part of which this question was asked
iterating over each value in an object,
checking its value,
if it's a string and starts with the character "$"
use the value to update it with an environment variable of the same name .
if it's an array
use the value to retrieve an environment variable of the same name
split the string with "," as delimiter, which returns an array of strings
Update the value with the array of strings
jq 'with_entries(.value |= (if (type=="array") then (env[.[0][1:]] | split(",")) elif (type=="string" and startswith("$")) then (env[.[1:]]) else . end))'

You need to export the Bash variables to be seen by jq:
export PROJECT1="randomtext1"
export PROJECT2="randomtext2"
export PROJECT4="randomtext3"
Then you can go with:
jq -n 'with_entries((.value | select(startswith("$"))) |= env[.[1:]])'
and get:
{
"host1": "randomtext1",
"host2": "randomtext2",
"host3": "xyz",
"host4": "randomtext3"
}

Exporting a large number of shell variables might not be such a good idea and does not address the problem of array-valued variables. It might therefore be a good idea to think along the lines of printing the variable=value details to a file, and then combining that file with the template. It’s easy to do and examples on the internet abound and probably here on SO as well. You could, for example, use printf like so:
printf "%s\t" ${BASH_VERSINFO[#]}
3 2 57 1
You might also find declare -p helpful.
See also https://github.com/stedolan/jq/wiki/Cookbook#arbitrary-strings-as-template-variables

Related

Replace value of object property in multiple JSON files

I'm working with multiple JSON files that are located in the same folder.
Files contain objects with the same properties and they are such as:
{
"identifier": "cameraA",
"alias": "a",
"rtsp": "192.168.1.1"
}
I want to replace a property for all the objects in the JSON files at the same time for a certain condition.
For example, let's say that I want to replace all the rtsp values of the objects with identifier equal to "cameraA".
I've been trying with something like:
jq 'if .identifier == \"cameraA" then .rtsp=\"cameraX" else . end' -c *.json
But it isn't working.
Is there a simple way to replace the property of an object among multiple JSON files?
jq can only write to STDIN and STDOUT, so the simplest approach would be to process one file at a time, e.g. putting your jq program inside a shell loop. sponge is often used when employing this approach.
However, there is an alternative that has the advantage of efficiency. It requires only one invocation of jq, the output of which would include the filename information (obtained from input_filename). This output would then be the input of an auxiliary process, e.g. awk.

How do you conditionally change a string value to a number in JQ?

I am pulling a secret from SecretsManager in AWS and using the resulting JSON to build a parameters JSON file that can pass this on to the cloud formation engine. Unfortunately, SecretsManager stores all values as strings, so when I try to pass these values to my cloud formation template it will fail because it is passing a string instead of a number and some cloud formation parameters need to be numbers (e.g. not a string).
In the example below, I want to tell JQ that "HEALTH_CHECK_UNHEALTHY_THRESHOLD_COUNT" and "AUTOSCALING_MAX_CAPACITY" are numbers. So, I prefix the key with "NUMBER::".
This serves two purposes. First, it tells the person viewing this secret that it will be converted to a number, second, it will tell JQ to convert the string value of "2" to 2. This needs to scale so that I can have 1..n keys that need to be converted in the JSON.
Consider this JSON:
{
"NUMBER::AUTOSCALING_MAX_CAPACITY": "12",
"SERVICE_PLATFORM_VERSION": "1.3.0",
"HEALTH_CHECK_PROTOCOL": "HTTPS",
"NUMBER::HEALTH_CHECK_UNHEALTHY_THRESHOLD_COUNT": "2"
}
Here is what I'd like to do with JQ:
JQ will copy over the key/value pairs for the majority of elements in the JSON "as is". If there is no "NUMBER::" prefix, they are copied over "as is".
However, if a key is prefixed with "NUMBER::" I'd like the following to happen:
a. JQ will remove the "NUMBER::" prefix from the key name.
b. JQ will convert the value from a string to a number.
The end result is a JSON that looks like this:
{
"AUTOSCALING_MAX_CAPACITY": 12,
"SERVICE_PLATFORM_VERSION": "1.3.0",
"HEALTH_CHECK_PROTOCOL": "HTTPS",
"HEALTH_CHECK_UNHEALTHY_THRESHOLD_COUNT": 2
}
What I've tried
I have tried using Map to do this with limited success. In this example I am looking for a specific field mainly as a test. I don't want to have to call out specific keys by name, but rather just use any key that begins with "NUMBER::" to do the conversions.
NOTE: The SECRET_STRING variable in the examples below contains the source JSON.
echo $SECRET_STRING | jq 'to_entries | map(if .key == "NUMBER::AUTOSCALING_MAX_CAPACITY" then . + {"value":.value} else . end ) | from_entries'**
I've also tried to use "tonumber" across the entire JSON. JQ will examine all the values and see if it can convert them to numbers. The problem is it fails when it hits the "SERVICE_PLATFORM_VERSION" key as it detects "1.3.0" as a number and it tries for make that a number, which of course is bogus.
Example: echo $SECRET_STRING | jq -r '.[] | tonumber'
Recap
I'd like to use JQ to convert JSON string values to number by use a prefix of "NUMBER::" in the key name.
Note: This problem does not exist when attempting to pull entries from the Systems Manager Parameter Store because AWS allows you use "resolve" entries as strings or numbers. The same feature does not exist in SecretsManager. I'd also like to use the SecretsManager to provide a list of some 30 or more configuration items to set up my stack. With the Parameter store you have to set up each config item as a separate entry, which we be a maintenance nightmare.
Select each entry with a key starting with NUMBER:: and update it to remove that prefix and convert the value to a number.
with_entries(
select(.key | startswith("NUMBER::")) |= (
(.key |= ltrimstr("NUMBER::")) |
(.value |= tonumber)
)
)
Online demo

How to pass a key to a jq file

I would like to write a simple jq file that allows me to count items grouped by a specified key.
I expect the script contents to be something similar too:
group_by($group) | map({group: $group, cnt: length})
and to invoke it something like
cat my.json | jq --from-file count_by.jq --args group .header.messageType
Whatever I've tried the argument always ends up as a string and is not usable as a key.
Since you have not followed the minimal complete verifiable example
guidelines, it's a bit difficult to know what the best approach to your problem will be, but whatever approach you take, it is important to bear in mind that --arg always passes in a JSON string. It cannot be used to pass in a jq program fragment unless the fragment is a JSON string.
So let's consider one option: passing in a JSON object representing a path that you can use in your program.
So the invocation could be:
jq -f count_by.jq --argjson group '["header", "messageType"]'
and the program would begin with:
group_by(getpath($group)) | ...
Having your cake ...
If you really want to pass in arguments such as .header.messageType, there is a way: convert the string $group into a jq path:
($group|split(".")|map(select(length>0))) as $path
So your jq filter would look like this:
($group|split(".")|map(select(length>0))) as $path
| group_by(getpath($path)) | map({group: $group, cnt: length})
Shell string interpolation
If you want a quick bash solution that comes with many caveats:
group=".header.messageType"
jq 'group_by('"$group"') | map({group: "'"$group"'", cnt: length}'

AWS CLI / jq - transforming JSON with tags, and showing information even for non-defined tags

I'm facing an issue when trying to process output of 'aws ec2 describe-instances' command with 'jq', and I really need some help.
I want to transform JSON output into CSV file with the list of all instances, with
columns 'Name,InstanceId,Tag-Client,Tag-CostCenter'.
I've been using jq's select with a command like:
aws ec2 describe-instances |
jq -r '.Reservations[].Instances[]
| (.Tags[]|select(.Key=="Name")|.Value) + "," + .InstanceId + ","
+ (.Tags[]|select(.Key=="Client")|.Value) + ","
+ (.Tags[]|select(.Key=="CostCenter")|.Value)'
However using selects in this way, only those entries containing all the tags are displayed, not showing those that contain one of the tags only.
I understand the behavior, which is similar to a grep, but I'm trying to figure out if it's possible to perform this operation using jq, so in the case that one tag is not defined, would just return string "" and not remove the whole line.
I've found a reference about using 'if' clauses in jq ([https://ilya-sher.org/2016/05/11/most-jq-you-will-ever-need/], but wondering in anyone has resolved such case without having to make this logic or splitting the command in different executions.
Whenever you are given an array of key/value pairs (the tags here) and you want to extract values by their key, it'll be easier to map them into an object so you can access them directly. Functions like from_entries will work well with this.
However, since you're also trying to retrieve values not within this tag array, you can approach it a little differently to save some steps. Using reduce or foreach, you can go through each of the tags and add it to an object that holds all the values you're interested in. Then you can map the values you want into an array then convert to a csv row.
So if your goal is to create rows of Tags[Name], InstaceId, Tags[Client], Tags[CostCenter] for each instance, you could do this:
# for each instance
.Reservations[].Instances[]
# map each instance to an object where we can easily extract the values
| reduce .Tags[] as $t (
{ InstanceId }; # we want the InstanceId from the instance
.[$t.Key] = $t.Value # add the values to the object
)
# map the desired values to an array
| [ .Name, .InstanceId, .Client, .CostCenter ]
# convert to csv
| #csv
And the good news is, if Name, Client, or CostCenter doesn't exist in the tag array, or even InstanceId, then they'll just be null which becomes empty when converted to csv.

Conditional variables in JQ json depending on argument value?

I am trying to build a json with jq with --arg arguments however I'd like for the json not to be able to have a condition if the variable is empty.
An example, if I run the following command
jq -n --arg myvar "${SOMEVAR}" '{ $myvar}'
I'd like the json in that case to be {} if myvar happens to be empty (Because the variable ${SOMEVAR} does not exist) and not { "myvar": "" } which is what I get by just running the command above.
Is there any way to achieve this through some sort of condition?
UPDATE:
Some more details about the use case
I want to build a json based on several environment variables but only include the variables that have a value.
Something like
{"varA": "value", "varB": "value"}
But only include varA if its value is defined and so on. The issue now is that if value is not defined, the property varA will still exist with an empty value and because of the multiple argument/variable nature, using an if/else to build the entire json as suggested will lead to a huge amount of conditions to cover for every possible combination of variables not existing
Suppose you have a template of variable names, in the form of an object as you have suggested you want:
{a, b, c}
Suppose also (for the sake of illustration) that you want to pull in the corresponding values from *ix environment variables. Then you just need to adjust the template, which can be done using this filter:
def adjust: with_entries( env[.key] as $v | select($v != null) | .value = $v );
Example:
Assuming the above filter, together with the following line, is in a file named adjust.jq:
{a,b,c} | adjust
then:
$ export a=123
$ jq -n -f -c adjust.jq
{"a":"123"}
You can use an if/else construct:
jq -n --arg myvar "${SOMEVAR}" 'if ($myvar|length > 0) then {$myvar} else {} end'
It's still not clear where the variable-value pairs are coming from, so maybe it would be simplest to construct the object containing the mapping before invoking jq, and then passing it in using the --argjson or --argfile option?