I have a template json file with list of hosts for an example. The same to be replaced with the dynamic value being generated by shell script.
Sample rds.json
{
"lob": "coaching",
"function": "badminton",
"hosts": [
"node1.rds.sports.com",
"node2.rds.sports.com",
"node3.rds.sports.com"
],
"adminserver": "node1.rds.sports.com",
"user_name": "coach",
"sudo_type": "sudo",
"group_name": "admin"
}
echo $myHosts:
"host1.rds.sports.com", "host2.rds.sports.com", "host3.rds.sports.com", "host4.rds.sports.com", "host5.rds.sports.com", "host6.rds.sports.com", "host7.rds.sports.com", "host8.rds.sports.com"
The value of $myHosts should get replaced in hosts elements key.
Desired output:
{
"lob": "coaching",
"function": "badminton",
"hosts": [
"host1.rds.sports.com",
"host2.rds.sports.com",
"host3.rds.sports.com",
"host4.rds.sports.com",
"host5.rds.sports.com",
"host6.rds.sports.com",
"host7.rds.sports.com",
"host8.rds.sports.com"
],
"adminserver": "hosts1.rds.sports.com",
"user_name": "coach",
"sudo_type": "sudo",
"group_name": "admin"
}
I'm going to assume the contents of myHosts is a valid JSON array body.
jq --argjson hosts "[$myHosts]" '.hosts = $hosts | .adminserver = .hosts[0]' rds.json
jqplay
Related
What I'm trying to do currently, is, within each environment, compare mainAccount and secondAccount values.
If they do match, then I will trigger some downstream code to check the file version. If they do not, then I will pass. That is not really relevant, however I am struggling to compare the values across each environment. Since each .json file will have different amounts of environments.
Meaning, in testing environment, I want to check if mainAccount = secondAccount, and same in production environment.
I'm running into issues parsing this JSON with jq:
json1
{
"file_version": 1.0,
"config": [
{
"environment": "testing",
"main": [
{
"mainAccount": "123"
}
],
"second": [
{
"secondAccount": "456"
}
]
},
{
"environment": "production",
"main": [
{
"mainAccount": "789"
}
],
"second": [
{
"secondAccount": "789"
}
]
}
]
}
Here's another sample .json file for comparsion:
json2
{
"file_version": 1.3,
"config": [
{
"environment": "testing",
"main": [
{
"mainAccount": "123"
}
],
"second": [
{
"secondAccount": "456"
}
]
},
{
"environment": "production",
"main": [
{
"mainAccount": "789"
}
],
"second": [
{
"secondAccount": "789"
}
]
},
{
"environment": "pre-production",
"main": [
{
"mainAccount": "456"
}
],
"second": [
{
"secondAccount": "789"
}
]
},
{
"environment": "staging",
"main": [
{
"mainAccount": "234"
}
],
"second": [
{
"secondAccount": "456"
}
]
}
]
}
If I run this command:
jq -r '.config[] | select(.main != null) | .main[].mainAccount
My output is:
123
789
If i store this output in a variable, it'll be 123 789 so comparing this to the "secondAccount" value is troublesome.
I think what I'm looking for is iteration here, however, I'm not sure how to implement this. I wanted to take a pythonic approach to check the length of the config array, create a for loop in that length range, then collect the value based on an index like
.config[0] | select(.main != null) | .main[].mainAccount
.config[1] | select(.main != null) | .main[].mainAccount
etc. The issue however, is that when I read in the .config[] value as a variable, bash doesn't interpret it like that. The length will be the length of characters, not, the amount of objects in the array.
EXPECTED OUTPUT
Nothing. I simply want to, for each .json file above, compare the mainAccount and secondAccount values with eachother, within each environment.
In json1, I want to compare mainAccount == secondAccount in environment: testing. Then mainAccount == secondAccount in environment: production.
Then move onto json 2 and compare mainAccount == secondAccount in environment: testing. Then environment production, pre-production, staging, so on and so forth.
Since all information is within this one JSON file it is better to do the processing in jq as much as possible and to keep the shell out.
Given your input you can try this jq:
jq '
.config[]
| {
environment,
condition: (.main[0].mainAccount == .second[0].secondAccount)
}' input.json
The result is:
{
"environment": "testing",
"condition": false
}
{
"environment": "production",
"condition": true
}
Some questions though:
Why are the values of first and second arrays objects and not object?
Is it really intended to match the first one of both?
Can there be more items in the arrays?
Also: If you want to process the results in a shell, I propose this expression because the output can be used (source or eval) in a shell:
jq -r '
.config[]
| "\(.environment)=\(.main[0].mainAccount == .second[0].secondAccount)"' input.json
The output is:
testing=false
production=true
You can do the comparison within jq, return the boolean result as its exit status using the -e option, and react upon that in bash, e.g. using an if statement.
if jq -e '
.config | map(select(.main != null) | .main[].mainAccount) | .[0] == .[1]
' file.json >/dev/null
then echo "equal"
else echo "not equal"
fi
not equal
Lets say I have an I/P json file as below. And I want to extract the O/P in a CSV format with the below fields. Specifically, I want to get the value of the key "Gamma" in the o/p if the key "Gamma" exists in "tags" map. If the key doesn't exists, it should just print a NULL value. The expected o/p is below.
generated_time,platform,id,,
2021-09-09:12:03:12,earth,2eeee67748,Ray,2021-08-25 09:41:06
2021-09-09:12:03:12,sun,xxxxx12334,NULL,2021-08-25 10:11:31
[
{
"generated_time": "generated_time",
"platform": "platform",
"id": "id"
},
{
"generated_time": "2021-09-09:12:03:12",
"platform": "earth",
"id": "2eeee67748",
"tags": {
"app": "map",
"Gamma": "Ray",
"null": [
"allow-all-humans"
]
},
"created": "2021-08-25 09:41:06"
},
{
"generated_time": "2021-09-09:12:03:12",
"platform": "sun",
"id": "xxxxx12334",
"tags": {
"component": "machine",
"environment": "hot",
"null": [
"aallow-all-humans"
]
},
"created": "2021-08-25 10:11:31"
}
]
jq has a builtin #csv which renders an array
as CSV with double quotes for strings, and quotes escaped by repetition.
If the additional quoting (as compared to your expected output) isn't an issue, the following
jq --raw-output '
# produce an array for each element in the input array
.[] | [
# containing the first three columns unchanged
.generated_time, .platform, .id,
# if the input element has a field named "tags"
if has("tags")
# then add two more columns and replace an inexistant Gamma with "NULL"
then (.tags.Gamma // "NULL", .created)
# otherwise add two empty columns instead
else (null, null) end
# and convert the array into CSV format
] | #csv
' input.json
will produce
"generated_time","platform","id",,
"2021-09-09:12:03:12","earth","2eeee67748","Ray","2021-08-25 09:41:06"
"2021-09-09:12:03:12","sun","xxxxx12334","NULL","2021-08-25 10:11:31"
I have a large JSON file (about 7K lines) with deeply nested items which has a missing required property collection that I need to add.
Current JSON object:
{
"item": [
{
"id": "123",
"name": "Customer",
"item": [
{
"id": "456",
"name": "Retrieve a customer"
....
Using a bash script, I need to add a top level property "collection" like this, which still contains the same nested items within it. This is my desired result:
{
"collection": {
"item": [
{
"id": "123",
"name": "Customer",
"item": [
{
"id": "456",
"name": "Retrieve a customer",
....
At the end of the JSON object I also need the matching closing } brace at the end of the file for my newly added collection: key. Is there a way to do this with JQ?
jq '{"collection": .}' <in.json >out.json
And if your JSON is the output of another jq command, just add the collection at the end, like:
# For example: delete an element and then wrap
# entries around a "records" attribute (assuming
# the date is already a JSON list):
jq '[.[] | del(.undesiredAttribute)] | {"records": .}'
Then the output is:
{"records":[{"name":"Foo"},{"name":"Bar"}]}
I have multiple JSON files one.json, two.json, three.json with the below format and I want to create a consolidated array from them using jq. So, from all the files I want to extract Name and Value field inside the Parameters and use them to create an array where the id value will be constructed from the Name value and value field will be constructed using Value field value.
input:
one.json:
{
"Parameters": [
{
"Name": "id1",
"Value": "one",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
two.json
{
"Parameters": [
{
"Name": "id2",
"Value": "xyz",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
three.json
{
"Parameters": [
{
"Name": "id3",
"Value": "xyz",
"Version": 2,
"LastModifiedDate": 1581663187.36
}
]
}
output:
[
{
"id": "id1",
"value": "one"
},
{
"id": "id2",
"value": "xyz"
},
{
"id": "id3",
"value": "xyz"
}
]
How to achieve this using jq
You can use a reduce expression instead of slurping the whole file into memory (-s); by iterative manipulation of the input file contents and then appending the required fields one at a time.
jq -n 'reduce inputs.Parameters[] as $d (.; . + [ { id: $d.Name, value: $d.Value } ])' one.json two.json three.json
The -n flag is to ensure that we construct the output JSON data from scratch over the input file contents made available over the inputs function. Since reduce works in an iterative manner, for each of the object in the input, we create a final array, creating the KV pair as desired.
I'm using jq to try to filter a JSON list based on the content of a list inside the objects within that list. Here's a sample of my JSON document:
{
"modules": [
{
"path": [
"root"
],
"outputs": {
"a": "b",
"c": "d"
}
},
{
"path": [
"other1"
],
"outputs": {
"e": "f",
"g": "h"
}
},
{
"path": [
"other2"
],
"outputs": {
"i": "j",
"k": "l"
}
}
]
}
I want to filter the modules list to the object where the path list contains "root", then return the outputs object. Essentially I want to return:
{"a":"b","c":"d"}
which I can do using jq .modules[0].outputs (see example on http://jqplay.org) but I don't want to make an assumption that the object I'm interested in is the 0th element of the modules list, instead I want to filter the modules list where the path list contains an element "root".
How can I do that?
Typical, as soon as I post the question I stumble upon the answer through trial and error.
.modules[] | select(.path == ["root"]).outputs
see example on jqplay.org