I have a bash $string containing the values: abc,def and a file.json looking like this:
[
{
"loc": "51.12345, 12.12345",
"city": "CityName1"
},
{
"loc": "65.12345, 15.12345",
"city": "CityName2"
}
]
I'm trying to update the city field with the values from the string to get this result:
[
{
"loc": "51.12345, 12.12345",
"city": "abc"
},
{
"loc": "65.12345, 15.12345",
"city": "def"
}
]
I'm trying this code but it doesn't work, any suggestions?
string="abc,def"; jq --arg variable "$string" '.city = $string' file.json
You're looking for something like this:
$ string=abc,def
$ jq --arg cities "$string" '[., ($cities / ",")] | transpose | map(.[0] + {city: .[1]})' file.json
[
{
"loc": "51.12345, 12.12345",
"city": "abc"
},
{
"loc": "65.12345, 15.12345",
"city": "def"
}
]
$
With nodejs:
> var json = [
... {
... "loc": "51.12345, 12.12345",
... "city": "CityName1"
... },
... {
... "loc": "65.12345, 15.12345",
... "city": "CityName2"
... }
... ]
> var c = 0
> ["abc", "def"].forEach(n => json[c++].city = n)
> console.log(JSON.stringify(json, null, 4))
[
{
"loc": "51.12345, 12.12345",
"city": "abc"
},
{
"loc": "65.12345, 15.12345",
"city": "def"
}
]
From a script :
#!/bin/bash
node<<EOF | sponge file.json
var json = $(< file.json)
var c = 0;
["abc", "def"].forEach(n => json[c++].city = n)
console.log(JSON.stringify(json, null, 4))
EOF
output file:
[
{
"loc": "51.12345, 12.12345",
"city": "abc"
},
{
"loc": "65.12345, 15.12345",
"city": "def"
}
]
Using reduce would be another way. Same lengths provided, it allows iterating over the entries along with their indices, which can then be used to access the input array.
string="abc,def"
jq --arg var "$string" '
reduce ($var / "," | to_entries)[] as {$key, $value} (.;
.[$key].city = $value
)
'
I'm getting an error with jq version 1.4
To make this approach compatible with jq 1.4, replace the variable destructuring with a plain variable, and access its parts later, e.g.
string="abc,def"
jq --arg var "$string" '
reduce ($var / "," | to_entries)[] as $item (.;
.[$item.key].city = $item.value
)
'
Output:
[
{
"loc": "51.12345, 12.12345",
"city": "abc"
},
{
"loc": "65.12345, 15.12345",
"city": "def"
}
]
Related
My json file has the below content:
{
"Fruits": {
"counter": 1,
"protocols": [
{
"id": "100",
"name": "lemon",
"category": "citrus"
},
{
"id": "350",
"name": "Orange",
"category": "citrus"
},
{
"id": "150",
"name": "lime",
"category": "citrus"
}
]
}
}
I am expecting an output as below
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus
Easy to do with jq:
$ jq -r '.Fruits.protocols[] | "Fruits:\(.name):\(.category)"' input.json
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus
The jq answer is better. Still posting a Ruby solution (if you cannot use jq), but it is less elegant:
ruby -e '
require "json";
l="";
ARGF.each { |x| l+=x };
obj=JSON.parse(l);
obj["Fruits"]["protocols"].each { |x| puts "Fruits:#{x["name"]}:#{x["category"]}" }
'
Here is the full example:
echo '{"Fruits":{"counter":1,"protocols":[{"id":"100","name":"lemon","category":"citrus"},{"id":"350","name":"Orange","category":"citrus" },{"id":"150","name":"lime","category":"citrus"}]}}' \
| ruby -e 'require "json";l="";ARGF.each { |x| l+=x } ; obj=JSON.parse(l) ; obj["Fruits"]["protocols"].each { |x| puts "Fruits:#{x["name"]}:#{x["category"]}" }'
Output:
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus
I'm pasting here a JSON example data which would require some manipulation to get a desired output which is mentioned in the next section to be read after this piece of JSON code.
I want to use jq for parsing my desired data.
{
"MetricAlarms": [
{
"EvaluationPeriods": 3,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"AlarmActions": [
"Unimportant:Random:alarm:ELK2[10.1.1.2]-Root-Disk-Alert"
],
"AlarmName": "Unimportant:Random:alarm:ELK1[10.1.1.0]-Root-Alert",
"Dimensions": [
{
"Name": "path",
"Value": "/"
},
{
"Name": "InstanceType",
"Value": "m5.2xlarge"
},
{
"Name": "fstype",
"Value": "ext4"
}
],
"DatapointsToAlarm": 3,
"MetricName": "disk_used_percent"
},
{
"EvaluationPeriods": 3,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"AlarmActions": [
"Unimportant:Random:alarm:ELK2[10.1.1.2]"
],
"AlarmName": "Unimportant:Random:alarm:ELK2[10.1.1.2]",
"Dimensions": [
{
"Name": "path",
"Value": "/"
},
{
"Name": "InstanceType",
"Value": "r5.2xlarge"
},
{
"Name": "fstype",
"Value": "ext4"
}
],
"DatapointsToAlarm": 3,
"MetricName": "disk_used_percent"
}
]
}
So when I Pass some Key1 & value1 as a parameter "Name": "InstanceType", to the JQ probably using cat | jq and output expected should be as below
m5.2xlarge
r5.2xlarge
A generic approach to search for a key-value pair (sk-sv) in input recursively and extract another key's value (pv) from objects found:
jq -r --arg sk Name \
--arg sv InstanceType \
--arg pv Value \
'.. | objects | select(contains({($sk): $sv})) | .[$pv]' file
I'd like to know a quick way to insert a json to json.
$ cat source.json
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": ""
},
{
"name": "SERVICE_PORT",
"value": "4321"
}
]
}
]
}
The SERVICE_MANIFEST is content of another json file
$ cat service_manifest.json
{
"connections": {
"port": "1234"
},
"name": "foo"
}
I try to make it with jq command
cat service_manifest.json |jq --arg SERVICE_MANIFEST - < source.json
But seems it doesn't work
Any ideas? The final result still should be a valid json file
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": {
"connections": {
"port": "1234"
},
"name": "foo"
}
},
...
]
}
],
...
}
Updates.
Thanks, here is the command I run from your sample.
$ jq --slurpfile sm service_manifest.json '.containerDefinitions[].environment[] |= (select(.name=="SERVICE_MANIFEST").value=$sm)' source.json
But the result is an array, not list.
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": [
{
"connections": {
"port": "1234"
},
"name": "foo"
}
]
},
{
"name": "SERVICE_PORT",
"value": "4321"
}
]
}
]
}
You can try this jq command:
jq --slurpfile sm SERVICE_MANIFEST '.containerDefinitions[].environment[] |= (select(.name=="SERVICE_MANIFEST").value=$sm[])' file
--slurpfile assigns the content of the file to the variable sm
The filter replaces the array .containerDefinitions[].environment[] with the content of the file only on the element having SERVICE_MANIFEST as name.
A simple solution would use --argfile and avoid select:
< source.json jq --argfile sm service_manifest.json '
.containerDefinitions[0].environment[0].value = $sm '
Or if you want only to update the object(s) with .name == "SERVICE_MANIFEST" you could use the filter:
.containerDefinitions[].environment
|= map(if .name == "SERVICE_MANIFEST"
then .value = $sm
else . end)
Variations
There is no need for any "--arg"-style parameter at all, as illustrated by the following:
jq -s '.[1] as $sm
| .[0] | .containerDefinitions[0].environment[0].value = $sm
' source.json service_manifest.json
I have two file json. I want to append two array of SomeFile2.json to SomeFile1.json as below.
SomeFile1.json
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25"
}
]
}
]
SomeFile2.json
[
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
},
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "InService"
}
]
I want the result as below:
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a"
"State": "InService"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25"
"State": "InService"
}
]
}
]
I'm processing in bash shell via jq. But, unsuccessful.
Since the contents of the second file are evidently intended to define a mapping from InstanceId to State, let's start by hypothesizing the following invocation of jq:
jq --argfile dict SomeFile2.json -f program.jq SomeFile1.json
Next, let's create a suitable dictionary:
reduce $dict[] as $x ({}; . + ($x|{(.InstanceId): .State}))) as $d
Now the rest is easy:
map(.Instances |= map(. + {State: $d[.InstanceId]}))
Putting the pieces together in program.jq:
(reduce $dict[] as $x ({}; . + ($x|{(.InstanceId): .State}))) as $d
| map(.Instances |= map(. + {State: $d[.InstanceId]}))
Alternatives
The dictionary as above can be constructed without using reduce, as follows:
($dict | map( {(.InstanceId): .State}) | add) as $d
Another alternative is to use INDEX/2:
(INDEX($dict[]; .InstanceId) | map_values(.State))) as $d
If your jq does not have INDEX/2 you can snarf its def from
https://raw.githubusercontent.com/stedolan/jq/master/src/builtin.jq
Since I find jq pretty hard, I started in a procedural way: using ruby's json module:
ruby -rjson -e '
states = JSON.parse(File.read(ARGV.shift)).map {|o| [o["InstanceId"], o["State"]]}.to_h
data = JSON.parse(File.read(ARGV.shift))
data.each do |obj|
obj["Instances"].each do |instance|
instance["State"] = states[instance["InstanceId"]] || "unknown"
end
end
puts JSON.pretty_generate data
' SomeFile2.json SomeFile1.json
But we want jq, so after some trial and error, and finding this in the manual: https://stedolan.github.io/jq/manual/#Complexassignments -- (note, I changed the state for one of the instances so I could verify the output better)
$ cat SomeFile2.json
[
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
},
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "NOTInService"
}
]
First, extract the states into an object mapping the id to the state:
$ state_map=$( jq -c 'map({"key":.InstanceId, "value":.State}) | from_entries' SomeFile2.json )
$ echo "$state_map"
{"i-0886ed703de64028a":"InService","i-0561634c4g3b4fa25":"NOTInService"}
Then, update the instances in the first file:
jq --argjson states "$state_map" '.[].Instances[] |= . + {"State": ($states[.InstanceId] // "unknown")}' SomeFile1.json
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "NOTInService"
}
]
}
]
I'm currently using jq with the 1pass CLI to try and create randomly generated passwords into a secure note. I'm having an issue with setting the fields.
These are two of my variables. I have 8 total I need to set.
section0_section_uuid="Section_0"
section1_section_uuid="Section_1"
And here are my commands to manipulate the template. I first read it in, change the first title, then save it to $template. I then pass $template into jq
template=$(cat template.json | jq --arg uuid "$section0_section_uuid" '.sections[0].title=$uuid')
template=$($template | jq --arg uuid "$section1_section_uuid" '.sections[1].title=$uuid')
echo $template
I get "file name too long." I don't think I'm passing the modified template variable in correctly. I need to do 7 more modifications to the template.json file.
Edit:
Here's the full template I'm trying to manipulate. It's 12 total changes to the template I have to make. 10 of the 12 are random numbers that I will generate. The remaining 2 of the 12 will be a generated usernames.
{
"fields": [],
"sections": [
{
"fields": [
{
"k": "concealed",
"n": "[CHANGE_ME]",
"t": "ROOT_USER_PASS",
"v": "[CHANGE_ME]"
},
{
"k": "concealed",
"n": "[CHANGE_ME]",
"t": "DEV_USER_PASS",
"v": "[CHANGE_ME]"
}
],
"name": "Section_[CHANGE_ME]",
"title": "Container SSH"
},
{
"fields": [
{
"k": "string",
"n": "[CHANGE_ME]",
"t": "placeholdertext",
"v": "[CHANGE_ME_LETTERS]"
},
{
"k": "string",
"n": "[CHANGE_ME]",
"t": "placeholdertext",
"v": "[CHANGE_ME_LETTERS]"
},
{
"k": "concealed",
"n": "[CHANGE_ME]",
"t": "placeholdertext",
"v": "[CHANGE_ME]"
}
],
"name": "Section_[CHANGE_ME]",
"title": "MySQL"
}
]
}
Why not make your template an actual jq filter, rather than a JSON blob to modify?
The contents of template.jq would be
{
sections: [
{ title: $t1 },
{ title: $t2 },
{ title: $t3 },
{ title: $t4 },
{ title: $t5 },
{ title: $t6 },
{ title: $t7 },
{ title: $t8 }
]
}
Then your command would simply be
$ jq -n --arg t1 foo --arg t2 bar ... -f template.jq
{
"sections": [
{
"title": "foo"
},
{
"title": "bar"
},
...
]
}
One benefit of doing it this way is that you can't accidentally forget a value; jq can only process the filter if you provide definitions for all 8 variables.
Here is a solution which uses jq to build a legal json array from bash variables and then uses that array with a second jq invocation to substitute the variables into the template at corresponding positions:
#!/bin/bash
# example template
template='{
"sections": [
{ "title": "x" },
{ "title": "y" }
]
}'
# bash variables
section0_section_uuid="Section_0"
section1_section_uuid="Section_1"
# put into json array with jq
uuids=$(jq -MRn '[inputs]' <<EOF
$section0_section_uuid
$section1_section_uuid
EOF)
# substitute json array into template
jq -M --argjson uuids "$uuids" '
reduce ($uuids|keys[]) as $k (.; .sections[$k].title = $uuids[$k])
' <<< "$template"
Sample Output
{
"sections": [
{
"title": "Section_0"
},
{
"title": "Section_1"
}
]
}
Try it online!
Here is a solution to a portion of the revised problem which works by replacing leaf values in the template with corresponding values from an object constructed from bash variables and passed to jq via --argjson. It should be straightforward to generalize to the complete template assuming more suitable names are chosen for replacement values then [CHANGE_ME] and [CHANGE_ME_LETTERS]
#!/bin/bash
# example template
template='{
"fields": [],
"sections": [ {
"fields": [ {
"k": "concealed",
"n": "[SSH_ROOT_USER_N]",
"t": "ROOT_USER_PASS",
"v": "[SSH_ROOT_USER_V]"
} ]
} ]
}'
# bash variables
SSH_ROOT_USER_N="abcd"
SSH_ROOT_USER_V="efgh"
# put into json object with jq
vars=$(jq -M . <<EOF
{
"[SSH_ROOT_USER_N]": "$SSH_ROOT_USER_N",
"[SSH_ROOT_USER_V]": "$SSH_ROOT_USER_V"
}
EOF)
# substitute variables into template
jq -M --argjson vars "$vars" '
reduce (tostream|select(length==2)) as [$p,$v] (
{}
; setpath($p;if $v|type!="string" then . else $vars[$v]//$v end)
)
' <<< "$template"
Sample Output
{
"fields": {},
"sections": [
{
"fields": [
{
"k": "concealed",
"n": "abcd",
"t": "ROOT_USER_PASS",
"v": "efgh"
}
]
}
]
}
Try it online!
The following approach to the problem is similar to #jq170727's (in particular, the jq program is agnostic both about the number of "section_uuid" variables, and the names of the template variables), but only one invocation of jq is required (rather than three).
The other significant difference is that reduce is used to avoid the penalties associated with using tostream. A minor difference is that inputs is used to avoid reading in the "section_uuid" variable values all at once.
Note: The fillin function defined below should be sufficient for basic templating.
In the following, the "template" file is assumed to be named template.json.
template.jq
# input: a JSON entity defining a template;
# vars: a JSON object defining TEMPLATEVARIABLE-VALUE pairs
def fillin(vars):
reduce paths as $p (.;
getpath($p) as $v
| if $v|type == "string" and vars[$v]
then setpath($p; vars[$v])
else .
end);
reduce inputs as $line ({i:0, value:$template};
(.value.sections[.i].title |= $line)
| .i +=1)
| .value
| fillin($vars)
The script
#!/bin/bash
### Set the bash variables - as many as needed
section0_section_uuid="Section_0"
section1_section_uuid="Section_1"
ROOT_USER_PASS=RUP
DEV_USER_PASS=DUP
### Preparations for calling jq
vars=$(cat<<EOF
{
"ROOT_USER_PASS": "$ROOT_USER_PASS",
"DEV_USER_PASS": "$DEV_USER_PASS"
}
EOF
)
cat << EOF | jq -nR --argfile template template.json --argjson vars "$vars" -f template.jq
$section0_section_uuid
$section1_section_uuid
EOF
Output
With the (expanded) example template, the output is:
{
"fields": [],
"sections": [
{
"fields": [
{
"k": "concealed",
"n": "[CHANGE_ME]",
"t": "RUP",
"v": "[CHANGE_ME]"
},
{
"k": "concealed",
"n": "[CHANGE_ME]",
"t": "DUP",
"v": "[CHANGE_ME]"
}
],
"name": "Section_[CHANGE_ME]",
"title": "Section_0"
},
{
"fields": [
{
"k": "string",
"n": "[CHANGE_ME]",
"t": "placeholdertext",
"v": "[CHANGE_ME_LETTERS]"
},
{
"k": "string",
"n": "[CHANGE_ME]",
"t": "placeholdertext",
"v": "[CHANGE_ME_LETTERS]"
},
{
"k": "concealed",
"n": "[CHANGE_ME]",
"t": "placeholdertext",
"v": "[CHANGE_ME]"
}
],
"name": "Section_[CHANGE_ME]",
"title": "Section_1"
}
]
}