insert a json file into json - json

I'd like to know a quick way to insert a json to json.
$ cat source.json
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": ""
},
{
"name": "SERVICE_PORT",
"value": "4321"
}
]
}
]
}
The SERVICE_MANIFEST is content of another json file
$ cat service_manifest.json
{
"connections": {
"port": "1234"
},
"name": "foo"
}
I try to make it with jq command
cat service_manifest.json |jq --arg SERVICE_MANIFEST - < source.json
But seems it doesn't work
Any ideas? The final result still should be a valid json file
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": {
"connections": {
"port": "1234"
},
"name": "foo"
}
},
...
]
}
],
...
}
Updates.
Thanks, here is the command I run from your sample.
$ jq --slurpfile sm service_manifest.json '.containerDefinitions[].environment[] |= (select(.name=="SERVICE_MANIFEST").value=$sm)' source.json
But the result is an array, not list.
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"environment": [
{
"name": "SERVICE_MANIFEST",
"value": [
{
"connections": {
"port": "1234"
},
"name": "foo"
}
]
},
{
"name": "SERVICE_PORT",
"value": "4321"
}
]
}
]
}

You can try this jq command:
jq --slurpfile sm SERVICE_MANIFEST '.containerDefinitions[].environment[] |= (select(.name=="SERVICE_MANIFEST").value=$sm[])' file
--slurpfile assigns the content of the file to the variable sm
The filter replaces the array .containerDefinitions[].environment[] with the content of the file only on the element having SERVICE_MANIFEST as name.

A simple solution would use --argfile and avoid select:
< source.json jq --argfile sm service_manifest.json '
.containerDefinitions[0].environment[0].value = $sm '
Or if you want only to update the object(s) with .name == "SERVICE_MANIFEST" you could use the filter:
.containerDefinitions[].environment
|= map(if .name == "SERVICE_MANIFEST"
then .value = $sm
else . end)
Variations
There is no need for any "--arg"-style parameter at all, as illustrated by the following:
jq -s '.[1] as $sm
| .[0] | .containerDefinitions[0].environment[0].value = $sm
' source.json service_manifest.json

Related

Using jq to fetch key-value pairs from a json file

I am trying to get key#value pairs of JSON file below using jq
{
"STUFF_RELATED1": "STUFF_RELATED1",
"STUFF_RELATED2": "STUFF_RELATED2",
"THINGS": {
"THING_2": {
"details": {
"stuff_branch": "user/dev"
},
"repository": "path/to/repo",
"branch": "master",
"revision": "dsfkes4s34jlis4jsj4lis4sli3"
},
"THING_1": {
"details": {
"stuff_branch": "master"
},
"repository": "path/to/repo",
"branch": "master",
"revision": "dsfkes4s34jlis4jsj4lis4sli3"
}
},
"STUFF": {
"revision": "4u324i324iy32g",
"branch": "master"
}
}
The key#value pair should look like this:
THING_1#dsfkes4s34jlis4jsj4lis4sli3
Currently I have tried this on my own:
jq -r ' .THINGS | keys[] as $k | "($k)#(.[$k].revision)" ' things.json
But it does not give the resul that I really want.:( Thanks in advance!
You need to escape ( :
jq -r ' .THINGS | keys[] as $k | "\($k)#\(.[$k].revision)" ' things.json

jq: sort object values

I want to sort this data structure by the object keys (easy with -S and sort the object values (the arrays) by the 'foo' property.
I can sort them with
jq -S '
. as $in
| keys[]
| . as $k
| $in[$k] | sort_by(.foo)
' < test.json
... but that loses the keys.
I've tried variations of adding | { "\($k)": . }, but then I end up with a list of objects instead of one object. I also tried variations of adding to $in (same problem) or using $in = $in * { ... }, but that gives me syntax errors.
The one solution I did find was to just have the separate objects and then pipe it into jq -s add, but ... I really wanted it to work the other way. :-)
Test data below:
{
"": [
{ "foo": "d" },
{ "foo": "g" },
{ "foo": "f" }
],
"c": [
{ "foo": "abc" },
{ "foo": "def" }
],
"e": [
{ "foo": "xyz" },
{ "foo": "def" }
],
"ab": [
{ "foo": "def" },
{ "foo": "abc" }
]
}
Maybe this?
jq -S '.[] |= sort_by(.foo)'
Output
{
"": [
{
"foo": "d"
},
{
"foo": "f"
},
{
"foo": "g"
}
],
"ab": [
{
"foo": "abc"
},
{
"foo": "def"
}
],
"c": [
{
"foo": "abc"
},
{
"foo": "def"
}
],
"e": [
{
"foo": "def"
},
{
"foo": "xyz"
}
]
}
#user197693 had a great answer. A suggestion I got in a private message elsewhere was to use
jq -S 'with_entries(.value |= sort_by(.foo))'
If for some reason using the -S command-line option is not a satisfactory option, you can also perform the by-key sort using the to_entries | sort_by(.key) | from_entries idiom. So a complete solution to the problem would be:
.[] |= sort_by(.foo)
| to_entries | sort_by(.key) | from_entries

Grep command from json file - Bash scripting

My json file has the below content:
{
"Fruits": {
"counter": 1,
"protocols": [
{
"id": "100",
"name": "lemon",
"category": "citrus"
},
{
"id": "350",
"name": "Orange",
"category": "citrus"
},
{
"id": "150",
"name": "lime",
"category": "citrus"
}
]
}
}
I am expecting an output as below
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus
Easy to do with jq:
$ jq -r '.Fruits.protocols[] | "Fruits:\(.name):\(.category)"' input.json
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus
The jq answer is better. Still posting a Ruby solution (if you cannot use jq), but it is less elegant:
ruby -e '
require "json";
l="";
ARGF.each { |x| l+=x };
obj=JSON.parse(l);
obj["Fruits"]["protocols"].each { |x| puts "Fruits:#{x["name"]}:#{x["category"]}" }
'
Here is the full example:
echo '{"Fruits":{"counter":1,"protocols":[{"id":"100","name":"lemon","category":"citrus"},{"id":"350","name":"Orange","category":"citrus" },{"id":"150","name":"lime","category":"citrus"}]}}' \
| ruby -e 'require "json";l="";ARGF.each { |x| l+=x } ; obj=JSON.parse(l) ; obj["Fruits"]["protocols"].each { |x| puts "Fruits:#{x["name"]}:#{x["category"]}" }'
Output:
Fruits:lemon:citrus
Fruits:Orange:citrus
Fruits:lime:citrus

how to merge two JSON objects using jq?

I have two file json. I want to append two array of SomeFile2.json to SomeFile1.json as below.
SomeFile1.json
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25"
}
]
}
]
SomeFile2.json
[
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
},
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "InService"
}
]
I want the result as below:
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a"
"State": "InService"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25"
"State": "InService"
}
]
}
]
I'm processing in bash shell via jq. But, unsuccessful.
Since the contents of the second file are evidently intended to define a mapping from InstanceId to State, let's start by hypothesizing the following invocation of jq:
jq --argfile dict SomeFile2.json -f program.jq SomeFile1.json
Next, let's create a suitable dictionary:
reduce $dict[] as $x ({}; . + ($x|{(.InstanceId): .State}))) as $d
Now the rest is easy:
map(.Instances |= map(. + {State: $d[.InstanceId]}))
Putting the pieces together in program.jq:
(reduce $dict[] as $x ({}; . + ($x|{(.InstanceId): .State}))) as $d
| map(.Instances |= map(. + {State: $d[.InstanceId]}))
Alternatives
The dictionary as above can be constructed without using reduce, as follows:
($dict | map( {(.InstanceId): .State}) | add) as $d
Another alternative is to use INDEX/2:
(INDEX($dict[]; .InstanceId) | map_values(.State))) as $d
If your jq does not have INDEX/2 you can snarf its def from
https://raw.githubusercontent.com/stedolan/jq/master/src/builtin.jq
Since I find jq pretty hard, I started in a procedural way: using ruby's json module:
ruby -rjson -e '
states = JSON.parse(File.read(ARGV.shift)).map {|o| [o["InstanceId"], o["State"]]}.to_h
data = JSON.parse(File.read(ARGV.shift))
data.each do |obj|
obj["Instances"].each do |instance|
instance["State"] = states[instance["InstanceId"]] || "unknown"
end
end
puts JSON.pretty_generate data
' SomeFile2.json SomeFile1.json
But we want jq, so after some trial and error, and finding this in the manual: https://stedolan.github.io/jq/manual/#Complexassignments -- (note, I changed the state for one of the instances so I could verify the output better)
$ cat SomeFile2.json
[
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
},
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "NOTInService"
}
]
First, extract the states into an object mapping the id to the state:
$ state_map=$( jq -c 'map({"key":.InstanceId, "value":.State}) | from_entries' SomeFile2.json )
$ echo "$state_map"
{"i-0886ed703de64028a":"InService","i-0561634c4g3b4fa25":"NOTInService"}
Then, update the instances in the first file:
jq --argjson states "$state_map" '.[].Instances[] |= . + {"State": ($states[.InstanceId] // "unknown")}' SomeFile1.json
[
{
"DNSName": "CLB-test-112a877451.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0886ed703de64028a",
"State": "InService"
}
]
},
{
"DNSName": "CLB-test1-156925981.ap-northeast-1.elb.amazonaws.com",
"Instances": [
{
"InstanceId": "i-0561634c4g3b4fa25",
"State": "NOTInService"
}
]
}
]

jq - add new field with updating whole file

I have json file which is constructed in simmilar way:
[
{
"_id":"1234",
"org":"org1",
"int":
{"url":"http://url.com.uk:1234"}},
{
"_id":"4321",
"org":"org2",
"int":
{"url":"http://url.com.us:4321"}},
...
]
Now im "jumping" from one entry to another and checking if under URL application is working properly. After check i want to add/update field "status". But i can't update whole file, im just getting:
$ jq --arg mod "GOOD" '.[0].int + {stat: $mod}' tmp.json
{
"url": "http://url.com.uk:1234",
"stat": "GOOD"
}
How can i with jq command get new updated whole file, not just only part of it?
If you put your data in data.json and the changes you want to make to
each record into a separate arg.json argument file like
{
"1234": { "int": { "stat": "GOOD" } },
"4321": { "int": { "stat": "BAD", "xxx": "yyy" } }
}
and run jq as
$ jq -M --argfile arg arg.json 'map(. + $arg[._id])' data.json
then it will output the updated data, e.g.
[
{
"_id": "1234",
"org": "org1",
"int": {
"stat": "GOOD"
}
},
{
"_id": "4321",
"org": "org2",
"int": {
"stat": "BAD",
"xxx": "yyy"
}
}
]
Note that the + replaces keys. If you want to merge keys you can use * e.g.
$ jq -M --argfile arg arg.json 'map(. * $arg[._id])' data.json
which generates
[
{
"_id": "1234",
"org": "org1",
"int": {
"url": "http://url.com.uk:1234",
"stat": "GOOD"
}
},
{
"_id": "4321",
"org": "org2",
"int": {
"url": "http://url.com.us:4321",
"stat": "BAD",
"xxx": "yyy"
}
}
]
If you want to update the data in place you could use sponge
as described in the answer Manipulate JSON with jq
e.g.
$ jq -M --argfile arg arg.json 'map(. * $arg[._id])' data.json | sponge data.json
You can map to array and resign the int by operation, like:
jq --arg mod "GOOD" '.[] | .int=.int + {stat: $mod}' tmp.json
{
"_id": "1234",
"org": "org1",
"int": {
"url": "http://url.com.uk:1234",
"stat": "GOOD"
}
}
{
"_id": "4321",
"org": "org2",
"int": {
"url": "http://url.com.us:4321",
"stat": "GOOD"
}
}