Trying to merge 2 JSON documents using JQ - json

I'm using JQ CLI to merge JSON from document to another. The issue I am facing is that I have select by the value of a property, rather than by a numeric array index
The first file contains a chunk of JSON jqtest.json:
{
"event": [
{
"listen": "test",
"script": {
"exec": [],
"type": "text/javascript"
}
}
]
}
The second file is where I want to merge the JSON into under "accounts" collection.json:
{
"item": [
{
"name": "accounts",
"item": [
{
"name": "Retrieves the collection of Account resources."
}
]
},
{
"name": "accounts mapped",
"item": [
{
"name": "Retrieves the collection of AccountMapped resources."
}
]
}
]
}
What i am trying to do is merge it under "accounts" and under "name": "Retrieves the collection of Account resources." I use the command:
jq -s '
.[0].event += .[1].item |
map(select(.name=="accounts")) |
.[].item
' jqtest.json collection.json
But when executed nothing is outputted. What am doing wrong with JQ or is there another tool i can use to accomplish this?
{
"item": [
{
"name": "accounts",
"item": [
{
"name": "Retrieves the collection of Account resources.",
"event": [
{
"listen": "test",
"script": {
"exec": [],
"type": "text/javascript"
}
}
]
},
{
"name": "accounts mapped",
"item": [
{
"name": "Retrieves the collection of AccountMapped resources."
}
]
}
]
}
]
}

To merge two objects, one can use obj1 + obj2. From this, it follows that obj1 += obj2 can be used to merge an object (obj2) into another existing object (obj1).
Maybe that's what you trying to use. If so, you were missing parens around the expression producing the object to merge into (causing the code to be misparsed), you have the operands to += backwards, you don't actually produce the correct objects on each side of += (or even objects at all), and you didn't narrow down your output (accidentally including jqtest in the output).
Fixed:
jq -s '
( .[1].item[] | select( .name == "accounts" ) | .item[] ) += .[0] | .[1]
' jqtest.json collection.json
Demo on jqplay
I find the following clearer (less mental overhead):
jq -s '
.[0] as $to_insert |
.[1] | ( .item[] | select( .name == "accounts" ) | .item[] ) += $to_insert
' jqtest.json collection.json
Demo
That said, I would avoid slurping in favour of --argfile.
jq --argfile to_insert jqtest.json '
( .item[] | select( .name == "accounts" ) | .item[] ) += $to_insert
' collection.json
Demo on jqplay

Related

How to replace multiple values in json file using jq and returning the whole content

I have a json like this
[{"version": 0.0,"Resources": [ {"TargetService": {"Type": "AWS::ECS::Service","Properties": {"TaskDefinition": "abc","LoadBalancerInfo": {"ContainerName": "def","ContainerPort": 8080}}}} ]}]
My attempt is to replace TaskDefinition key value from "abc" to "123" and ContainerName key value from "def" to "456 in one command and return the whole update json.
This is what i tried
echo $APP_SPEC | jq --arg TASK_DEFINITION "123" '(.[].Resources[].TargetService | select(.Properties)).TaskDefinition |=$TASK_DEFINITION')
But the substistuion is not happening properly and the value gets appended at the end of the josn as below.
Incorrect response:
[ { "version": 0, "Resources": [ { "TargetService": { "Type": "AWS::ECS::Service", "Properties": { "TaskDefinition": "abc", "LoadBalancerInfo": { "ContainerName": "container_name", "ContainerPort": 8080 } }, "TaskDefinition": "123" } } ] } ]
You don't need select, you can specify the path directly:
.[].Resources[].TargetService.Properties.TaskDefinition = "123"
| .[].Resources[].TargetService.Properties.LoadBalancerInfo.ContainerName = "456"
Or group by the top level property:
.[].Resources[].TargetService.Properties |= (
.TaskDefinition |= "123"
| .LoadBalancerInfo.ContainerName |= "456"
)
It's also possible to recursively merge the expected object into your existing object using *=:
.[].Resources[].TargetService.Properties *= {
TaskDefinition: "123",
LoadBalancerInfo: {
ContainerName: "456"
}
}
Output of all three variants:
[
{
"version": 0,
"Resources": [
{
"TargetService": {
"Type": "AWS::ECS::Service",
"Properties": {
"TaskDefinition": "123",
"LoadBalancerInfo": {
"ContainerName": "def",
"ContainerPort": 456
}
}
}
}
]
}
]
Use the update operator |=
.[].Resources[].TargetService.Properties |= (
.TaskDefinition = "123"
| .LoadBalancerInfo.ContainerName = "456"
)
Demo
Note that select only filters its input, it does not descend into the filter criterion. So, if you only want to make the update if the .Properties field exists, use select and descend into it.
( .[].Resources[].TargetService
| select(.Properties).Properties
) |= (
.TaskDefinition = "123"
| .LoadBalancerInfo.ContainerName = "456"
)
Demo
Note that the filter select(.Properties) will produce false if the content of that field, albeit existing, evaluates to null or false. If you want to consider such cases also as "existent", use has in the select filter to test for it.
( .[].Resources[].TargetService
| select(has("Properties")).Properties
) |= (
.TaskDefinition = "123"
| .LoadBalancerInfo.ContainerName = "456"
)
Demo
Both the above solutions works perfectly fine. Thanks #Knittl and #pmf. For others who are stuck with this kind of problem here is my complete command to pass two variables to jq and replace the values in one shot.
echo $APP_SPEC | jq --arg TASK_DEFINITION "$NEW_TASK_DEFINITION_ARN" --arg CONTAINER_NAME "$CONTAINER_NAME" '.[].Resources[].TargetService.Properties |= (.TaskDefinition = $TASK_DEFINITION | .LoadBalancerInfo.ContainerName = $CONTAINER_NAME )'

iterate on key to get the contents of the key

IF I have json that looks like
{
"items": [
{
"name": "issue1",
"spec": {
"config": {
"setting1": abc,
"setting2": {
"name": "xyz"
}
},
"files": {
"name": "cde",
"path": "/home"
},
"program": {
"name": "apache"
}
}
}
]
}
and I want to have iteration at .items[0].spec where the key config,files,program's contents can be shown.. something like
config:
{
"setting1": abc,
"setting2": {
"name": "xyz"
}
}
files:
{
"name": "cde",
"path": "/home"
}
program:
{
"name": "apache"
}
and there might be things more or less than config/files/programs for each items.
I know that I can get the list of keys by jq -r '.items[0].spec| to_entries[].key' but not sure about formating the output and getting the contents of .items[].spec.xxxxx like above.
I also got very close by jq -r '{test: .items[0].spec | with_entries(select(.value != null)) }' but I want to seperate each item per key
Also this got me even closer! but how do you get the json syntax of the object to display uncompressed?
jq -r '.items[].spec | keys[] as $k | "\($k):\n \(.[$k])"'
The "," operator is the magic ingredient you seem to be looking for:
.items[].spec | keys[] as $k | $k, .[$k]

Using jq to fetch key-value pairs from a json file

I am trying to get key#value pairs of JSON file below using jq
{
"STUFF_RELATED1": "STUFF_RELATED1",
"STUFF_RELATED2": "STUFF_RELATED2",
"THINGS": {
"THING_2": {
"details": {
"stuff_branch": "user/dev"
},
"repository": "path/to/repo",
"branch": "master",
"revision": "dsfkes4s34jlis4jsj4lis4sli3"
},
"THING_1": {
"details": {
"stuff_branch": "master"
},
"repository": "path/to/repo",
"branch": "master",
"revision": "dsfkes4s34jlis4jsj4lis4sli3"
}
},
"STUFF": {
"revision": "4u324i324iy32g",
"branch": "master"
}
}
The key#value pair should look like this:
THING_1#dsfkes4s34jlis4jsj4lis4sli3
Currently I have tried this on my own:
jq -r ' .THINGS | keys[] as $k | "($k)#(.[$k].revision)" ' things.json
But it does not give the resul that I really want.:( Thanks in advance!
You need to escape ( :
jq -r ' .THINGS | keys[] as $k | "\($k)#\(.[$k].revision)" ' things.json

Parse JSON and JSON values with jq

I have an API that returns JSON - big blocks of it. Some of the key value pairs have more blocks of JSON as the value associated with a key. jq does a great job of parsing the main JSON levels. But I can't find a way to get it to 'recurse' into the values associated with the keys and pretty print them as well.
Here is the start of one of the JSON returns. Note it is only a small percent of the full return:
{
"code": 200,
"status": "OK",
"data": {
"PlayFabId": "xxxxxxx",
"InfoResultPayload": {
"AccountInfo": {
"PlayFabId": "xxxxxxxx",
"Created": "2018-03-22T19:23:29.018Z",
"TitleInfo": {
"Origination": "IOS",
"Created": "2018-03-22T19:23:29.033Z",
"LastLogin": "2018-03-22T19:23:29.033Z",
"FirstLogin": "2018-03-22T19:23:29.033Z",
"isBanned": false
},
"PrivateInfo": {},
"IosDeviceInfo": {
"IosDeviceId": "xxxxxxxxx"
}
},
"UserVirtualCurrency": {
"GT": 10,
"MB": 70
},
"UserVirtualCurrencyRechargeTimes": {},
"UserData": {},
"UserDataVersion": 15,
"UserReadOnlyData": {
"DataVersion": {
"Value": "6",
"LastUpdated": "2018-03-22T19:48:59.543Z",
"Permission": "Public"
},
"achievements": {
"Value": "[{\"id\":0,\"gamePack\":\"GAME.PACK.0.KK\",\"marblesAmount\":50,\"achievements\":[{\"id\":2,\"name\":\"Correct Round 4\",\"description\":\"Round 4 answered correctly\",\"maxValue\":10,\"increment\":1,\"currentValue\":3,\"valueUnit\":\"unit\",\"awardOnIncrement\":true,\"marbles\":10,\"image\":\"https://www.jamandcandy.com/kissinkuzzins/achievements/icons/sphinx\",\"SuccessKey\":[\"0_3_4_0\",\"0_5_4_0\",\"0_6_4_0\",\"0_7_4_0\",\"0_8_4_0\",\"0_9_4_0\",\"0_10_4_0\"],\"event\":\"Player_answered_round\",\"achieved\":false},{\"id\":0,\"name\":\"Complete
This was parsed using jq but as you can see when you get to the
"achievements": { "Vales": "[{\"id\":0,\"gamePack\":\"GAME.PACK.0.KK\",\"marblesAmount\":50,\
lq does no further parse the value at is also JSON.
Is there a filter I am missing to get it to parse the values as well as the higher level structure?
Is there a filter I am missing ...?
The filter you'll need is fromjson, but it should only be applied to the stringified JSON; consider therefore using |= as illustrated using your fragment:
echo '{"achievements": { "Vales": "[{\"id\":0,\"gamePack\":\"GAME.PACK.0.KK\",\"marblesAmount\":50}]"}}' |
jq '.achievements.Vales |= fromjson'
{
"achievements": {
"Vales": [
{
"id": 0,
"gamePack": "GAME.PACK.0.KK",
"marblesAmount": 50
}
]
}
}
recursively/1
If you want to apply fromjson recursively wherever possible, then recursively is your friend:
def recursively(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | recursively(f) )} )
elif type == "array" then map( recursively(f) )
else try (f as $f | if $f == . then . else ($f | recursively(f)) end) catch $in
end;
This would be applied as follows:
recursively(fromjson)
Example
{a: ({b: "xyzzy"}) | tojson} | tojson
| recursively(fromjson)
yields:
{
"a": {
"b": "xyzzy"
}
}

jq get the value of x based on y in a complex json file

jq strikes again. Trying to get the value of DATABASES_DEFAULT based on the name in a json file that has a whole lot of names and I'm completely lost.
My file looks like the following (output of an aws ecs describe-task-definition) only much more complex; I've stripped this to the most basic example I can where the structure is still intact.
{
"taskDefinition": {
"status": "bar",
"family": "bar2",
"volumes": [],
"taskDefinitionArn": "bar3",
"containerDefinitions": [
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo"
}
],
"name": "baz",
"links": []
},
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo2"
}
],
"name": "boo",
"links": []
}
],
"revision": 1
}
}
I need the value of DATABASES_DEFAULT where the name is baz. Note that there are a lot of keypairs with name, I'm specifically talking about the one outside of environment.
I've been tinkering with this but only got this far before realizing that I don't understand how to access nested values.
jq '.[] | select(.name==DATABASES_DEFAULT) | .value'
which is returning
jq: error: DATABASES_DEFAULT/0 is not defined at <top-level>, line 1:
.[] | select(.name==DATABASES_DEFAULT) | .value
jq: 1 compile error
Obviously this a) doesn't work, and b) even if it did, it's independant of the name value. My thought was to return all the db defaults and then identify the one with baz, but I don't know if that's the right approach.
I like to think of it as digging down into the structure, so first you open the outer layers:
.taskDefinition.containerDefinitions[]
Now select the one you want:
select(.name =="baz")
Open the inner structure:
.environment[]
Select the desired object:
select(.name == "DATABASES_DEFAULT")
Choose the key you want:
.value
Taken together:
parse.jq
.taskDefinition.containerDefinitions[] |
select(.name =="baz") |
.environment[] |
select(.name == "DATABASES_DEFAULT") |
.value
Run it like this:
<infile jq -f parse.jq
Output:
"foo"
The following seems to work:
.taskDefinition.containerDefinitions[] |
select(
select(
.environment[] | .name == "DATABASES_DEFAULT"
).name == "baz"
)
The output is the object with the name key mapped to "baz".
$ jq '.taskDefinition.containerDefinitions[] | select(select(.environment[]|.name == "DATABASES_DEFAULT").name=="baz")' tmp.json
{
"dnsSearchDomains": [],
"environment": [
{
"name": "bar4",
"value": "bar5"
},
{
"name": "bar6",
"value": "bar7"
},
{
"name": "DATABASES_DEFAULT",
"value": "foo"
}
],
"name": "baz",
"links": []
}