Merging two json files with jq - json

I have the following json files:
File1:
[
{
"id": 1,
"name": "serviceName",
"owner": {
"id": 1,
"name": "Nicole"
}
}
]
and File2:
[
{
"id": 1,
"name": "Nicole",
"email": "nicole#email.com"
}
]
I would like to have them merged like this:
[
{
"id": 1,
"name": "serviceName",
"owner": {
"id": 1,
"name": "Nicole",
"email": "nicole#email.com"
}
}
]
I'm trying the approach from here and try to use the following:
jq --argfile new file2.json '
($new | INDEX(.ID)) as $dict
| .owner
|= (if $dict[.ID] then . + $dict[.ID] else . end)
' file1.json
But that just results in an error.
Can anyone maybe provide me with some tips?

Your approach fails because of the followings:
Field names are case-sensitive. Having {"id": 1}, use .id, not .ID.
Field names are strings. Having {"id": 1} and INDEX(.id) as $dict, use $dict[.id | tostring] or $dict[.id | #text] or $dict["\(.id)"] to convert the number 1 into the string "1" in the object index.
Your files contain arrays. While INDEX defaults to read .[] from file2.json, you need to do it for file1.json yourself.
jq --argfile new file2.json '
($new | INDEX(.id)) as $dict | .[].owner |= (
if $dict[.id | #text] then . + $dict[.id | #text] else . end
)
' file1.json
[
{
"id": 1,
"name": "serviceName",
"owner": {
"id": 1,
"name": "Nicole",
"email": "nicole#email.com"
}
}
]
As a suggestion, you could also employ JOIN to merge on a given key:
jq '
JOIN(
INDEX(input[]; .id); .[]; .id | #text; .[0].owner += .[1] | .[0]
)
' file1.json file2.json
[
{
"id": 1,
"name": "serviceName",
"owner": {
"id": 1,
"name": "Nicole",
"email": "nicole#email.com"
}
}
]
Demo

Related

Use JQ to create new object where the key comes from one object and the value comes from another

I have the following input:
{
"Columns": [
{
"email": 123,
"name": 456,
"firstName": 789,
"lastName": 450,
"admin": 900,
"licensedSheetCreator": 617,
"groupAdmin": 354,
"resourceViewer": 804,
"id": 730,
"status": 523,
"sheetCount": 298
}
]
}
{
"Users": [
{
"email": "abc#def.com",
"name": "Abc Def",
"firstName": "Abc",
"lastName": "Def",
"admin": false,
"licensedSheetCreator": true,
"groupAdmin": false,
"resourceViewer": true,
"id": 521,
"status": "ACTIVE",
"sheetCount": 0
},
{
"email": "aaa#bbb.com",
"name": "Aaa Bob",
"firstName": "Aaa",
"lastName": "Bob",
"admin": false,
"licensedSheetCreator": true,
"groupAdmin": false,
"resourceViewer": false,
"id": 352,
"status": "ACTIVE",
"sheetCount": 0
}
]
}
I need to change the key for all key value pairs in users to match the value in Columns, like so:
{
"Columns": [
{
"email": 123,
"name": 456,
"firstName": 789,
"lastName": 450,
"admin": 900,
"licensedSheetCreator": 617,
"groupAdmin": 354,
"resourceViewer": 804,
"id": 730,
"status": 523,
"sheetCount": 298
}
]
}
{
"Users": [
{
123: "abc#def.com",
456: "Abc Def",
789: "Abc",
450: "Def",
900: false,
617: true,
354: false,
804: true,
730: 521,
523: "ACTIVE",
298: 0
},
{
123: "aaa#bbb.com",
456: "Aaa Bob",
789: "Aaa",
450: "Bob",
900: false,
617: true,
354: false,
804: false,
730: 352,
523: "ACTIVE",
298: 0
}
]
}
I don't mind if I update the Users array or create a new array of objects.
I have tried several combinations of with entries, to entries, from entries, trying to search for keys using variables but the more I dive into it, the more confused I get.
Elements of a stream are processed independently. So we have to change the input.
We could group the stream elements into an array. For an input stream, this can be achieved using --slurp/-s.[1]
jq -s '
( .[0].Columns[0] | map_values( tostring ) ) as $map |
(
.[0],
(
.[1:][] |
.Users[] |= with_entries(
.key = $map[ .key ]
)
)
)
'
Demo on jqplay
Alternatively, we could use --null-input/-n in conjunction with input and/or inputs to read the input.
jq -n '
input |
( .Columns[0] | map_values( tostring ) ) as $map |
(
.,
(
inputs |
.Users[] |= with_entries(
.key = $map[ .key ]
)
)
)
'
Demo on jqplay
Note that your desired output isn't valid JSON. Object keys must be strings. So the above produces a slightly different document than requested.
Note that I assumed that .Columns is always an array of one exactly one element. This is a nonsense assumption, but it's the only way the question makes sense.
For a stream the code generates, you could place the stream generator in an array constructor ([]). reduce can also be used to collect from a stream. For example, map( ... ) can be written as [ .[] | ... ] and as reduce .[] as $_ ( []; . + [ $_ | ... ] ).
The following has the merit of simplicity, though it does not sort the keys.
It assumes jq is invoked with the -n option and of course produces a stream of valid JSON objects:
input
| . as $Columns
| .Columns[0] as $dict
| input # Users
| .Users[] |= with_entries(.key |= ($dict[.]|tostring))
| $Columns, .
If having the keys sorted is important, then you could easily add suitable code to do that; alternatively, if you don't mind having the keys of all objects sorted, you could use the -S command-line option.

Trying to merge 2 JSON documents using JQ

I'm using JQ CLI to merge JSON from document to another. The issue I am facing is that I have select by the value of a property, rather than by a numeric array index
The first file contains a chunk of JSON jqtest.json:
{
"event": [
{
"listen": "test",
"script": {
"exec": [],
"type": "text/javascript"
}
}
]
}
The second file is where I want to merge the JSON into under "accounts" collection.json:
{
"item": [
{
"name": "accounts",
"item": [
{
"name": "Retrieves the collection of Account resources."
}
]
},
{
"name": "accounts mapped",
"item": [
{
"name": "Retrieves the collection of AccountMapped resources."
}
]
}
]
}
What i am trying to do is merge it under "accounts" and under "name": "Retrieves the collection of Account resources." I use the command:
jq -s '
.[0].event += .[1].item |
map(select(.name=="accounts")) |
.[].item
' jqtest.json collection.json
But when executed nothing is outputted. What am doing wrong with JQ or is there another tool i can use to accomplish this?
{
"item": [
{
"name": "accounts",
"item": [
{
"name": "Retrieves the collection of Account resources.",
"event": [
{
"listen": "test",
"script": {
"exec": [],
"type": "text/javascript"
}
}
]
},
{
"name": "accounts mapped",
"item": [
{
"name": "Retrieves the collection of AccountMapped resources."
}
]
}
]
}
]
}
To merge two objects, one can use obj1 + obj2. From this, it follows that obj1 += obj2 can be used to merge an object (obj2) into another existing object (obj1).
Maybe that's what you trying to use. If so, you were missing parens around the expression producing the object to merge into (causing the code to be misparsed), you have the operands to += backwards, you don't actually produce the correct objects on each side of += (or even objects at all), and you didn't narrow down your output (accidentally including jqtest in the output).
Fixed:
jq -s '
( .[1].item[] | select( .name == "accounts" ) | .item[] ) += .[0] | .[1]
' jqtest.json collection.json
Demo on jqplay
I find the following clearer (less mental overhead):
jq -s '
.[0] as $to_insert |
.[1] | ( .item[] | select( .name == "accounts" ) | .item[] ) += $to_insert
' jqtest.json collection.json
Demo
That said, I would avoid slurping in favour of --argfile.
jq --argfile to_insert jqtest.json '
( .item[] | select( .name == "accounts" ) | .item[] ) += $to_insert
' collection.json
Demo on jqplay

Using jq to fetch key-value pairs from a json file

I am trying to get key#value pairs of JSON file below using jq
{
"STUFF_RELATED1": "STUFF_RELATED1",
"STUFF_RELATED2": "STUFF_RELATED2",
"THINGS": {
"THING_2": {
"details": {
"stuff_branch": "user/dev"
},
"repository": "path/to/repo",
"branch": "master",
"revision": "dsfkes4s34jlis4jsj4lis4sli3"
},
"THING_1": {
"details": {
"stuff_branch": "master"
},
"repository": "path/to/repo",
"branch": "master",
"revision": "dsfkes4s34jlis4jsj4lis4sli3"
}
},
"STUFF": {
"revision": "4u324i324iy32g",
"branch": "master"
}
}
The key#value pair should look like this:
THING_1#dsfkes4s34jlis4jsj4lis4sli3
Currently I have tried this on my own:
jq -r ' .THINGS | keys[] as $k | "($k)#(.[$k].revision)" ' things.json
But it does not give the resul that I really want.:( Thanks in advance!
You need to escape ( :
jq -r ' .THINGS | keys[] as $k | "\($k)#\(.[$k].revision)" ' things.json

jq: extract a specific key from one object to another

I have two JSON files.
file1.json:
{
"Fruits": [
{
"name": "Apple",
"something_else": 123,
"id": 652090
},
{
"name": "Orange",
"something_else": 456,
"id": 28748
}
]}
file2.json:
{
"Fruits": [
{
"weight": 5,
"id": 652090
},
{
"weight": 7,
"id": 28748
}
]}
I want to combine objects from both files if they have a common key 'id', but to extract only 'name' property from file1. How do I do that using jq?
This is what I want to get:
{
"Fruits": [
{
"name": "Apple",
"weight": 5,
"id": 652090
},
{
"name": "Orange",
"weight": 7,
"id": 28748
},
]}
Combine Fruits arrays, group it by id, select groups with 2 elements because we want fruits present in both files. For each selected group; add name field from first group element to second, and collect results in an array.
jq -n '[inputs.Fruits[]]
| reduce (group_by(.id)[] | select(length==2)) as $f
([]; . + [$f[1] + ($f[0] | {name})])' file1.json file2.json
Note that the order files are given on the command line is important, the file with names should be given before the other.
Combining objects with same id and extracting a subset of fields is way much easier though:
jq -n '[inputs.Fruits[]]
| group_by(.id)
| map(select(length==2) | add | {name, id, weight})
' file1.json file2.json
There's plenty of ways this could be constructed. Here's another way:
$ jq '.Fruits |= (. + input.Fruits | [group_by(.id)[] | add | {name,weight,id}])' \
file1.json file2.json
{
"Fruits": [
{
"name": "Orange",
"weight": 7,
"id": 28748
},
{
"name": "Apple",
"weight": 5,
"id": 652090
}
]
}

How to format a csv file using json data?

I have a json file that I need to convert to a csv file, but I am a little wary of trusting a json-to-csv converter site as the outputted data seems to be incorrect... so I was hoping to get some help here!
I have the following json file structure:
{
"GroupName": "GrpName13",
"Number": 3,
"Notes": "Test Group ",
"Units": [
{
"UnitNumber": "TestUnit13",
"DataSource": "Factory",
"ContractNumber": "TestContract13",
"CarNumber": "2",
"ControllerTypeMessageId" : 4,
"NumberOfLandings": 4,
"CreatedBy": "user1",
"CommissionModeMessageId": 2,
"Details": [
{
"DetailName": "TestFloor13",
"DetailNumber": "5"
}
],
"UnitDevices": [
{
"DeviceTypeMessageId": 1,
"CreatedBy": "user1"
}
]
}
]
}
The issue I think Im seeing is that the converters seem to not be able to comprehend the many nested data values. And the reason I think the converters are wrong is because when I try to convert back to json using them, I dont receive the same structure.
Does anyone know how to manually format this json into csv format, or know of a reliable converter than can handle nested values?
Try
www.json-buddy.com/convert-json-csv-xml.htm
if not working for you then you can try this tool
http://download.cnet.com/JSON-to-CSV/3000-2383_4-76680683.html
should be helpful!
I have tried your json on this for url:
http://www.convertcsv.com/json-to-csv.htm
As a result:
UnitNumber,DataSource,ContractNumber,CarNumber,ControllerTypeMessageId,NumberOfLandings,CreatedBy,CommissionModeMessageId,Details/0/DetailName,Details/0/DetailNumber,UnitDevices/0/DeviceTypeMessageId,UnitDevices/0/CreatedBy
TestUnit13,Factory,TestContract13,2,4,4,user1,2,TestFloor13,5,1,user1
Because it could save the path of the key,like the 'DeviceTypeMessageId' in list 'UnitDevices': it will named the columns name with 'UnitDevices/0/DeviceTypeMessageId', this could avoid the same name mistake, so you can get the columns name by its converter rules.
Hope helpful.
Here is a solution using jq
If the file filter.jq contains
def denormalize:
def headers($p):
keys_unsorted[] as $k
| if .[$k]|type == "array" then (.[$k]|first|headers("\($p)\($k)_"))
else "\($p)\($k)"
end
;
def setup:
[
keys_unsorted[] as $k
| if .[$k]|type == "array" then [ .[$k][]| setup ]
else .[$k]
end
]
;
def iter:
if length == 0 then []
elif .[0]|type != "array" then
[.[0]] + (.[1:] | iter)
else
(.[0][] | iter) as $x
| (.[1:] | iter) as $y
| [$x[]] + $y
end
;
[ headers("") ], (setup | iter)
;
denormalize | #csv
and data.json contains (note extra samples added)
{
"GroupName": "GrpName13",
"Notes": "Test Group ",
"Number": 3,
"Units": [
{
"CarNumber": "2",
"CommissionModeMessageId": 2,
"ContractNumber": "TestContract13",
"ControllerTypeMessageId": 4,
"CreatedBy": "user1",
"DataSource": "Factory",
"Details": [
{
"DetailName": "TestFloor13",
"DetailNumber": "5"
}
],
"NumberOfLandings": 4,
"UnitDevices": [
{
"CreatedBy": "user1",
"DeviceTypeMessageId": 1
},
{
"CreatedBy": "user10",
"DeviceTypeMessageId": 10
}
],
"UnitNumber": "TestUnit13"
},
{
"CarNumber": "99",
"CommissionModeMessageId": 99,
"ContractNumber": "Contract99",
"ControllerTypeMessageId": 99,
"CreatedBy": "user99",
"DataSource": "Another Factory",
"Details": [
{
"DetailName": "TestFloor99",
"DetailNumber": "99"
}
],
"NumberOfLandings": 99,
"UnitDevices": [
{
"CreatedBy": "user99",
"DeviceTypeMessageId": 99
}
],
"UnitNumber": "Unit99"
}
]
}
then the command
jq -M -r -f filter.jq data.json
will produce
"GroupName","Notes","Number","Units_CarNumber","Units_CommissionModeMessageId","Units_ContractNumber","Units_ControllerTypeMessageId","Units_CreatedBy","Units_DataSource","Units_Details_DetailName","Units_Details_DetailNumber","Units_NumberOfLandings","Units_UnitDevices_CreatedBy","Units_UnitDevices_DeviceTypeMessageId","Units_UnitNumber"
"GrpName13","Test Group ",3,"2",2,"TestContract13",4,"user1","Factory","TestFloor13","5",4,"user1",1,"TestUnit13"
"GrpName13","Test Group ",3,"2",2,"TestContract13",4,"user1","Factory","TestFloor13","5",4,"user10",10,"TestUnit13"
"GrpName13","Test Group ",3,"99",99,"Contract99",99,"user99","Another Factory","TestFloor99","99",99,"user99",99,"Unit99"