Rename duplicate Keys in Json array data - json

I have a json data as below.
{
"Data":
[
"User": [
{"Name": "Solomon", "Age":20},
{"Name": "Absolom", "Age":30},
]
"Country": [
{"Name" : "US", "Resident" : "Permanent"},
{"Name" : "UK", "Resident" : "Temporary"}
]]}
There are two tags with same keys,
in Users there is Name key and in Country also i have Name key. I need to preprocess the json file to differentiate the keys. My expected result is below. Tried through awk and sed commands, but i could not find proper solution. Any suggestion would be helpful.
Expected result:
{
"Data":
[
"User": [
{"User_Name": "Solomon", "User_Age":20},
{"User_Name": "Absolom", "User_Age":30},
]
"Country": [
{"Country_Name" : "US", "Country_Resident" : "Permanent"},
{"Country_Name" : "UK", "Country_Resident" : "Temporary"}
]]}
Tag name should be appended to the attribute name.
This is what i have tried,
jq '[.[] | .["User_Name"] = .Name]' file_name.json
But it changes for both the tages User as well as Country

with the permission of the OP, here's a jtc based solution while waiting for the jq's (assuming the input JSON is fixed):
bash $ <file.json jtc -w'<Data>l[:]<L>k<.*>L:<>k' -u'"{L}_{}";' -tc
{
"Data": {
"Country": [
{ "Country_Name": "US", "Country_Resident": "Permanent" },
{ "Country_Name": "UK", "Country_Resident": "Temporary" }
],
"User": [
{ "User_Age": 20, "User_Name": "Solomon" },
{ "User_Age": 30, "User_Name": "Absolom" }
]
}
}
bash $
Explanation of the jtc parameters:
-w'<Data>l[:]<L>k<.*>L:<>k' :
walk path (-w) selects Data label (<Data>l)
and then each of the nested elements ([:]),
and memorizes its key/label into the namespace L (<L>k),
then finds further each labeled element using REGEX label search (<.*>L:)
and finally reinterpret found element's key/label as the value (<>k)
-u'"{L}_{}";':
for each found label (in step 1) update operation (-u) is applied using template
"{L}_{}";', where {L} is interpolated with preserved in the namespace L value and {} is getting interpolated with the currently found label (at the each iteration of the walk path)
the trailing ; (or any other symbol) is required to distinguish the argument of -u from a literal JSON.
-tc is used to display JSON in a semi-compact form.
PS. I'm the creator of jtc unix JSON processing tool. The disclaimer is required by SO.

As originally posted, neither the illustrative input nor the corresponding output is valid JSON, but the following has been tested using JSON based on the shown input:
.Data |= ( (.User |= map(with_entries(.key |= ("User_" + .))))
| (.Country |= map(with_entries(.key |= ("Country_" + .)))) )
Of course, the above may need tweaking depending on the actual requirements, and can be generalized in various ways, e.g. as shown below.
A generalization
.Data |= with_entries( (.key + "_") as $newkey
| .value |= map(with_entries(.key |= ($newkey + .))))

Here is an approach using jq Streaming
fromstream(tostream | .[0] |= if length < 4 then . else .[3]="\(.[1])_\(.[3])" end)
It works by using tostream to convert your input to a stream of arrays
[["Data","Country",0,"Name"],"US"]
[["Data","Country",0,"Resident"],"Permanent"]
[["Data","Country",0,"Resident"]]
[["Data","Country",1,"Name"],"UK"]
[["Data","Country",1,"Resident"],"Temporary"]
[["Data","Country",1,"Resident"]]
[["Data","Country",1]]
[["Data","User",0,"Age"],20]
[["Data","User",0,"Name"],"Solomon"]
[["Data","User",0,"Name"]]
[["Data","User",1,"Age"],30]
[["Data","User",1,"Name"],"Absolom"]
[["Data","User",1,"Name"]]
[["Data","User",1]]
[["Data","User"]]
[["Data"]]
then applying a simple update assignment |= expression to transform the stream into
[["Data","Country",0,"Country_Name"],"US"]
[["Data","Country",0,"Country_Resident"],"Permanent"]
[["Data","Country",0,"Country_Resident"]]
[["Data","Country",1,"Country_Name"],"UK"]
[["Data","Country",1,"Country_Resident"],"Temporary"]
[["Data","Country",1,"Country_Resident"]]
[["Data","Country",1]]
[["Data","User",0,"User_Age"],20]
[["Data","User",0,"User_Name"],"Solomon"]
[["Data","User",0,"User_Name"]]
[["Data","User",1,"User_Age"],30]
[["Data","User",1,"User_Name"],"Absolom"]
[["Data","User",1,"User_Name"]]
[["Data","User",1]]
[["Data","User"]]
[["Data"]]
then reversing the transformation with fromstream.
Try it online!

Related

Merging multiple JSON Lines files into a single JSON object

I'm trying to merge / reduce many JSON objects and somehow I'm not getting the expected result.
I'm only interested in getting all keys, the values and the number of items inside arrays are irrelevant.
file1.json:
{
"customerId": "xx",
"emails": [
{
"address": "james#zz.com",
"customType": "",
"type": "custom"
},
{
"address": "sales#x.com",
"primary": true
},
{
"address": "info#x.com"
}
]
}
{
"id": "654",
"emails": [
{
"address": "peter#x.com",
"primary": true
}
]
}
The desired output is a JSON object with all possible keys from all input objects. The values are irrelevant, any value from any input object is OK. But all keys from input objects must be present in output object:
{
"emails": [
{
"address": "james#zz.com", <--- any existing value works
"customType": "", <--- any existing value works
"type": "custom", <--- any existing value works
"primary": true <--- any existing value works
}
],
"customerId": "xx", <--- any existing value works
"id": "654" <--- any existing value works
}
I tried reducing it, but it misses many of the keys in the array:
$ jq -s 'reduce .[] as $item ({}; . + $item)' file1.json
{
"customerId": "xx",
"emails": [
{
"address": "peter#x.com",
"primary": true
}
],
"id": "654"
}
The structure of the objects contained in file1.json is unknown, so the solution must be agnostic of any keys/values and the solution must not assume any structure or depth.
Is it possible to fix this somehow considering how jq works? Or is it possible to solve this issue using another tool?
PS: For those of you that are curious, this is useful to infer a schema that can be created in a database. Given an arbitrary number of JSON objects with an arbitrary structure, it's easy to create a single JSON squished/merged/fused structure that will "accommodate" all JSON objects.
BigQuery is able to autodetect a schema, but only 500 lines are analyzed to come up with it. This presents problems if objects have different structures past that 500 line mark.
With this approach I can squish a JSON Lines file with 1000000s of objects into one line that can be then imported into BigQuery with the autodetect schema flag and it will work every time since BigQuery only has one line to analyze and this line is the "super-schema" of all the objects. After extracting the autodetected schema I can manually fine tune it to make sure types are correct and then recreate the table specifying my tuned schema:
$ ls -1 users*.json | wc --lines
3672
$ cat users*.json > users-all.json
$ cat users-all.json | wc --lines
146482633
$ jq 'squish' users-all.json > users-all-squished.json
$ cat users-all-squished.json | wc --lines
1
$ bq load --autodetect users users-all-squished.json
$ bq show schema --format=prettyjson users > users-schema.json
$ vi users-schema.json
$ bq rm --table users
$ bq mk --table users --schema=users-schema.json
$ bq load users users-all.json
[Some options are missing or changed for readability]
Here is a solution that produces the expected result in the sample example, and seems to meet all the stated requirements. It is similar to one proposed by #pmf on this page.
jq -n --stream '
def squish: map(if type == "number" then 0 else . end);
reduce (inputs | select(length==2)) as [$p, $v] ({}; setpath($p|squish; $v))
'
Output
For the example given in the Q, the output is:
{
"customerId": "xx",
"emails": [
{
"address": "peter#x.com",
"customType": "",
"type": "custom",
"primary": true
}
],
"id": "654"
}
As #peak has pointed out, some aspects are underspecified. For instance, what should happen with .customerId and .id? Are they always the same across all files (as suggested by the sample files provided)? Do you want the items of the .emails array just thrown into one large array, or do you want to have them "merged" by some criteria (e.g. by a common value in their .address field)? Here are some stubs to start from:
Simply concatenate the .emails arrays and take all other parts from the first file:
jq 'reduce inputs as $in (.; .emails += $in.emails)' file*.json
# or simpler
jq '.emails += [inputs.emails[]]' file*.json
Demo Demo
{
"emails": [
{
"address": "cc#xx.com"
},
{
"address": "james#zz.com",
"customType": "",
"type": "custom"
},
{
"address": "james#x.com"
},
{
"address": "sales#x.com",
"primary": true
},
{
"address": "info#x.com"
},
{
"address": "james#x.com"
},
{
"address": "sales#x.com",
"primary": true
},
{
"address": "info#x.com"
}
],
"customerId": "xx",
"id": "654"
}
Merge the objects in the .emails array by a common value in their .address field, with latter values overwriting former values for other fields with colliding names, and discard all other parts from the files:
jq -n 'reduce inputs.emails[] as $e ({}; .[$e.address] += $e) | map(.)' file*.json
Demo
[
{
"address": "cc#xx.com"
},
{
"address": "james#zz.com",
"customType": "",
"type": "custom"
},
{
"address": "james#x.com"
},
{
"address": "sales#x.com",
"primary": true
},
{
"address": "info#x.com"
}
]
If you are only interested in a list of unique field names for a given address, regardless of the counts and values used, you can also go with:
jq -n '
reduce inputs.emails[] as $e ({}; .[$e.address][$e | keys_unsorted[]] = 1)
| map_values(keys)
'
Demo
{
"cc#xx.com": [
"address"
],
"james#zz.com": [
"address",
"customType",
"type"
],
"james#x.com": [
"address"
],
"sales#x.com": [
"address",
"primary"
],
"info#x.com": [
"address"
]
}
The structure of the objects contained in file1.json is unknown, so the solution must be agnostic of any keys/values and the solution must not assume any structure or depth.
You can use the --stream flag to break down the structure into an array of paths and values, discard the values part and make the paths unique:
jq --stream -nc '[inputs[0]] | unique[]' file*.json
["customerId"]
["emails"]
["emails",0,"address"]
["emails",0,"customType"]
["emails",0,"primary"]
["emails",0,"type"]
["emails",1,"address"]
["emails",2]
["emails",2,"address"]
["emails",2,"primary"]
["emails",3]
["emails",3,"address"]
["id"]
Trying to build a representation of this, similar to any of the input files, comes with a lot of caveats. For instance, how would you represent in a single structure if one file had .emails as an array of objects, and another had .emails as just an atomic value, say, a string. You would not be able to represent this plurality without introducing new, possibly ambiguous structures (e.g. putting all possibilities into an array).
Therefore, having a list of paths could be a fair compromise. Judging by your desired output, you want to focus more on the object structure, so you could further reduce complexity by discarding the array indices. Depending on your use case, you could replace them with a single value to retain the information of the presence of an array, or discard them entirely:
jq --stream -nc '[inputs[0] | map(numbers = 0)] | unique[]' file*.json
["customerId"]
["emails"]
["emails",0]
["emails",0,"address"]
["emails",0,"customType"]
["emails",0,"primary"]
["emails",0,"type"]
["id"]
jq --stream -nc '[inputs[0] | map(strings)] | unique[]' file*.json
["customerId"]
["emails"]
["emails","address"]
["emails","customType"]
["emails","primary"]
["emails","type"]
["id"]
The following program meets these two key requirements:
"all keys from input objects must be present in output object";
"the solution must be agnostic of any keys/values and the solution must not assume any structure or depth."
The approach is the same as one suggested by #pmf, and for the example given in the Q, produces results that are very similar to the one that is shown:
jq -n --stream '
def squish: map(select(type == "string"));
reduce (inputs | select(length==2)) as [$p, $v] ({};
setpath($p|squish; $v))
'
With the given input, this produces:
{
"customerId": "xx",
"emails": {
"address": "peter#x.com",
"customType": "",
"type": "custom",
"primary": true
},
"id": "654"
}

Merge and Sort JSON using JQ

I have a file containing the following structure and unknown number of results:
{
"results": [
[
{
"field": "AccountID",
"value": "5177497"
},
{
"field": "Requests",
"value": "50900"
}
],
[
{
"field": "AccountID",
"value": "pro"
},
{
"field": "Requests",
"value": "251"
}
]
],
"statistics": {
"Matched": 51498,
"Scanned": 8673577,
"ScannedByte": 2.72400814E10
},
"status": "HOLD"
}
{
"results": [
[
{
"field": "AccountID",
"value": "5577497"
},
{
"field": "Requests",
"value": "51900"
}
],
"statistics": {
"Matched": 51498,
"Scanned": 8673577,
"ScannedByte": 2.72400814E10
},
"status": "HOLD"
}
There are multiple such results which are indexed as an array with the results folder. They are not seperated by a comma.
I am trying to just print The "AccountID" sorted by "Requests" in ZSH using jq. I have tried flattening them and using:
jq -r '.results[][0] |.value ' filename
jq -r '.results[][1] |.value ' filename
To get the Account ID and Requests seperately and sorting them. I don't think bash has a dictionary that can be used. The problem lies in the file as the Field and value are not key value pair but are both pairs. Therefore extracting them using the above two lines into seperate arrays and sorting by the second array seems a bit too long. I was wondering if there is a way to combine both the operations.
The other way is to combine it all to a string and sort it in ascending order. Python would probably have the best solution but the code requires to be a zsh or bash script.
Solutions that use sed, jq or any other ZSH supported compilers are welcome. If there is a way to create a dictionary in bash, please do let me know.
The projectd output requirement is just the Account ID vs Request Number.
5577497 has 51900 requests
5177497 has 50900 requests
pro has 251 requests
If you don't mind learning a little jq, it will probably be best to write a small jq program to do what you want.
To get you started, consider the following jq program, which assumes your input is a stream of valid JSON objects with a "results" key similar to your sample:
[inputs | .results[] | map( { (.field) : .value} ) | add]
After making minor changes to your input so that it consists of valid JSON objects, an invocation of jq with the -n option produces an array of AccountID/Requests objects:
[
{
"AccountID": "5177497",
"Requests": "50900"
},
{
"AccountID": "pro",
"Requests": "251"
},
{
"AccountID": "5577497",
"Requests": "51900"
}
]
You could (for example) now use jq's group_by to group these objects by AccountID, and thereby produce the result you want.
jq -S '.results[] | map( { (.field) : .value} ) | add' query-results-aggregate \
| jq -s -c 'group_by(.number_of_requests) | .[]'
This does the trick. Thanks to peak for the guidance.

Create object from stream of objects keyed by a property in jq

This is a question about the command line json processor jq. It is not about javascript or jQuery or anything else with js and qs in its name :-)
I have input data like:
{ "id": "person1", "name": "wilma", "age": "quite old"}
{ "id": "person2", "name": "fred"}
{ "id": "person1", "name": "betty", "x": "extra"}
I want output like this:
{
"person1": { "name": "betty", "age": "quite old", "x": "extra" },
"person2": { "name": "fred" }
}
I have tried various things!
e.g.
jq -s '.[] | { (.id) : . }' <data
gives
{ "person1": { "id": "person1", "name": "wilma", "age": "quite old" }}
{ "person2": { "id": "person2", "name": "fred" }}
{ "person1": { "id": "person1", "name": "betty", "x": "extra" }}
Which is sort of there, except it's outputting a stream of objects instead of just one. I need to merge all those objects together.
jqplay.org example
I've also tried using group_by(.id)[]|add which merges each item but still results in a stream. https://jqplay.org/s/lh6QUQ0DO4
You could tweak your attempt as follows:
jq -s 'map({ (.id) : . }) | add' <data
However, it would be more efficient to use inputs and reduce with the -n command-line option instead of -s.
Of course, using this approach runs the risk of collisions.
You might also want to add del(.id)
Ah! I've got it! Or I've got one solution - please post if there's a better way.
jq -s '[group_by(.id)[]| add | { (.id) : . } ]|add' <data
https://jqplay.org/s/BfAdRBZUMW
group_by groups the inputs by their .id value and produces an array of arrays - the inner arrays are the values that match on id.
for each group the inner arrays are passed to add which, because the things in the inner arrays are objects, merges them.
That leaves a 2 item array. We feed that to an object constructor which plucks the id as the key and the whole item as the value. This still leaves an array of items.
the outer [] (starts at start of pattern) says take all those and feed it to add (again), which merges the final objects created in (3).
It works, but there may be a cleaner way.
EDIT
This is uglier but produces the same result and is ~24% faster on a 9MB dataset.
jq -s 'reduce [.[]|{ (.id) : . }][] as $item ({}; . * $item )' <data
This uses reduce <list> as <$var> (<initiation>; <iteration>) starting with an empty object {} and using the merge operator * starting from the incoming item . to create the output. I'm surprised it's faster, but I understand that group_by does a sort, so I guess that's an additional time cost.

Getting only desired properties from nested array values with jq

The structure I ultimately want would be:
{
"catalog": [
{
"name": "X",
"catalog": [
{ "name": "Y", "uniqueId": "Z" },
{ "name": "Q", "uniqueId": "B" }
]
}
]
}
This is what the existing structure looks like except there are many other properties at each level (https://gist.github.com/ajcrites/e0e0ca4ca3a08ff2dc401ec872e6094c). I just want to filter those out and get a JSON format that looks specifically like this.
I have started out with: jq '.catalog', but this returns only the array. I still want the catalog property name there. I can do this with jq '{catalog: .catalog[]}, but this prints out each catalog object individually which makes the whole output invalid JSON. I still want the properties to be in the array. Is there a way to filter specific property key-values within arrays using jq?
The following transforms the given input to the desired output and may well be what you want:
{catalog}
| .catalog |= map( {name, catalog} )
| .catalog[].catalog |= map( {name, uniqueId} )
| .catalog |= .[0:1]
However, it's not clear to me that this is really what you want, as you don't discuss the duplication in the given JSON input. So maybe you don't really want the last line in the above, or maybe you want duplicates to be handled in some other way, or ....
Anyway, the trick to keeping things simple here is to use |=.
An alternative approach would be to use del to delete the unwanted properties (rather than selecting the ones you want), but in the present case, that would be (at best) tedious.
You could start by using tostream to convert your sample.json
into a stream of [path, value] arrays as you can see by running
jq -c tostream sample.json
This will generate
[["catalog",0,"catalog",0,"name"],"Y"]
[["catalog",0,"catalog",0,"prop11"],""]
[["catalog",0,"catalog",0,"uniqueId"],"Z"]
[["catalog",0,"catalog",0,"uniqueId"]]
[["catalog",0,"catalog",1,"name"],"Y"]
[["catalog",0,"catalog",1,"prop11"],""]
...
reduce and setpath can be used to convert back into the
original form with a filter such as:
reduce (tostream|select(length==2)) as [$p,$v] (
{};
setpath($p;$v)
)
Adding conditionals makes it easy to omit properties at any level.
For example the following removes leaf attributes starting with "prop":
reduce (tostream|select(length==2)) as [$p,$v] (
{};
if $p[-1]|startswith("prop")
then .
else setpath($p;$v)
end
)
With your sample.json this produces
{
"catalog": [
{
"catalog": [
{
"name": "Y",
"uniqueId": "Z"
},
{
"name": "Y",
"uniqueId": "Z"
}
],
"name": "X"
},
{
"catalog": [
{
"name": "Y",
"uniqueId": "Z"
},
{
"name": "Y",
"uniqueId": "Z"
}
],
"name": "X"
}
]
}
If the goal is to remove certain properties, then one could do so using walk/1. For example, to remove properties whose names start with "prop":
walk(if type == "object"
then with_entries(select(.key|startswith("prop") | not))
else . end)
The same approach would also be applicable if the focus is on retaining certain properties, e.g.:
walk(if type == "object"
then with_entries(select(.key == "name" or .key == "uniqueId" or .key == "catalog"))
else . end)
You could build up a file that contains paths into the json (expressed as arrays) that you want to keep. Then filter out values that do not fit in those paths.
paths.json:
["catalog","name"]
["catalog","catalog","name"]
["catalog","catalog","uniqueId"]
Then filter values based on their paths. Using streams is a great way to go for this since it gives you access to these paths directly:
$ jq --slurpfile paths paths.json '
def keep_path($path): any($paths[]; . == [$path[] | select(strings)]);
fromstream(tostream | select(length == 1 or keep_path(.[0])))
' input.json

Using jq to list keys in a JSON object

I have a hierarchically deep JSON object created by a scientific instrument, so the file is somewhat large (1.3MB) and not readily readable by people. I would like to get a list of keys, up to a certain depth, for the JSON object. For example, given an input object like this
{
"acquisition_parameters": {
"laser": {
"wavelength": {
"value": 632,
"units": "nm"
}
},
"date": "02/03/2525",
"camera": {}
},
"software": {
"repo": "github.com/username/repo",
"commit": "a7642f",
"branch": "develop"
},
"data": [{},{},{}]
}
I would like an output like such.
{
"acquisition_parameters": [
"laser",
"date",
"camera"
],
"software": [
"repo",
"commit",
"branch"
]
}
This is mainly for the purpose of being able to enumerate what is in a JSON object. After processing the JSON objects from the instrument begin to diverge: for example, some may have a field like .frame.cross_section.stats.fwhm, while others may have .sample.species, so it would be convenient to be able to interrogate the JSON object on the command line.
The following should do exactly what you want
jq '[(keys - ["data"])[] as $key | { ($key): .[$key] | keys }] | add'
This will give the following output, using the input you described above:
{
"acquisition_parameters": [
"camera",
"date",
"laser"
],
"software": [
"branch",
"commit",
"repo"
]
}
Given your purpose you might have an easier time using the paths builtin to list all the paths in the input and then truncate at the desired depth:
$ echo '{"a":{"b":{"c":{"d":true}}}}' | jq -c '[paths|.[0:2]]|unique'
[["a"],["a","b"]]
Here is another variation uing reduce and setpath which assumes you have a specific set of top-level keys you want to examine:
. as $v
| reduce ("acquisition_parameters", "software") as $k (
{}; setpath([$k]; $v[$k] | keys)
)