I have a requirement where in 2 parameter files needs to be merged to one using Jq
param1.json
[
"name=xyz",
"age=40",
"email=qqqq"
]
param2.json
[
"name=xyz",
"age=42",
"drivingLicense=2761"
]
I need a resultant value to be
[
"name=xyz",
"age=42",
"email=qqqq",
"drivingLicense=2761"
]
When I try to use Jq add jq -s '.[0] + .[1]' param1.json param2.json the resultant
[
"name=xyz",
"age=40",
"email=qqqq",
"name=xyz",
"age=42",
"drivingLicense=2761"
]
I tried using jq '. * input' param1.json param2.json but that is not working either
What is the best way to merge them
TIA
This approach makes use of the circumstance that object field names are unique. On collision, latter items overwrite former ones.
jq -s '[add | with_entries(.key = (.value | .[:index("=")]))[]]'
[
"name=xyz",
"age=42",
"email=qqqq",
"drivingLicense=2761"
]
Demo
Note: Instead of add you can, of course, still use .[0] + .[1] or . + input (the latter without -s).
You can first convert your arrays into objects, then add those objects together; then convert to an array again:
$ jq -s 'map(map(./"="|{(first):.[1:]|join("=")})|add)|add|to_entries|map(join("="))' param1.json param2.json
[
"name=xyz",
"age=42",
"email=qqqq",
"drivingLicense=2761"
]
If your values cannot contain an equal sign, then {(first):.[1:]|join("=")} can be simplified to {(first):last}.
Or merging the arrays to one big array before converting to objects:
add
|map(./"="|{(first):.[1:]|join("=")})
|add
|to_entries
|map(join("="))
Levaraging the fact that this can be reformulated as a grouping problem, you can group by the "key" of your string, then select the last item in each group (A reusable function to build group objects can help but is not required).
$ jq -s 'add | map(./"=") | group_by(first) | map(last|join("="))' param1.json param2.json
[
"age=42",
"drivingLicense=2761",
"email=qqqq",
"name=xyz"
]
Related
I have a json file I am parsing with jq. This is a sample of the file
[{
"key1":{...},
"key2":{...}
}]
[{
"key1":{...},
"key2":{...}
}]
...
each line is a list containing a json (which I know is not technically a json format but jq still works on such a file)
The below jq command works:
cat file.json | jq -r '.[] | [.key1,.key2]'
The above correctly shows:
[
<value_of_key1>,<value_of_key2>
]
[
<value_of_key1>,<value_of_key2>
]
However, I want .key1,.key2 to be dynamic since these keys can change. So I want to pass a variable to jq. Something like:
$KEYS=.key1,.key2
cat file.json | jq -r --arg var "$KEYS" '.[] | [$var]'
But the above is returning the keys themselves:
[
".key1,.key2"
]
[
".key1,.key2"
]
why is this happening? what is the correct command to make this happen?
This answer does not help me. I am not getting any errors as the OP in that question.
Fetching the value of a jq variable doesn't cause it to be executed as jq code.
Furthermore, jq lacks the facility to take a string, compile it as jq code, and evaluate the result. (This is commonly known as eval.)
So, short of a writing a jq parser and evaluator in jq, you will need to impose limits and/or accept a different format.
For example,
keys='[ [ "key1", "childkey" ], [ "key2", "childkey2" ] ]' # JSON
jq --argjson keys "$keys" '.[] | [ getpath( $keys[] ) ]' file.json
or
keys='key1.childkey,key2.childkey2'
jq --arg keys "$keys" '
( ( $keys / "," ) | map( . / "." ) ) as $keys |
.[] | [ getpath( $keys[] ) ]
' file.json
Suppose you have:
cat file
[{
"key1":1,
"key2":2
}]
[{
"key1":1,
"key2":2
}]
You can use a jq command like so:
jq '.[] | [.key1,.key2]' file
[
1,
2
]
[
1,
2
]
You can use -f to execute a filter from a file and nothing keeps you from creating the file separately from the shell variables.
Example:
keys=".key1"
echo ".[] | [${keys}]" >jqf
jq -f jqf file
[
1
]
[
1
]
Or just build the string directly into jq:
# note double " causing string interpolation
jq ".[] | [${keys}]" file
You can use --argjson option and destructuring.
file.json
[{"key1":{"a":1},"key2":{"b":2}}]
[{"key1":{"c":1},"key2":{"d":2}}]
$ in='["key1","key2"]' jq -c --argjson keys "$in" '$keys as [$key1,$key2] | .[] | [.[$key1,$key2]]' file.json
output:
[{"a":1},{"b":2}]
[{"c":1},{"d":2}]
Elaborating on ikegami's answer.
To start with here's my version of the answer:
$ in='key1.a,key2.b'; jq -c --arg keys "$in" '($keys/","|map(./".")) as $paths | .[] | [getpath($paths[])]' <<<$'[{"key1":{"a":1},"key2":{"b":2}}] [{"key1":{"a":3},"key2":{"b":4}}]'
This gives output
[1,2]
[3,4]
Let's try it.
We have input
[{"key1":{"a":1},"key2":{"b":2}}]
[{"key1":{"a":3},"key2":{"b":4}}]
And we want to construct array
[["key1","a"],["key2","b"]]
then use it on getpath(PATHS) builtin to extract values out of our input.
To start with we are given in shell variable with string value key1.a,key2.b. Let's call this $keys.
Then $keys/"," gives
["key1.a","key2.b"]
["key1.a","key2.b"]
After that $keys/","|map(./".") gives what we want.
[["key1","a"],["key2","b"]]
[["key1","a"],["key2","b"]]
Let's call this $paths.
Now if we do .[]|[getpath($paths[])] we get the values from our input equivalent to
[.[] | .key1.a, .key2.b]
which is
[1,2]
[3,4]
I need to be able to produce the following but without having to explicit array's indexes, so that I don't need to know input array's lenght
echo '[{"name":"John", "age":30, "car":null},{"name":"Marc", "age":32, "car":null}]' | jq -r '{(.[0].name):.[0].age,(.[1].name):.[1].age}'
Produces :
{ "John": 30, "Marc": 32}
Use add to merge the objects.
jq '[ .[] | { (.name) : .age } ] | add'
I have a bit of a weird data setup, I have the following json files:
file 1:
[
["04-05-2020",
12
],
["03-05-2020",
16
]
]
file 2:
[
["04-05-2020",
50
],
["03-05-2020",
70
]
]
I want to merge the 2 json files using the Dates (which are not specified as keys) and reassign keys and values to the output, such that the output is something like:
file 1:
[
{date: "04-05-2020",
value1 : 12,
value2 : 50
},
{date: "03-05-2020",
value1 : 16,
value2: 70
}
]
My thoughts are I might have to merge the files together first and do some kind of reduce operation on the dates in the array, but my attempts have so far been unsuccessful. Or perhaps I should be formatting the Array first into Key + Value and then do a jq -s 'add'? I'm actually not sure how to reformat this.
One way of doing it using reduce:
reduce inputs[] as [$date, $value] ({};
if has($date) then
.[$date] += {value2: $value}
else
.[$date] = {$date, value1: $value}
end
) | map(.)
Note that you need to specify -n/--null-input option on the command line in order for inputs to work.
Online demo
If the arrays in file1.json and file2.json are in lockstep as in your example, you could simply write:
.[0] |= map({date: .[0], value1: .[1]})
| .[1] |= map({date: .[0], value2: .[1]})
| transpose
| map(add)
using an invocation along the lines:
jq -s -f program.jq file1.json file2.json
Of course there are many variations.
I have a series of JSON files containing an array of records, e.g.
$ cat f1.json
{
"records": [
{"a": 1},
{"a": 3}
]
}
$ cat f2.json
{
"records": [
{"a": 2}
]
}
I want to 1) extract a single field from each record and 2) output a single array containing all the field values from all input files.
The first part is easy:
jq '.records | map(.a)' f?.json
[
1,
3
]
[
2
]
But I cannot figure out how to get jq to concatenate those output arrays into a single array!
I'm not married to jq; I'll happily use another tool if necessary. But I would love to know how to do this with jq, because it's something I have been trying to figure out for years.
Assuming your jq has inputs (which is true of jq 1.5 and later), it would be most efficient to use it, e.g. along the lines of:
jq -n '[inputs.records[].a]' f*.json
Use -s (or --slurp):
jq -s 'map(.records[].a)' f?.json
You need to use --slurp so that jq will apply its filter on the aggregation of all inputs rather than on each input. When using this option, jq's input will be an array of the inputs which you need to account for.
I would use the following :
jq --slurp 'map(.records | map(.a)) | add' f?.json
We apply your current transformation to each elements of the slurped array of inputs (your previous individual inputs), then we merge those transformed arrays into one with add.
If your input files are large, slurping the file could eat up lots of memory in which you case you can reduce which works in iterative manner, appending the contents of the array .a one object at a time
jq -n 'reduce inputs.records[].a as $d (.; . += [$d])' f?.json
The -n flag is to ensure to construct the output JSON from scratch with the data available from inputs. The reduce function takes the initial value of . which because of the null input would be just null. Then for each of the input objects . += [$d] ensures that the array contents of .a are appended together.
As a compromise between the readability of --slurp and the efficiency of reduce, you can run jq twice. The first is a slightly altered version of your original command, the second slurps the undifferentiated output into a single array.
$ jq '.records[] | .a' f?.json | jq -s
[
1,
3,
2
]
--slurp (-s) key is needed and map() to do so in one shot
$ cat f1.json
{
"records": [
{"a": 1},
{"a": 3}
]
}
$ cat f2.json
{
"records": [
{"a": 2}
]
}
$ jq -s 'map(.records[].a)' f?.json
[
1,
3,
2
]
I have a JSON file with the following format:
[
{
"id": "00001",
"attr": {
"a": "foo",
"b": "bar",
...
}
},
{
"id": "00002",
"attr": {
...
},
...
},
...
]
and a text file with a list of ids, one per line. I'd like to use jq to filter only the records whose ids are mentioned in the text file. I.e. if the list contains "00001", only the first one should be printed.
Note, that I can't simply grep since each record may have an arbitrary number of attributes and sub-attributes.
There are basically two ways to proceed:
read the file of ids from STDIN
read the JSON from STDIN
Both are feasible, but here we illustrate (2) as it leads to a simple but efficient solution.
Suppose the JSON file is named in.json and the list of ids is in a file named ids.txt like so:
00001
00010
Notice that this file has no quotation marks. If it does, then the following can be significantly simplified as shown in the postscript.
The trick is to convert ids.txt into a JSON array. With the above assumption about quotation marks, this can be done by:
jq -R . ids.txt | jq -s .
Assuming a reasonable shell, a simple solution is now at hand:
jq --argjson ids "$(jq -R . ids.txt | jq -s .)" '
map( select( .id as $id | $ids | index($id) ))' in.json
Faster
Assuming your jq has any/2, then a simpler and more efficient solution can be obtaining by defining:
def isin($a): . as $in | any($a[]; $in == .);
The required jq filter is then just:
map( select( .id | isin($ids) ) )
If these two lines of jq are put into a file named select.jq, the required incantation is simply:
jq --argjson ids "$(jq -R . ids.txt | jq -s)" -f select.jq in.json
Postscript
If the index file consists of a stream of valid JSON texts (e.g., strings with quotation marks) and if your jq supports the --slurpfile option, the invocation can be further simplified to:
jq --slurpfile ids ids.txt -f select.jq in.json
Or if you want everything as a one-liner:
jq --slurpfile ids ids.txt 'map(select(.id as $id|any($ids[];$id==.)))' in.json