I have several json files I want to combine. Some are arrays of objects and some are single objects. I want to effectively concatenate all of this into a single array.
For example:
[
{ "name": "file1" }
]
{ "name": "file2" }
{ "name": "file3" }
And I want to end up with:
[
{ "name": "file1" }
{ "name": "file2" },
{ "name": "file3" },
]
How can I do this using jq or similar?
The following illustrates an efficient way to accomplish the required task:
jq -n 'reduce inputs as $in (null;
. + if $in|type == "array" then $in else [$in] end)
' $(find . -name '*.json') > combined.json
The -n command-line option is necessary to avoid skipping the first file.
This did it:
jq -n '[inputs] | add' $(find . -name '*.json') > combined.json
Related
Could you please assist me to on how I can merge two json variables in bash to get the desired output mentioned below {without manually lopping over .data[] array} ? I tired echo "${firstJsonoObj} ${SecondJsonoObj}" | jq -s add but it didn't parse through the array.
firstJsonoObj='{"data" :[{"id": "123"},{"id": "124"}]}'
SecondJsonoObj='{"etag" :" 234324"}'
desired output
{"data" :[{"id": "123", "etag" :" 234324"},{"id": "124", "etag" :" 234324"}]}
Thanks in advance!
You can append to each data element using +=:
#!/bin/bash
firstJsonoObj='{"data" :[{"id": "123"},{"id": "124"}]}'
SecondJsonoObj='{"etag" :" 234324"}'
jq -c ".data[] += $SecondJsonoObj" <<< "$firstJsonoObj"
Output:
{"data":[{"id":"123","etag":" 234324"},{"id":"124","etag":" 234324"}]}
Please don't use double quotes to inject data from shell into code. jq provides the --arg and --argjson options to do that safely:
#!/bin/bash
firstJsonoObj='{"data" :[{"id": "123"},{"id": "124"}]}'
SecondJsonoObj='{"etag" :" 234324"}'
jq --argjson x "$SecondJsonoObj" '.data[] += $x' <<< "$firstJsonoObj"
# or
jq --argjson a "$firstJsonoObj" --argjson b "$SecondJsonoObj" -n '$a | .data[] += $b'
{
"data": [
{
"id": "123",
"etag": " 234324"
},
{
"id": "124",
"etag": " 234324"
}
]
}
jq -s add will not work because you want to add the second document to a deeper level within the first. Use .data[] += input (without -s), with . acessing the first and ìnput accessing the second input:
echo "${firstJsonoObj} ${SecondJsonoObj}" | jq '.data[] += input'
Or, as bash is tagged, use a Heredoc:
jq '.data[] += input' <<< "${firstJsonoObj} ${SecondJsonoObj}"
Output:
{
"data": [
{
"id": "123",
"etag": " 234324"
},
{
"id": "124",
"etag": " 234324"
}
]
}
Demo
I'm trying to use jq to iterate over some delimited text files, and generate objects from the rows.
I also want to add some "static" objects (json shell variable in the example below) to the generated results.
I've come up with the below solution, which does produce the output I want. But, because I'm not very confident in jq, every time I solve a problem with it, it feels like a monkey banging on a typewriter rather than a carefully crafted answer. So, I'm imaginging this could be incorrect.
data.txt
apple|fruit
Tesla|car
sparrow|bird
Test (bash shell):
$ json='[
{ "object": "love", "type": "emotion" },
{ "object": "Ukraine", "type": "country" }
]'
$ jq --slurp --raw-input --argjson extra "$json" '
split("\n") |
map(select(length>0)) |
map(split("|") | {
object: .[0],
type: .[1]
}) as $data |
$data + $extra' data.txt
Output:
[
{
"object": "apple",
"type": "fruit"
},
{
"object": "Tesla",
"type": "car"
},
{
"object": "sparrow",
"type": "bird"
},
{
"object": "love",
"type": "emotion"
},
{
"object": "Ukraine",
"type": "country"
}
]
Is this efficient?
I don't know if it's more efficient but you could shorten the code using --raw-input or -R without --slurp or -s to linewise read in a stream of raw text (no need to split by newlines), the / operator to do the "column" splitting within a line, and reduce to successively build up your final structure, starting with your "static" data.
jq -Rn --argjson extra "$json" '
reduce (inputs / "|") as [$object, $type] ($extra; . + [{$object, $type}])
' data.txt
If you want the "static" data at the end, add it afterwards and start with an empty array:
jq -Rn --argjson extra "$json" '
reduce (inputs / "|") as [$object, $type] ([]; . + [{$object, $type}]) + $extra
' data.txt
You can try this :
jq -nR --argjson extra "$json" '
[inputs / "|" | {object:.[0], type:.[1]}] + $extra' data.txt
[inputs / "|" | {object: .[0], type: .[1]}]
Demo
https://jqplay.org/s/XkDdy9-lBq
Or
reduce (inputs / "|") as [$obj, $typ] ([]; .+[{$obj, $typ}])
Demo
https://jqplay.org/s/5N3M-pfJIR
disclaimer: indeed, there are already different answers (like JQ Join JSON files by key or denormalizing JSON with jq) for but none of them helped me yet or did have different circumstances I was unable to derive a solution from ;/
I have 2 files, both are lists of objects where one of them ha field references to object ids of the other one
given
[
{
"id": "5b9f50ccdcdf200283f29052",
"reference": {
"id": "5de82d5072f4a72ad5d5dcc1"
}
}
]
and
[
{
"id": "5de82d5072f4a72ad5d5dcc1",
"name": "FooBar"
}
]
my goal would be to get a denormalized object list:
expected
[
{
"id": "5b9f50ccdcdf200283f29052",
"reference": {
"id": "5de82d5072f4a72ad5d5dcc1",
"name": "FooBar"
}
}
]
while I'm able to do the main parts, I didn't challenged to bring both together yet:
with
example 1
jq -s '(.[1][] | select(.id == "5de82d5072f4a72ad5d5dcc1"))' objects.json referredObjects.json
I get
{
"id": "5de82d5072f4a72ad5d5dcc1",
"name": "FooBar"
}
and with
example 2
jq -s '.[0][] | .reference = {}' objects.json referredObjects.json
I can manipulate any .reference getting
{
"id": "5b9f50ccdcdf200283f29052",
"reference": {}
}
(even I loose the list structure)
But: I can't do s.th. like
execpted "join"
jq -s '.[0][] as $obj | $obj.reference = (.[1][] | select(.id == $obj.reference.id))' objects.json referredObjects.json
even approaches with foreach or reduce looks promising
jq -s '[foreach .[0][] as $obj ({}; .reference.id = ""; . + $obj )]' objects.json referredObjects.json
=>
[
{
"reference": {
"id": "5de82d5072f4a72ad5d5dcc1"
},
"id": "5b9f50ccdcdf200283f29052"
}
]
where I expected to get the same as in second example
I end up in headaches and looking forward to write a ineffective while routine in any language ... hopefully I would appreciate any help on this
~Marcel
Transform the second file into an object where ids and names are paired and use it as a reference while updating the first file.
$ jq '(map({(.id): .}) | add) as $idx
| input
| map_values(.reference = $idx[.reference.id])' file2 file1
[
{
"id": "5b9f50ccdcdf200283f29052",
"reference": {
"id": "5de82d5072f4a72ad5d5dcc1",
"name": "FooBar"
}
}
]
The following solution uses the same strategy as used in the solution by #OguzIsmail but uses the built-in function INDEX/2 to construct the dictionary from the second file.
The important point is that this strategy allows the arrays in both files to be of arbitrary size.
Invocation
jq --argfile file2 file2.json -f program.jq file1.json
program.jq
INDEX($file2[]; .id) as $dict
| map(.reference.id as $id | .reference = $dict[$id])
here are 3 JSON files
File1
{
"component1": [
]
}
File2
{
"component2": [
]
}
File3
{
"component3": [
]
}
Don't find the jq command line that would give this JSON file as jq output:
{
"components": {
"component1": [
],
"component2": [
],
"component3": [
]
}
}
Many thanks for your support
Best Regards.
Iterate over the input objects one a time from inputs and append it to the components using the reduce function
jq -n 'reduce inputs as $d (.; .components += $d )' file{1..3}.json
You can simply use add, e.g.
jq -s '{components: add}' file{1..3}.json
or:
jq -n '{components: [inputs]|add}' file{1..3}.json
Using jq how can I convert an array into object indexed by filename, or read multiple files into one object indexed by their filename?
e.g.
jq -s 'map(select(.roles[]? | contains ("mysql")))' -C dir/file1.json dir/file2.json
This gives me the data I want, but I need to know which file they came from.
So instead of
[
{ "roles": ["mysql"] },
{ "roles": ["mysql", "php"] }
]
for output, I want:
{
"file1": { "roles": ["mysql"] },
"file2": { "roles": ["mysql", "php"] }
}
I do want the ".json" file extension stripped too if possible, and just the basename (dir excluded).
Example
file1.json
{ "roles": ["mysql"] }
file2.json
{ "roles": ["mysql", "php"] }
file3.json
{ }
My real files obviously have other stuff in them too, but that should be enough for this example. file3 is simply to demonstrate "roles" is sometimes missing.
In other words: I'm trying to find files that contain "mysql" in their list of "roles". I need the filename and contents combined into one JSON object.
To simplify the problem further:
jq 'input_filename' f1 f2
Gives me all the filenames like I want, but I don't know how to combine them into one object or array.
Whereas,
jq -s 'map(input_filename)' f1 f2
Gives me the same filename repeated once for each file. e.g. [ "f1", "f1" ] instead of [ "f1", "f2" ]
If your jq has inputs (as does jq 1.5) then the task can be accomplished with just one invocation of jq.
Also, it might be more efficient to use any than iterating over all the elements of .roles.
The trick is to invoke jq with the -n option, e.g.
jq -n '
[inputs
| select(.roles and any(.roles[]; contains("mysql")))
| {(input_filename | gsub(".*/|\\.json$";"")): .}]
| add' file*.json
jq approach:
jq 'if (.roles[] | contains("mysql")) then {(input_filename | gsub(".*/|\\.json$";"")): .}
else empty end' ./file1.json ./file2.json | jq -s 'add'
The expected output:
{
"file1": {
"roles": [
"mysql"
]
},
"file2": {
"roles": [
"mysql",
"php"
]
}
}