I have files with json structure like this:
[
{
"uid": 11111,
"something": {
(...)
}
},
{
"uid": 22222,
"something": {
(...)
}
}
]
I'll read all files at one time (cat *) and i'd like to know which part is from which file, so i need to group it in some way.
So, my idea is to move content of each file to higher (parent) object with own members.
[
{
"var1": "val1"
"var2": "val2"
{
"uid": 11111,
"something": {
(...)
}
},
{
"uid": 22222,
"something": {
(...)
}
}
}
How to do that with jq?
#!/bin/bash
# For simplicity, assume each file in FILELIST contains a single JSON entity.
# Then instead of using cat FILELIST, use mycat FILELIST, e.g. mycat *.json
function mycat {
for file
do
jq --arg file "$file" '{"file": $file, "contents": .}' "$file"
done
}
If you have a sufficiently recent version of jq (e.g. jq 1.5) then one alternative would be:
jq '{file: input_filename, contents: .}' FILELIST
Related
I have two json files.
File 1:
{
"data": {
"items": []
}
}
File 2:
[
{
"name": "first name",
"path": [{
"matcher": "exact",
}]
},
{
...
}
]
I want to add all items from File 2 to the .data.items list in File 1.
How can I accomplish this?
Thanks in advance!
if you're willing to normalize the detail file:
items=$(jq -c ' . | { data: { items: . }} ' 2.json)
jq -s '{ data: { items: map(.data.items[])}}' 1.json <(echo $items)
Assuming both files contain valid JSON, you could do worse than:
jq --argfile extra 2.json '.data.items += $extra' 1.json
Using jq how can I convert an array into object indexed by filename, or read multiple files into one object indexed by their filename?
e.g.
jq -s 'map(select(.roles[]? | contains ("mysql")))' -C dir/file1.json dir/file2.json
This gives me the data I want, but I need to know which file they came from.
So instead of
[
{ "roles": ["mysql"] },
{ "roles": ["mysql", "php"] }
]
for output, I want:
{
"file1": { "roles": ["mysql"] },
"file2": { "roles": ["mysql", "php"] }
}
I do want the ".json" file extension stripped too if possible, and just the basename (dir excluded).
Example
file1.json
{ "roles": ["mysql"] }
file2.json
{ "roles": ["mysql", "php"] }
file3.json
{ }
My real files obviously have other stuff in them too, but that should be enough for this example. file3 is simply to demonstrate "roles" is sometimes missing.
In other words: I'm trying to find files that contain "mysql" in their list of "roles". I need the filename and contents combined into one JSON object.
To simplify the problem further:
jq 'input_filename' f1 f2
Gives me all the filenames like I want, but I don't know how to combine them into one object or array.
Whereas,
jq -s 'map(input_filename)' f1 f2
Gives me the same filename repeated once for each file. e.g. [ "f1", "f1" ] instead of [ "f1", "f2" ]
If your jq has inputs (as does jq 1.5) then the task can be accomplished with just one invocation of jq.
Also, it might be more efficient to use any than iterating over all the elements of .roles.
The trick is to invoke jq with the -n option, e.g.
jq -n '
[inputs
| select(.roles and any(.roles[]; contains("mysql")))
| {(input_filename | gsub(".*/|\\.json$";"")): .}]
| add' file*.json
jq approach:
jq 'if (.roles[] | contains("mysql")) then {(input_filename | gsub(".*/|\\.json$";"")): .}
else empty end' ./file1.json ./file2.json | jq -s 'add'
The expected output:
{
"file1": {
"roles": [
"mysql"
]
},
"file2": {
"roles": [
"mysql",
"php"
]
}
}
Using jq, how can I take a json object from a file (input_02.json), and append it to output.json, while retaining everything already in output.json (e.g. an object originating from file input_01.json).
The object to be appended in both cases is literally the entire contents of the file, with the file's "id" field as the object's key.
I'm taking a large list of input files (all with the same syntax) and essentially combining them like that.
The command i'm using to create the object to be appended is as follows:
jq '{(.id):(.)} ' input_01.json
which gives me:
{
"input1_id": {
}
}
input_1.json:
{
"id": "input1_id",
"val": "testVal1"
}
input2.json:
{
"id": "input2_id",
"val": "testVal2"
}
desired output:
{
"input1_id": {
"id": "input1_id",
"val": "testVal1"
},
"input2_id": {
"id": "input2_id",
"val": "testVal2"
}
}
You’re on the right track with {(.id):(.)}. The following should handle the case you mentioned, and might give you some ideas about similar cases:
program.jq: map({(.id):(.)}) | add
Invocation:
jq -s -f program.jq input_01.json input_02.json
You could use "jf" for this from https://pypi.python.org/pypi/jf
$ pip install jf
$ jf 'chain(), {y["id"]: y for y in x}' input1.json input2.json
{
"input2_id": {
"id": "input2_id",
"val": "testVal2"
},
"input1_id": {
"id": "input1_id",
"val": "testVal1"
}
}
I have two json file, one contains a map key name and a type, the other is a flat json file.
eg. first file contains something like this:
[ { "field": "col1", "type": "int" }, { "field" : "col2", "type" : "string" }]
second file is a large jsons object file separated by line break:
{ "col1":123, "col2": "foo"}
{ "col1":123, "col2": "foo"}
...
can I use JQ to generate an output json like this:
{ "col1":{ "int" : 123 }, "col2": { "string" : "foo"} }
{ "col1":{ "int" : 123 }, "col2": { "string" : "foo"} }
....
Sure. You might want to transform your first file in an easier to consume format first: map the .type to the .field properties to an object (to use as a dictionary)
reduce .[] as $i ({}; .[$i.field] = $i.type)
Then you could go through your second file to use these mappings to update the values. Use --argfile to read the contents of the first file into a variable.
$ jq --argfile file1 file1.json '
(reduce $file1[] as $i ({}; .[$i.field] = $i.type)) as $map
| with_entries(.value = { ($map[.key]): .value })
' file2.json
which yields:
{
"col1": {
"int": 123
},
"col2": {
"string": "foo"
}
}
{
"col1": {
"int": 123
},
"col2": {
"string": "foo"
}
}
Yes. You could use the --slurpfile option but your dictionary is already a single JSON entity (a JSON object in your case), so it would be simpler to read the dictionary using the --argfile option.
Assuming that:
your jq filter is in a file, say merge.jq;
your dictionary is in dictionary.json;
your input stream is in input.json
the jq invocation would look like this:
jq -f merge.jq --argfile dict dictionary.json input.json
With the above, you would of course refer to the dictionary as $dict in merge.jq
(Of course you could specify the filter on the jq command line, if that's what you prefer.)
Now, over to you!
Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=