I want print "[]" in every row with jq to make a big file json proof
I have:
{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}
{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}
{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}
{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}
What i want:
[[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]]
Now i can push the json line by line to a program.
If you input are just plain objects:
Use --slurp to create a single array, then use map([ . ]) to wrap each object in his own array
jq --slurp --compact-output 'map([ . ])'
[[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}],[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}], ...
JqPlay demo
Assuming your file contains a stream of stand-alone JSON objects, the filter is simply [.]. The filter will be applied to each input object.
jq -c '[.]' file.json
Output:
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
To wrap everything in one big array:
jq -s . file.json
# or:
jq -s <file.json
Output:
[
{
"test": "index",
"test2": "fdsdfs",
"test3": "dfs0D0WOQAA3"
},
{
"test": "index",
"test2": "fdsdfs",
"test3": "dfs0D0WOQAA3"
},
{
"test": "index",
"test2": "fdsdfs",
"test3": "dfs0D0WOQAA3"
},
{
"test": "index",
"test2": "fdsdfs",
"test3": "dfs0D0WOQAA3"
}
]
And to wrap each single item in an array and all arrays in a top-level array:
jq 'map([.])' file.json
Output:
[
[
{
"test": "index",
"test2": "fdsdfs",
"test3": "dfs0D0WOQAA3"
}
],
[
{
"test": "index",
"test2": "fdsdfs",
"test3": "dfs0D0WOQAA3"
}
],
[
{
"test": "index",
"test2": "fdsdfs",
"test3": "dfs0D0WOQAA3"
}
],
[
{
"test": "index",
"test2": "fdsdfs",
"test3": "dfs0D0WOQAA3"
}
]
]
I assume you just forgot to put the commas in your otherwise valid JSON output, in which case you can find reasonable answers elsewhere on this page.
If not, and your desired output is accurate as shown, one way to solve it with jq is to read in the lines as raw text, prepend and append to certain lines as you see fit, and output again as raw text:
jq -Rnr '[inputs] | (first, .[]) |= "[" + . | (last, .[]) += "]" | .[]'
[[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]
[{"test":"index","test2":"fdsdfs","test3":"dfs0D0WOQAA3"}]]
Demo
But, to be fair, you actually don't need a JSON processor if it is just plain text being manipulated. Using sed or awk would be more reasonable choices here.
Related
Given the following JSON file (sample.json)
{
"api": "3.0.0",
"data": {
"description": "something",
"title": "hello",
"version": "1.0",
"app": {
"name": "abc",
"id": "xyz"
}
}
}
I wish to add the following JSON object at root level to the file above:
{
"heading": {
"user": ["$username"]
}
}
Where $username is a Bash variable.
Is there a better way to achieve this than the following?
blob=$(jq -n --arg foo API_NAME '{"heading": {"user": [env.username]}}')
jq --argjson obj "$(echo $blob)" '. + $obj' < sample.json
Just move what you create as blob directly into the other filter, ending up with just one jq call:
jq --arg username "$username" '. + {heading: {user: [$username]}}' sample.json
Here is my json file :
[
{
"name": "1"
},
{
"name": "2"
},
{
"name": "3"
},
{
"name": "4"
}
]
i would like to get all object in a file one by line :
{"name":"1"}
{"name":"2"}
{"name":"3"}
{"name":"4"}
and my file is very big and i'am using the stream option.
Here is my attempt so far :
jq --stream -c '.[]' car.json > result.json
but it gives me :
[0,"name"]
"1"
[1,"name"]
"2"
This topic is covered in the jq FAQ. For the situation you describe you might be able to use the simpler of the two possibilities given there:
jq -cn --stream 'fromstream(1|truncate_stream(inputs))'
I would like to aggregate two or more files into a single json, and aggregate arrays under a same key.
file1.json
{
"shapes": [
{
"id": "1",
"name": "circle"
},
{
"id": "2",
"name": "square"
}
]
}
file2.json
{
"shapes": [
{
"id": "3",
"name": "triangle"
}
]
}
Expected result :
{
"shapes": [
{
"id": "1",
"name": "circle"
},
{
"id": "2",
"name": "square"
},
{
"id": "3",
"name": "triangle"
}
]
}
I can do this with the following jq command :
jq -s '{shapes: map(.shapes)|add }' file*.json
But this requires me to know the shapes attribute and hardcode it. Is there a simple way I can get the same result without ever using the key name explicitly?
Here is a solution that’s suitable when each top-level object has only one key, and that is both efficient and conceptually simple. It assumes jq is invoked with the -n option.
reduce inputs as $in (null;
($in|keys_unsorted[0]) as $k | { ($k): (.[$k] + $in[$k]) })
or slightly more compactly:
reduce inputs as $in (null; ($in|keys_unsorted[0]) as $k | .[$k] += $in[$k] )
Here is a solution that also solves a more general problem: first, it handles arbitrarily many input files; and second, it forms the "sum" by key, for every key, on the assumption that every top-level key is array-valued.
The generic function:
# the values at each key are assumed to be arrays
def aggregate(stream):
reduce stream as $o ({};
reduce ($o|keys_unsorted[]) as $k (.;
.[$k] += $o[$k] ));
To avoid "slurping", we will use inputs:
aggregate(inputs)
The invocation must therefore use the -n command-line option:
jq -n -f program.jq *.json
Try the following code. This can handle any number of files. All inputs are assumed to be json objects with all values inside as arrays. All such arrays are aggregated after grouping by keys. It outputs an object which has keys associated with corresponding aggregated arrays.
jq -s 'map(to_entries)|add|group_by(.key)|
map( { "key": (.[0].key), "value": (map(.value)|add)})|
from_entries' file1.json file2.json
For your sample input this gives:
{
"shapes": [
{
"id": "1",
"name": "circle"
},
{
"id": "2",
"name": "square"
},
{
"id": "3",
"name": "triangle"
}
]
}
I have multiple json objects which could be less when i merge the arrays if a object key matches the same value as the next json object. I'm trying to accomplish this with jq.
I think i have to use group_by(.name) first to group matching keys. I'm also using slurp to first wrap all objects into one big array.
I don't have anything working for now.
given:
{
"name": "a",
"list": [ "a1", "a2" ]
}
{
"name": "a",
"list": [ "a3", "a4" ]
}
{
"name": "b",
"list": [ "b1", "b2" ]
}
should result in:
{
"name": "a",
"list": [ "a1", "a2", "a3", "a4" ]
}
{
"name": "b",
"list": [ "b1", "b2" ]
}
You can use reduce like this:
$ jq -c -n 'reduce inputs as $p ({}; .[$p.name] |= { name : $p.name, list : (.list + $p.list) }) | .[]' file
{"name":"a","list":["a1","a2","a3","a4"]}
{"name":"b","list":["b1","b2"]}
Here's a simple and efficient solution that uses a common "aggregate by" technique:
reduce inputs as $kv ({}; .[$kv.name] += $kv.list)
| keys_unsorted[] as $k
| {name: $k, list: .[$k]}
Since inputs has been used here, the -n command-line option of jq should be specified.
Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=