This question already has an answer here:
"Argument list too long" while slurping JSON files [duplicate]
(1 answer)
Closed 1 year ago.
There are more than 6k JSON files, each containing exactly one JSON object. I want to prepare one list of objects from these JSONs.
When I am running below jq command I am getting an error.
Kedar.Javalkar#KD2806 MINGW64 /c/zz
$ jq -s '.' inventoryItem_*.json > inventory_items_result_$(date +"%Y%m%d_%H%M%S").json
bash: /usr/bin/jq: Argument list too long
I tried ulimit -s unlimited but the same error
I am using a windows 10 git bash
This is a job that xargs is created to fix -- splitting lists of items into individual command lines that are within the permitted limit.
Because running jq -s a single time is different from concatenating the results of multiple smaller runs, it's appropriate to use xargs to combine cat invocations using the manner described in the linked duplicate.
printf '%s\0' inventoryItem_*.json \
| xargs -0 cat \
| jq -s . \
>"inventory_items_result_$(date +"%Y%m%d_%H%M%S").json"
Related
This question already has answers here:
Argument list too long error for rm, cp, mv commands
(31 answers)
Closed 1 year ago.
I have thousands of JSON files, and I want to merge them into a single one. I'm using the command below to do this.
jq -s . -- *.json > result.json
But I am getting argument list too long error, probably because of the number of files I'm trying to merge. Is there any workaround for this issue?
Built-in commands are immune to that limitation, and printf is one of them. In conjunction with xargs, it would help a lot to achieve this.
printf '%s\0' *.json | xargs -0 cat -- | jq -s .
This question already has answers here:
How do I assign the output of a command into an array?
(3 answers)
Closed 2 years ago.
I'm trying to get all results of jq query in a array in bash, but it gives me all results in the first array element.
Could you help me? I would like to having every result in one array element
bash-4.2$ try=$(tm run jobs:status::get -s "tm=serverA&work=*&status=Failed" | $PATH_API/jq -r '.statuses[].name // empty')
bash-4.2$ echo $try
job_B job_C
bash-4.2$ echo "${#try[#]}"
1
bash-4.2$
If the status names are sufficiently uncomplicated, you could add an extra pair of parentheses:
try=($(tm run ....))
(Consider also invoking jq without the -r option.)
Otherwise, you could consider using readarray, or other techniques for initializing bash arrays.
This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Bash script store command output into variable
(2 answers)
Closed 3 years ago.
I am using curl to create a session to log into the switch.
Below the script that i use
curl --noproxy 10.23.193.1 -X POST http://10.23.193.1:80/rest/v3/login-sessions -d '{"userName":"admin", "password":"password"}'
After this is executed I get the following output
{"uri":"/login-sessions","cookie":"sessionId=DfZNCFbfoc3LDuMgjLXRiS8ZmEo4MWENCOJM0Iu14R1uMT9kKqbe1Rx6AedmoeT"}
My requirement here is to first only print this part of the string "sessionId=DfZNCFbfoc3LDuMgjLXRiS8ZmEo4MWENCOJM0Iu14R1uMT9kKqbe1Rx6AedmoeT"
Secondly I would want to know how to store the above string in a variable so that I could call the same variable for subsequent operations
I ran the following, but I am not getting any output.
curl --noproxy 10.23.193.1 -X POST http://10.23.193.1:80/rest/v3/login-sessions -d '{"userName":"admin", "password":"password"}' | grep -`E ""cookie":"
Avoid using simple tools like grep or sed to parse JSON as they won't handle things like quotes or multi-line data correctly. It's best to use a JSON-aware program such as jq.
With jq it's simple and robust:
curl ... | jq '.cookie'
To store the cookie in a variable use the -r flag to have JQ print out the raw, unquoted string.
cookie=$(curl ... | jq -r '.cookie')
Further reading:
jq Manual
Parsing JSON with Unix tools
How to parse JSON with shell scripting in Linux?
This question already has answers here:
Use jq to Convert json File to csv
(1 answer)
Converting json map to csv using jq
(3 answers)
Closed 4 years ago.
I want to efficiently rewrite a large json, which has always the same field names, into a csv, ignoring its keys.
To give a concrete example, here is a large JSON file (tempST.json):
https://gist.githubusercontent.com/pedro-roberto/b81672a89368bc8674dae21af3173e68/raw/e4afc62b9aa3092c8722cdbc4b4b4b6d5bbc1b4b/tempST.json
If I rewrite just fields time, ancestorcount and descendantcount from this JSON into a CSV I should get:
1535995526,1,1
1535974524,1,1
1535974528,1,2
...
1535997274,1,1
The following script tempSpeedTest.sh writes the value of the fields time, ancestorcount and descendantcount into each line of the csv:
rm tempOutput.csv
jq -c '.[]' < tempST.json | while read line; do
descendantcount=$(echo $line | jq '.descendantcount')
ancestorcount=$(echo $line | jq '.ancestorcount')
time=$(echo $line | jq '.time')
echo "${time},${ancestorcount},${descendantcount}" >> tempOutput.csv
done
However the script takes around 3 minutes to run, which is unsatisfying:
>time bash tempSpeedTest.sh
real 2m50.254s
user 2m43.128s
sys 0m34.811s
What is a faster way to achieve the same result?
jq -r '.[] | [.time, .descendantcount, .ancestorcount] | #csv' <tempST.json >tempOutput.csv
See this running at https://jqplay.org/s/QJz5FCmuc9
This question already has answers here:
Read JSON data in a shell script [duplicate]
(4 answers)
Closed 7 years ago.
Let's say that we have this kind of JSON file:
{
...
"quotes":{
"SOMETHING":10,
...
"SOMETHING_ELSE":120.4,
...
} }
How can I obtain those values and use them in order to add them together?
Am I able to do even this?
#!/bin/bash
#code ...
echo "$SOMETHING + $SOMETHING_ELSE" | bc
#code ...
#exit
I will obtain the JSON file with wget command. All I want is the content from this file.
Can you help me, please? I am a beginner in shell programming.
I usually use jq, a really fast json parser, to do this kind of things (because parsing a json file with tools like awk or sed is really error-prone).
Given an input file like this:
# file: input.json
{
"quotes":{
"SOMETHING":10,
"SOMETHING_ELSE":120.4
}
}
You can obtain the sum of the 2 fields with a simple filter:
jq '.quotes.SOMETHING + .quotes.SOMETHING_ELSE' input.json
# output -> 130.4
NOTE: jq is available in every major linux distribution. In a debian-derivative system you can install with a sudo apt-get install jq.
This will print out the sum of the selected lines' floats.
#!/bin/bash
awk '{ if ($1 ~ /"SOMETHING":/) {print}; if ($1 ~ /"SOMETHING_ELSE":/) {print} }' $1 | cut -d: -f2 | cut -d, -f1 | awk '{s+=$1};END{print s}'
This finds the lines you want, the plucks out the numbers, and adds them.
You should look up and learn jq as shown in Read the json data in shell script.
The tools in a "normal" shell installation like awk and sed all predate JSON by decades, and are a very very bad fit. jq is worth the time to learn.
Or use Python instead.