This question already has answers here:
Argument list too long error for rm, cp, mv commands
(31 answers)
Closed 1 year ago.
I have thousands of JSON files, and I want to merge them into a single one. I'm using the command below to do this.
jq -s . -- *.json > result.json
But I am getting argument list too long error, probably because of the number of files I'm trying to merge. Is there any workaround for this issue?
Built-in commands are immune to that limitation, and printf is one of them. In conjunction with xargs, it would help a lot to achieve this.
printf '%s\0' *.json | xargs -0 cat -- | jq -s .
Related
This question already has an answer here:
"Argument list too long" while slurping JSON files [duplicate]
(1 answer)
Closed 1 year ago.
There are more than 6k JSON files, each containing exactly one JSON object. I want to prepare one list of objects from these JSONs.
When I am running below jq command I am getting an error.
Kedar.Javalkar#KD2806 MINGW64 /c/zz
$ jq -s '.' inventoryItem_*.json > inventory_items_result_$(date +"%Y%m%d_%H%M%S").json
bash: /usr/bin/jq: Argument list too long
I tried ulimit -s unlimited but the same error
I am using a windows 10 git bash
This is a job that xargs is created to fix -- splitting lists of items into individual command lines that are within the permitted limit.
Because running jq -s a single time is different from concatenating the results of multiple smaller runs, it's appropriate to use xargs to combine cat invocations using the manner described in the linked duplicate.
printf '%s\0' inventoryItem_*.json \
| xargs -0 cat \
| jq -s . \
>"inventory_items_result_$(date +"%Y%m%d_%H%M%S").json"
This question already has answers here:
How do I assign the output of a command into an array?
(3 answers)
Closed 2 years ago.
I'm trying to get all results of jq query in a array in bash, but it gives me all results in the first array element.
Could you help me? I would like to having every result in one array element
bash-4.2$ try=$(tm run jobs:status::get -s "tm=serverA&work=*&status=Failed" | $PATH_API/jq -r '.statuses[].name // empty')
bash-4.2$ echo $try
job_B job_C
bash-4.2$ echo "${#try[#]}"
1
bash-4.2$
If the status names are sufficiently uncomplicated, you could add an extra pair of parentheses:
try=($(tm run ....))
(Consider also invoking jq without the -r option.)
Otherwise, you could consider using readarray, or other techniques for initializing bash arrays.
This question already has answers here:
Capturing Groups From a Grep RegEx
(10 answers)
Closed 3 years ago.
I need to capture a specific string using grep, on Bash Command Line. I cannot use sudo install/apt-install to install any addition software/packages like jq. I am not using any Python Interpreter; simply Bash Command Line.
I tested my regex string with the desired input string on an online regex tester, and it works. Yet when I try it with grep, it provides me the whole text, and not the captured text.
Regex: '":"(.+)","'
Input:
HTTP_RESPONSE='{"access_token":"eyJhbGciOiJSUzI1NiIsImtpZCI6IkE4M0UwQTFEQTY1MzE0NkZENUQxOTFDMzRDNTQ0RDJDODYyMzMzMzkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiJxRDRLSGFaVEZHX1YwWkhEVEZSTkxJWWpNemsifQ.eyJuYmYiOjE1NTkzMjk5OTIsImV4cCI6MTU1OTMzMzU5MiwiaXNzIjoiaHR0cHM6Ly9jaW5jaHktbnByLmNsb3VkLnJlcy5ibmdmLmxvY2FsL2NpbmNoeXNzbyIsImF1ZCI6WyJodHRwczovL2NpbmNoeS1ucHIuY2xvdWQucmVzLmJuZ2YubG9jYWwvY2luY2h5c3NvL3Jlc291cmNlcyIsImpzX2FwaSJdLCJjbGllbnRfaWQiOiJhcGkiLCJzdWIiOiIxIiwiYXV0aF90aW1lIjoxNTU5MzI5OTkyLCJpZHAiOiJsb2NhbCIsInByb2ZpbGUiOiJBZG1pbmlzdHJhdG9yIiwiZW1haWwiOiJhZG1pbkBjaW5jaHkuY28iLCJyb2xlIjoiQ2luY2h5IFVzZXIgQWNjb3VudCIsImlkIjoiYWRtaW4iLCJzY29wZSI6WyJqc19hcGkiXSwiYW1yIjpbImN1c3RvbSJdfQ.gI_MYzkanGmDXPd8Wp1Q2XOOhAvnWxzOxSUngxYv8UFO1ozXm5ZxFDpUVXFodTQHKEk2B9h9VMXwyg80bxqqT7csHnjg43tylgEIKpupcubqXT1HzH_bMv8povGU6S75b8p8SWQKyYihm4nBdMDJFZMtKMg95ByBlaHpXV_6vuLUB0qFfcbWi5rHgHnJOT08ZJcHJUozS05FVZt_lU2zepNrwSqOvY3AnXpz9Z8KfrlARB46ukB3tBJzNvMDvxi44wfwpnky-4dxq0CcHoa766l6E66gjaLmR3ApHy4YKnP_DxDSR6mSHnVTEzPN7mt_1rklseK0TVd0kK3CpUg2aQ","expires_in":3600,"token_type":"Bearer"}'
echo $HTTP_RESPONSE | grep -E '":"(.+)","'
Obtained Result:
{"access_token":"eyJhbGciOiJSUzI1NiIsImtpZCI6IkE4M0UwQTFEQTY1MzE0NkZENUQxOTFDMzRDNTQ0RDJDODYyMzMzMzkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiJxRDRLSGFaVEZHX1YwWkhEVEZSTkxJWWpNemsifQ.eyJuYmYiOjE1NTkzMjk5OTIsImV4cCI6MTU1OTMzMzU5MiwiaXNzIjoiaHR0cHM6Ly9jaW5jaHktbnByLmNsb3VkLnJlcy5ibmdmLmxvY2FsL2NpbmNoeXNzbyIsImF1ZCI6WyJodHRwczovL2NpbmNoeS1ucHIuY2xvdWQucmVzLmJuZ2YubG9jYWwvY2luY2h5c3NvL3Jlc291cmNlcyIsImpzX2FwaSJdLCJjbGllbnRfaWQiOiJhcGkiLCJzdWIiOiIxIiwiYXV0aF90aW1lIjoxNTU5MzI5OTkyLCJpZHAiOiJsb2NhbCIsInByb2ZpbGUiOiJBZG1pbmlzdHJhdG9yIiwiZW1haWwiOiJhZG1pbkBjaW5jaHkuY28iLCJyb2xlIjoiQ2luY2h5IFVzZXIgQWNjb3VudCIsImlkIjoiYWRtaW4iLCJzY29wZSI6WyJqc19hcGkiXSwiYW1yIjpbImN1c3RvbSJdfQ.gI_MYzkanGmDXPd8Wp1Q2XOOhAvnWxzOxSUngxYv8UFO1ozXm5ZxFDpUVXFodTQHKEk2B9h9VMXwyg80bxqqT7csHnjg43tylgEIKpupcubqXT1HzH_bMv8povGU6S75b8p8SWQKyYihm4nBdMDJFZMtKMg95ByBlaHpXV_6vuLUB0qFfcbWi5rHgHnJOT08ZJcHJUozS05FVZt_lU2zepNrwSqOvY3AnXpz9Z8KfrlARB46ukB3tBJzNvMDvxi44wfwpnky-4dxq0CcHoa766l6E66gjaLmR3ApHy4YKnP_DxDSR6mSHnVTEzPN7mt_1rklseK0TVd0kK3CpUg2aQ","expires_in":3600,"token_type":"Bearer"}
Expected:
eyJhbGciOiJSUzI1NiIsImtpZCI6IkE4M0UwQTFEQTY1MzE0NkZENUQxOTFDMzRDNTQ0RDJDODYyMzMzMzkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiJxRDRLSGFaVEZHX1YwWkhEVEZSTkxJWWpNemsifQ.eyJuYmYiOjE1NTkzMjk5OTIsImV4cCI6MTU1OTMzMzU5MiwiaXNzIjoiaHR0cHM6Ly9jaW5jaHktbnByLmNsb3VkLnJlcy5ibmdmLmxvY2FsL2NpbmNoeXNzbyIsImF1ZCI6WyJodHRwczovL2NpbmNoeS1ucHIuY2xvdWQucmVzLmJuZ2YubG9jYWwvY2luY2h5c3NvL3Jlc291cmNlcyIsImpzX2FwaSJdLCJjbGllbnRfaWQiOiJhcGkiLCJzdWIiOiIxIiwiYXV0aF90aW1lIjoxNTU5MzI5OTkyLCJpZHAiOiJsb2NhbCIsInByb2ZpbGUiOiJBZG1pbmlzdHJhdG9yIiwiZW1haWwiOiJhZG1pbkBjaW5jaHkuY28iLCJyb2xlIjoiQ2luY2h5IFVzZXIgQWNjb3VudCIsImlkIjoiYWRtaW4iLCJzY29wZSI6WyJqc19hcGkiXSwiYW1yIjpbImN1c3RvbSJdfQ.gI_MYzkanGmDXPd8Wp1Q2XOOhAvnWxzOxSUngxYv8UFO1ozXm5ZxFDpUVXFodTQHKEk2B9h9VMXwyg80bxqqT7csHnjg43tylgEIKpupcubqXT1HzH_bMv8povGU6S75b8p8SWQKyYihm4nBdMDJFZMtKMg95ByBlaHpXV_6vuLUB0qFfcbWi5rHgHnJOT08ZJcHJUozS05FVZt_lU2zepNrwSqOvY3AnXpz9Z8KfrlARB46ukB3tBJzNvMDvxi44wfwpnky-4dxq0CcHoa766l6E66gjaLmR3ApHy4YKnP_DxDSR6mSHnVTEzPN7mt_1rklseK0TVd0kK3CpUg2aQ
With GNU grep and Perl-compatible regular expression (PCRE), if no more suitable tool (jq, e.g.) is available:
echo "$HTTP_RESPONSE" | grep -oP '(?<="access_token":").*?(?=")'
This question already has answers here:
Use jq to Convert json File to csv
(1 answer)
Converting json map to csv using jq
(3 answers)
Closed 4 years ago.
I want to efficiently rewrite a large json, which has always the same field names, into a csv, ignoring its keys.
To give a concrete example, here is a large JSON file (tempST.json):
https://gist.githubusercontent.com/pedro-roberto/b81672a89368bc8674dae21af3173e68/raw/e4afc62b9aa3092c8722cdbc4b4b4b6d5bbc1b4b/tempST.json
If I rewrite just fields time, ancestorcount and descendantcount from this JSON into a CSV I should get:
1535995526,1,1
1535974524,1,1
1535974528,1,2
...
1535997274,1,1
The following script tempSpeedTest.sh writes the value of the fields time, ancestorcount and descendantcount into each line of the csv:
rm tempOutput.csv
jq -c '.[]' < tempST.json | while read line; do
descendantcount=$(echo $line | jq '.descendantcount')
ancestorcount=$(echo $line | jq '.ancestorcount')
time=$(echo $line | jq '.time')
echo "${time},${ancestorcount},${descendantcount}" >> tempOutput.csv
done
However the script takes around 3 minutes to run, which is unsatisfying:
>time bash tempSpeedTest.sh
real 2m50.254s
user 2m43.128s
sys 0m34.811s
What is a faster way to achieve the same result?
jq -r '.[] | [.time, .descendantcount, .ancestorcount] | #csv' <tempST.json >tempOutput.csv
See this running at https://jqplay.org/s/QJz5FCmuc9
This question already has answers here:
Read JSON data in a shell script [duplicate]
(4 answers)
Closed 7 years ago.
Let's say that we have this kind of JSON file:
{
...
"quotes":{
"SOMETHING":10,
...
"SOMETHING_ELSE":120.4,
...
} }
How can I obtain those values and use them in order to add them together?
Am I able to do even this?
#!/bin/bash
#code ...
echo "$SOMETHING + $SOMETHING_ELSE" | bc
#code ...
#exit
I will obtain the JSON file with wget command. All I want is the content from this file.
Can you help me, please? I am a beginner in shell programming.
I usually use jq, a really fast json parser, to do this kind of things (because parsing a json file with tools like awk or sed is really error-prone).
Given an input file like this:
# file: input.json
{
"quotes":{
"SOMETHING":10,
"SOMETHING_ELSE":120.4
}
}
You can obtain the sum of the 2 fields with a simple filter:
jq '.quotes.SOMETHING + .quotes.SOMETHING_ELSE' input.json
# output -> 130.4
NOTE: jq is available in every major linux distribution. In a debian-derivative system you can install with a sudo apt-get install jq.
This will print out the sum of the selected lines' floats.
#!/bin/bash
awk '{ if ($1 ~ /"SOMETHING":/) {print}; if ($1 ~ /"SOMETHING_ELSE":/) {print} }' $1 | cut -d: -f2 | cut -d, -f1 | awk '{s+=$1};END{print s}'
This finds the lines you want, the plucks out the numbers, and adds them.
You should look up and learn jq as shown in Read the json data in shell script.
The tools in a "normal" shell installation like awk and sed all predate JSON by decades, and are a very very bad fit. jq is worth the time to learn.
Or use Python instead.