This task is similar to this one but in my case I would like to go other way around.
So say we have input:
[
{
"name": "John",
"email": "john#company.com"
},
{
"name": "Brad",
"email": "brad#company.com"
}
]
and desired output is:
{
"name": "John",
"email": "john#company.com"
}
{
"name": "Brad",
"email": "brad#company.com"
}
I tried to write a bash function which will do it in loop:
#!/bin/bash
json=`cat $1`
length=`echo $json | jq '. | length'`
for (( i=0; i<$length ; i++ ))
do
echo $json | jq ".[$i]"
done
but it is obviously extremly slow...
Is there any way how to use jq better for this?
You can use this :
jq '.[]' file
If you use the .[index] syntax, but omit the index entirely, it will return all of the elements of an array.
Test:
$ jq '.[]' file
{
"email": "john#company.com",
"name": "John"
}
{
"email": "brad#company.com",
"name": "Brad"
}
you can apply ".[]" filter.
This tutorial is very informative
https://stedolan.github.io/jq/tutorial/
Related
I'm trying to transform array to object by specific key. It works fine without using stream, but not possible when stream is applied.
Data:
[
{
"id": "1",
"userId": "fa51531d"
}
,
{
"id": "2",
"userId": "a167869a"
}
]
I tried running this command but it throws an error.
jq -n --stream 'fromstream(1|truncate_stream(inputs)) | INDEX(.id)' test.json > result.json
Data above should be transformed to:
{
"1": {
"userId": "fa51531d",
"id": "1"
},
"2": {
"userId": "a167869a",
"id": "2"
},
}
I want to achieve the same result as with jq 'INDEX(.id) but I need to use stream (because of big JSON file).
If you are trying to recreate the whole input object, the stream-based approach is rendered pointless. That said, using this approach, there's no need to truncate. So either replace 1 with 0:
jq -n --stream 'fromstream(0|truncate_stream(inputs)) | INDEX(.id)'
Or just omit it entirely (which reveals its futility):
jq -n --stream 'fromstream(inputs) | INDEX(.id)'
What would make more sense, is to output a stream of objects, each indexed as with INDEX. Maybe you were looking for this:
jq -n --stream 'fromstream(1|truncate_stream(inputs)) | {(.id):.}'
{
"1": {
"id": "1",
"userId": "fa51531d"
}
}
{
"2": {
"id": "2",
"userId": "a167869a"
}
}
To transform your JSON Array to JSON Object maybe you can use this
jq reduce .[] as $item ({}; .[$item.id] = $item)
but if you want to stream the JSON, i don't have the solutions
cmiiw
If your stream really looks like in your question, this should do:
jq 'INDEX(.id)' test.json
Output:
{
"1": {
"id": "1",
"userId": "fa51531d"
},
"2": {
"id": "2",
"userId": "a167869a"
}
}
Given the following JSON file (sample.json)
{
"api": "3.0.0",
"data": {
"description": "something",
"title": "hello",
"version": "1.0",
"app": {
"name": "abc",
"id": "xyz"
}
}
}
I wish to add the following JSON object at root level to the file above:
{
"heading": {
"user": ["$username"]
}
}
Where $username is a Bash variable.
Is there a better way to achieve this than the following?
blob=$(jq -n --arg foo API_NAME '{"heading": {"user": [env.username]}}')
jq --argjson obj "$(echo $blob)" '. + $obj' < sample.json
Just move what you create as blob directly into the other filter, ending up with just one jq call:
jq --arg username "$username" '. + {heading: {user: [$username]}}' sample.json
i have a details.json file with a lot of entries and a shops.txt file like below. I like to have a little script which compares two values and just return the matching json entries.
[
{
"userName": "Anne",
"email": "anne#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "EFHJKI-KJEFT-DHMNEB",
"prod": "Car",
},
"store": {
"id": "05611a7f-a679-12ad-a3u2-0745e3650a03",
"storeName": "shop-a57ca0a3-120c-1a73-153b-fa4231cab768",
}
},
{
"userName": "Tom",
"email": "tom#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "DFSGSE-FGEAR-GWRTGW",
"prod": "Bike",
},
"store": null
},
]
This is the other file "shops.txt" (can be a lot more of shops inside)
shop-a57ca0a3-120c-1a73-153b-fa4231cab768
The script is looping through the shops, for every shop it loops through the json and should compare the currentShop with the store.shop from json and then echo the user and the shop.
But I can not access the specific parameters inside the json. How can I do this?
#!/bin/bash
shops="shops.txt"
while IFS= read -r line
do
currentShop="$line"
jq -c '.[.userName, .store.storeName]' details.json | while read i; do
if [[ $i.store.storeName == *$currentShop* ]]; then
echo $i.userName
echo $currentShop
fi
done
done < "$shops"
First of all, you might want to 'clean' your json, remove any trailing ,'s etc.
After looping through each line in the file we just need one select() to get the matching object.
The script could look something like:
#!/bin/bash
while read shop; do
echo "Check: $shop"
jq -r --arg storeName "$shop" '.[] | select(.store.storeName == "\($storeName)") | "\(.userName) - \(.store.storeName)"' details.json
done < "shops.txt"
Which will produce
Check: shop-a57ca0a3-120c-1a73-153b-fa4231cab768
Anne - shop-a57ca0a3-120c-1a73-153b-fa4231cab768
I guess this could be combined into a single jq call, but it seems like you want to loop over each entry found
You can test this jq selector on this online JqPlay Demo.
I was able to access the values with the following command:
echo $i | jq -r '.userName'
I have a usecase where i have to send REST request in bulk.
JSON File: emp.json
[
{
"field": {
"empID": "sapid",
"location": "India"
}
}
]
My shell script:
func emp_details
{
START=1
END=1000000
CURRENT=1
while [ $END -gt $CURRENT ];
do
CURRENT=$((CURRENT+1))
cat emp.json | jq --arg new "$CURRENT" '.[].field.empID |= $new' > temp.json
cat temp.json
curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d #temp.json
done
}
The above json and script is working. I am able send the request properly.
I am looking for an approach to prepare the json file with mutiple empID before triggering the CURL.
For Example:
[
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
}
]
But am not sure how to traverse through each individual empID field and replace its value with dynamic CURRENT values.
Any help is much appreciated
You're looking for the range built-in.
.[] | [.field.empID = range(1;1000000)]
demo at jqplay.org
You don't need bash processing at all for this. You can use the range() function in jq to create the number range from 1 to million and create multiple objects using the reduce() function
jq -n 'reduce range(1; 1000000) as $data (.; . + [{"field": { "empID": $data, "location": "India"}}])'
This creates a million objects inside the array with empID set, starting from 1. Modify the value inside range() to customize the numbers.
Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=