How to pass dynamic values to JSON with duplicate fields - json

I have a usecase where i have to send REST request in bulk.
JSON File: emp.json
[
{
"field": {
"empID": "sapid",
"location": "India"
}
}
]
My shell script:
func emp_details
{
START=1
END=1000000
CURRENT=1
while [ $END -gt $CURRENT ];
do
CURRENT=$((CURRENT+1))
cat emp.json | jq --arg new "$CURRENT" '.[].field.empID |= $new' > temp.json
cat temp.json
curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d #temp.json
done
}
The above json and script is working. I am able send the request properly.
I am looking for an approach to prepare the json file with mutiple empID before triggering the CURL.
For Example:
[
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
}
]
But am not sure how to traverse through each individual empID field and replace its value with dynamic CURRENT values.
Any help is much appreciated

You're looking for the range built-in.
.[] | [.field.empID = range(1;1000000)]
demo at jqplay.org

You don't need bash processing at all for this. You can use the range() function in jq to create the number range from 1 to million and create multiple objects using the reduce() function
jq -n 'reduce range(1; 1000000) as $data (.; . + [{"field": { "empID": $data, "location": "India"}}])'
This creates a million objects inside the array with empID set, starting from 1. Modify the value inside range() to customize the numbers.

Related

Bash. How to access parameters while looping through json with jq?

i have a details.json file with a lot of entries and a shops.txt file like below. I like to have a little script which compares two values and just return the matching json entries.
[
{
"userName": "Anne",
"email": "anne#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "EFHJKI-KJEFT-DHMNEB",
"prod": "Car",
},
"store": {
"id": "05611a7f-a679-12ad-a3u2-0745e3650a03",
"storeName": "shop-a57ca0a3-120c-1a73-153b-fa4231cab768",
}
},
{
"userName": "Tom",
"email": "tom#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "DFSGSE-FGEAR-GWRTGW",
"prod": "Bike",
},
"store": null
},
]
This is the other file "shops.txt" (can be a lot more of shops inside)
shop-a57ca0a3-120c-1a73-153b-fa4231cab768
The script is looping through the shops, for every shop it loops through the json and should compare the currentShop with the store.shop from json and then echo the user and the shop.
But I can not access the specific parameters inside the json. How can I do this?
#!/bin/bash
shops="shops.txt"
while IFS= read -r line
do
currentShop="$line"
jq -c '.[.userName, .store.storeName]' details.json | while read i; do
if [[ $i.store.storeName == *$currentShop* ]]; then
echo $i.userName
echo $currentShop
fi
done
done < "$shops"
First of all, you might want to 'clean' your json, remove any trailing ,'s etc.
After looping through each line in the file we just need one select() to get the matching object.
The script could look something like:
#!/bin/bash
while read shop; do
echo "Check: $shop"
jq -r --arg storeName "$shop" '.[] | select(.store.storeName == "\($storeName)") | "\(.userName) - \(.store.storeName)"' details.json
done < "shops.txt"
Which will produce
Check: shop-a57ca0a3-120c-1a73-153b-fa4231cab768
Anne - shop-a57ca0a3-120c-1a73-153b-fa4231cab768
I guess this could be combined into a single jq call, but it seems like you want to loop over each entry found
You can test this jq selector on this online JqPlay Demo.
I was able to access the values with the following command:
echo $i | jq -r '.userName'

Merge and Sort JSON using JQ

I have a file containing the following structure and unknown number of results:
{
"results": [
[
{
"field": "AccountID",
"value": "5177497"
},
{
"field": "Requests",
"value": "50900"
}
],
[
{
"field": "AccountID",
"value": "pro"
},
{
"field": "Requests",
"value": "251"
}
]
],
"statistics": {
"Matched": 51498,
"Scanned": 8673577,
"ScannedByte": 2.72400814E10
},
"status": "HOLD"
}
{
"results": [
[
{
"field": "AccountID",
"value": "5577497"
},
{
"field": "Requests",
"value": "51900"
}
],
"statistics": {
"Matched": 51498,
"Scanned": 8673577,
"ScannedByte": 2.72400814E10
},
"status": "HOLD"
}
There are multiple such results which are indexed as an array with the results folder. They are not seperated by a comma.
I am trying to just print The "AccountID" sorted by "Requests" in ZSH using jq. I have tried flattening them and using:
jq -r '.results[][0] |.value ' filename
jq -r '.results[][1] |.value ' filename
To get the Account ID and Requests seperately and sorting them. I don't think bash has a dictionary that can be used. The problem lies in the file as the Field and value are not key value pair but are both pairs. Therefore extracting them using the above two lines into seperate arrays and sorting by the second array seems a bit too long. I was wondering if there is a way to combine both the operations.
The other way is to combine it all to a string and sort it in ascending order. Python would probably have the best solution but the code requires to be a zsh or bash script.
Solutions that use sed, jq or any other ZSH supported compilers are welcome. If there is a way to create a dictionary in bash, please do let me know.
The projectd output requirement is just the Account ID vs Request Number.
5577497 has 51900 requests
5177497 has 50900 requests
pro has 251 requests
If you don't mind learning a little jq, it will probably be best to write a small jq program to do what you want.
To get you started, consider the following jq program, which assumes your input is a stream of valid JSON objects with a "results" key similar to your sample:
[inputs | .results[] | map( { (.field) : .value} ) | add]
After making minor changes to your input so that it consists of valid JSON objects, an invocation of jq with the -n option produces an array of AccountID/Requests objects:
[
{
"AccountID": "5177497",
"Requests": "50900"
},
{
"AccountID": "pro",
"Requests": "251"
},
{
"AccountID": "5577497",
"Requests": "51900"
}
]
You could (for example) now use jq's group_by to group these objects by AccountID, and thereby produce the result you want.
jq -S '.results[] | map( { (.field) : .value} ) | add' query-results-aggregate \
| jq -s -c 'group_by(.number_of_requests) | .[]'
This does the trick. Thanks to peak for the guidance.

Perform string manipulation on a value and return the original JSON document with jq

In my JSON document I have a string that I need manipulated and then have the entire document returned with the 'fixed' values.
The input document is:
{
"records" : [
{
"time": "123456789000"
},
{
"time": "123456789000"
}
]
}
I want to find the "time" key and replace the string by dropping off the last 3 chars. The resulting document would be:
{
"records" : [
{
"time": "123456789"
},
{
"time": "123456789"
}
]
}
I've been trying to understand the jq query syntax but I'm not coming right. I'm still struggling to return the whole document when filtering on a specific value. All I have so far is:
.records[] | select(.time | contains("123456789000"))
Here is a solution using |= and string slicing
.records[].time |= .[:-3]
Sample Run (assuming data in data.json)
$ jq -M '.records[].time |= .[:-3]' data.json
{
"records": [
{
"time": "123456789"
},
{
"time": "123456789"
}
]
}
Try it online at jqplay.org
With jq sub() function:
jq '.records[].time |= sub("[0-9]{3}$";"")' file
The output:
{
"records": [
{
"time": "123456789"
},
{
"time": "123456789"
}
]
}
Or even simpler: via dividing the time value by 1000:
jq '.records[].time |= (tonumber / 1000 | tostring)' file
The following works with jq version 1.4 or later:
jq '.records[].time |= .[:-3]' file.json
(The expression .[:-3] is short for .[0:-3]; the negative integer here counts from the right.)
With jq 1.3, the following filter would work in your particular case:
.records[].time |= (tonumber | ./1000 | tostring)

json array into json stream with jq

This task is similar to this one but in my case I would like to go other way around.
So say we have input:
[
{
"name": "John",
"email": "john#company.com"
},
{
"name": "Brad",
"email": "brad#company.com"
}
]
and desired output is:
{
"name": "John",
"email": "john#company.com"
}
{
"name": "Brad",
"email": "brad#company.com"
}
I tried to write a bash function which will do it in loop:
#!/bin/bash
json=`cat $1`
length=`echo $json | jq '. | length'`
for (( i=0; i<$length ; i++ ))
do
echo $json | jq ".[$i]"
done
but it is obviously extremly slow...
Is there any way how to use jq better for this?
You can use this :
jq '.[]' file
If you use the .[index] syntax, but omit the index entirely, it will return all of the elements of an array.
Test:
$ jq '.[]' file
{
"email": "john#company.com",
"name": "John"
}
{
"email": "brad#company.com",
"name": "Brad"
}
you can apply ".[]" filter.
This tutorial is very informative
https://stedolan.github.io/jq/tutorial/

Update inner attribute of JSON with jq

Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=