This question already has answers here:
Convert a nested JSON of objects into array into a bash array using jq
(2 answers)
Is there a way to read all key-value pairs in the JSON file and then initialize all variables accordingly in shell?
(1 answer)
Accessing a JSON object in Bash - associative array / list / another model
(7 answers)
Closed 5 years ago.
I have json string like this:
jstring='[{"userQuery":"select name from abc;","user":"abc"},{"userQuery":"select name from xyz;","user":"xyz"},{"userQuery":"select name from ppp;","user":"ppp"}]'
I wrote a simple for loop using jq to extract values but not getting the desired result.
for i in `echo $jstring | jq '.[] | [.user, .userQuery]'`; do echo ${i}; done
With the help of this line : echo $jstring | jq '.[] | [.user, .userQuery]'. I am able to extract below info:
[ "abc", "select name from abc;"][ "xyz", "select name from xyz;"][ "ppp", "select name from ppp;"]
Now, I want two variable "user" & "query" for each array and store that info.
Eg: For [ "abc", "select name from abc;"] -- user: abc, query: "select name from abc" and store them.
I am not sure how to iterate over json using jq and get individual values and store them.
I assume you mean you want to store them as shell variables. With bash you can write:
while read -r user query; do
echo "user->$user"
echo "query->$query"
done < <( jq -r '.[] | "\(.user) \(.userQuery)"' <<< "$jstring" )
user->abc
query->select name from abc;
user->xyz
query->select name from xyz;
user->ppp
query->select name from ppp;
The secret sauce is in the formulation of the string inside jq that refers to the object properties with \(.user)
I'm assuming that the user name does not contain spaces. Otherwise we have to use a different separator in that string, and use IFS with the shell read command.
jq + bash solution:
#!/bin/bash
jstring='[{"userQuery":"select name from abc;","user":"abc"},{"userQuery":"select name from xyz;","user":"xyz"},{"userQuery":"select name from ppp;","user":"ppp"}]'
while IFS=$'\t' read -r user query; do
msg="User: ${user} is running : ${query}"
mail -s "User query" "youremail#gmail.com" <<< "$msg"
done < <(jq '.[] | [.user,.userQuery] | #tsv' <<< "$jstring")
Related
This question already has answers here:
Passing bash variable to jq
(10 answers)
jq select value from array
(3 answers)
How to select items in JQ based on value in array
(4 answers)
Closed 6 months ago.
I am writing a shell script that will read from a text file line by line some item names stored in the $entry variable within a while loop. With a curl command, I retrieve a json which is given as sample json response. I need to be able to compare the input text value which is saved in $entry to all the values within the item_name array and return true/false depending on if the $entry value is present using jq library. I am able to retrieve the item names values using the jq command written below, but I am not familiar how to compare with the input string. when I use "select" command to compare, I get syntax error. I am quite new to jq and not proficient with using filters/select statement for comparison, any pointers on how to resolve it?
bash variable
$entry = "some value from a text file"
sample json response
"response": {
"numFound": 2,
"docs": [{
"country": "USA",
"item_name": [
"parachutes",
"air balloon"
]
},
{
"country": "Brazil",
"item_name": [
"gliders",
"aircraft carriers"
]
}
]
}
jq command
echo "$RESPONSE" | jq '.' | jq '.["response"]["docs"]' | jq '.[].item_name' | jq - r '.[]'
Update:
The jq command that works as expected is
echo $RESPONSE | jq -e '.' | jq -r '.["response"]["docs"] | .[].item_name ' | jq -s 'add' | jq 'any(. == "'"$entry"'") | if . == true then "Exact match" else "No Exact match" end'
I have a json file test.json with the content:
[
{
"name": "Akshay",
"id": "234"
},
{
"name": "Amit",
"id": "28"
}
]
I have a shell script with content:
#!/bin/bash
function display
{
echo "name is $1 and id is $2"
}
cat test.json | jq '.[].name,.[].id' | while read line; do display $line; done
I want name and id of a single item to be passed together as arguments to the function display but the output is something like this :
name is "Akshay" and id is
name is "Amit" and id is
name is "234" and id is
name is "28" and id is
What should be the correct way to implement the code?
PS: I specifically want to use jq so please base the answer in terms of jq
Two major issues, and some additional items that may not matter for your current example use case but can be important when you're dealing with real-world data from untrusted sources:
Your current code iterates over all names before writing any ids.
Your current code uses newline separators, but doesn't make any effort to read multiple lines into each while loop iteration.
Your code uses newline separators, but newlines can be present inside strings; consequently, this is constraining the input domain.
When you pipe into a while loop, that loop is run in a subshell; when the pipeline exits, the subshell does too, so any variables set by the loop are lost.
Starting up a copy of /bin/cat and making jq read a pipe from its output is silly and inefficient compared to letting jq read from test.json directly.
We can fix all of those:
To write names and ids in pairs, you'd want something more like jq '.[] | (.name, .id)'
To read both a name and an id for each element of the loop, you'd want while IFS= read -r name && IFS= read -r id; do ... to iterate over those pairs.
To switch from newlines to NULs (the NUL being the only character that can't exist in a C string, or thus a bash string), you'd want to use the -j argument to jq, and then add explicit "\u0000" elements to the content being written. To read this NUL-delimited content on the bash side, you'd need to add the -d '' argument to each read.
To move the while read loop out of the subshell, we can use process substitution, as described in BashFAQ #24.
To let jq read directly from test.json, use either <test.json to have the shell connect the file directly to jq's stdin, or pass the filename on jq's command line.
Doing everything described above in a manner robust against input data containing JSON-encoded NULs would look like the following:
#!/bin/bash
display() {
echo "name is $1 and id is $2"
}
cat >test.json <<'EOF'
[
{ "name": "Akshay", "id": "234" },
{ "name": "Amit", "id": "28" }
]
EOF
while IFS= read -r -d '' name && IFS= read -r -d '' id; do
display "$name" "$id"
done < <(jq -j '
def stripnuls: sub("\u0000"; "<NUL>");
.[] | ((.name | stripnuls), "\u0000", (.id | stripnuls), "\u0000")
' <test.json)
You can see the above running at https://replit.com/#CharlesDuffy2/BelovedForestgreenUnits#main.sh
You can use string interpolation.
jq '.[] | "The name is \(.name) and id \(.id)"'
Result:
"The name is Akshay and id 234"
"The name is Amit and id 28"
"The name is hi and id 28"
If you want to get rid of the double-quotes from each object, then:
jq --raw-output '.[] | "The name is \(.name) and is \(.id)"'
https://jqplay.org/s/-lkpHROTBk0
This question already has answers here:
Exporting JSON to environment variables
(7 answers)
Closed 1 year ago.
I am trying to read the json file and store the keys as bash variable and the values as variable value. For instance here is my test.json:
{
"Day":'0',
"num":'85',
"Qtr":"[20183, 20184, 20191, 20192]",
"pSize":"75"
}
I need to store the variables like this in a bash file:
$Day=0
$num=85
$Qtr=[20183, 20184, 20191, 20192]
$psize=75
I found a way to extract the value using jq, but I am unable to store the key as variable in bash
-bash-4.2$ Qtr=`jq .Qtr test.json`
-bash-4.2$ echo $Qtr
"[20183, 20184, 20191, 20192]"
Could someone please provide a snippet as to how to loop through a json file and store the key as variables and values as values?
Thank you in advance
Would you please try the following:
#!/bin/bash
while IFS== read key value; do
printf -v "$key" "$value"
done < <(jq -r 'to_entries|map("\(.key)=\(.value|tostring)")|.[]' test.json)
# test of the variables
echo "Day: $day"
echo "Qtr: $Qtr"
echo "num: $num"
echo "pSize: $pSize"
The jq command jq -r 'to_entries|map("\(.key)=\(.value|tostring)")|.[]' test.json converts the json object into key=value pairs.
The output of jq is fed to the while loop via the process substitution
<(command).
The -v varname option to the printf command assigns the variable
indirecly expressed with varname to the output.
I'm trying to pull a list of product categories from an API using jq and some nested for-loops. I have to pull the category ID first, then I'm able to pull product details. Some of the category IDs have hypens and jq seems to be treating them like math instead of a string, and I've tried every manner of quoting but I'm still running into this error. In Powershell, I'm able to pull the list just fine, but I really need this to work in bash.
Here's the expected list:
aprons
backpacks
beanies
bracelet
coaster
cutting-board
dress-shirts
duffel-bags
earring
full-brim-hats
generic-dropoff
hats
etc...
And trying to recreate the same script in Bash, here's the output:
aprons
backpacks
beanies
bracelet
coaster
parse error: Invalid numeric literal at line 1, column 6
parse error: Invalid numeric literal at line 1, column 7
parse error: Invalid numeric literal at line 1, column 5
earring
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 1, column 5
parse error: Invalid numeric literal at line 1, column 8
hats
etc...
You can see that it's running into this error with all values that contain hyphens. Here's my current script:
#!/bin/bash
CATEGORIES=$(curl -s https://api.scalablepress.com/v2/categories)
IFS=$' \t\n'
for CATEGORY in $(echo $CATEGORIES | jq -rc '.[]')
do
CATEGORY_IDS=$(echo $CATEGORY | jq -rc '."categoryId"')
for CATEGORY_ID in $(echo $CATEGORY_IDS)
do
echo $CATEGORY_ID
PRODUCT_IDS=$(curl -s https://api.scalablepress.com/v2/categories/$CATEGORY_ID | jq -rc '.products[].id')
#for PRODUCT_ID in $(echo $PRODUCT_IDS)
#do
#echo $PRODUCT_ID
#done
done
done
This is a publicly available API so you should be able to copy this script and produce the same results. All of the guides I've seen have said to put double quotes around the field you're trying to parse if it contains hyphens, but I'm having no luck trying that.
you can loop over categories ids right away, without doing all the "echos" that break the json. the two loops can be rewritten as:
#!/bin/bash
CATURL="https://api.scalablepress.com/v2/categories"
curl -s "$CATURL" | jq -rc '.[] | .categoryId' | while read catid; do
echo "$catid"
curl -s "$CATURL/$catid" | jq -rc '.products[].id'
done
this will print category id followed by all products ids which from you code seems like your end result:
$ ./pullcat.sh
aprons
port-authority-port-authority-â-medium-length-apron-with-pouch-pockets
port-authority-port-authority-â-full-length-apron-with-pockets
port-authority-easy-care-reversible-waist-apron-with-stain-release
port-authority-easy-care-waist-apron-with-stain-release
backpacks
port-authority-â-wheeled-backpack
nike-performance-backpack
port-authority-â-value-backpack
port-authority-â-basic-backpack
port-authority-â-cyber-backpack
port-authority-â-commuter-backpack
port-authority-â-contrast-honeycomb-backpack
port-authority-â-camo-xtreme-backpack
port-authority-â-xtreme-backpack
port-authority-â-xcapeâ-computer-backpack
port-authority-â-nailhead-backpack
nike-elite-backpack
port-authority-â-urban-backpack
eddie-bauer-eddie-bauer-â-ripstop-backpack
the-north-face-aurora-ii-backpack
the-north-face-fall-line-backpack
the-north-face-groundwork-backpack
the-north-face-connector-backpack
beanies
rabbit-skins-infant-baby-rib-cap
yupoong-adult-cuffed-knit-cap
ultra-club-adult-knit-beanie-with-cuff
ultra-club-adult-knit-beanie
ultra-club-adult-two-tone-knit-beanie
ultra-club-adult-knit-beanie-with-lid
ultra-club-adult-waffle-beanie
ultra-club-adult-knit-pom-pom-beanie-with-cuff
bayside-beanie
...
if you want just the categories ids, you can of course "drop" while loop:
#!/bin/bash
CATURL="https://api.scalablepress.com/v2/categories"
curl -s "$CATURL" | jq -rc '.[] | .categoryId'
$ ./pullcat.sh
aprons
backpacks
beanies
bracelet
coaster
cutting-board
dress-shirts
duffel-bags
earring
full-brim-hats
generic-dropoff
hats
hoodies
infant-shirts
ladies-dress-shirts
ladies-dresses
ladies-long-sleeve
ladies-pants
ladies-performance-shirts
ladies-polos
ladies-short-sleeve
ladies-tank-tops
large-bags
...
You can select the key categoryId for each object in the array by applying the selector: curl -s https://api.scalablepress.com/v2/categories | jq 'map(.categoryId)'
This will give you a JSON array with only the values you're interested in. Then you can use the antislurp filter .[] to turn the array into individual results. jq can then output raw strings with the -r switch.
Combining everything, you can achieve what you're looking for with a one-liner:
curl -s https://api.scalablepress.com/v2/categories | jq -r 'map(.categoryId) | .[]'
Even better, you can antislurp first, and then select the key you're looking for: curl -s https://api.scalablepress.com/v2/categories | jq -r '.[] | .categoryId'
I have a json string in the following format:
json='{"x": [{"y":"yyy1", "z":"zzz1 zzz1"}, {"y":"yyy2", "z":"zzz2 zzz2"}]}'
I'm trying to get an array var of all "y" and separate array of "z" with jq like
y_arr=$(echo "${json}" | jq '.x | [] | .y') # => outputs 2 array elements
y_arr=$(echo "${json}" | jq '.x | [] | .z') # => outputs 4 array elements due to space
Do you have any idea how to fix this?
Also, do you know any better way of parsing the json as single key/value array instead of two separate arrays?
Assuming your bash has readarray (aka mapfile) and that having the bash array values be valid JSON is acceptable, you could write:
readarray z_array < <(echo "${json}" | jq -c '.x[] | .z')
With your $json, this would yield an array of JSON strings (with quotation marks).
shell loop
If your shell does not support readarray,
then you could use the same invocation of jq but read the values in a bash loop, e.g.:
declare -a array
while read v
do
array+=("$v")
done < <(echo "${json}" | jq -c '.x[] | .z')
Strings
If all the .z values are JSON strings, then provided these strings do not have embedded newline characters, and that you want the bash array values to be "raw" strings (as opposed to JSON strings), you could add the -r option to the invocation of jq.
Associative Arrays
If your bash supports associative arrays, consider:
declare -A a
i=0
while read v
do
i=$((i+1))
a["$v"]=$i
done < <(echo "${json}" | jq -c '.x[].z')
declare -p a
for i in "${!a[#]}" ; do
echo $i
done
This produces:
declare -A a=(["\"zzz2 zzz2\""]="2" ["\"zzz1 zzz1\""]="1" )
"zzz2 zzz2"
"zzz1 zzz1"