This question already has answers here:
A better way to extract JSON value in bash script
(2 answers)
Closed 6 years ago.
I'm currently trying to grab and assign the N_596164000673190002 to a variable from a curl command.
This is the command:
curl -L -H 'X-Cisco-Meraki-API-Key: mykeygoeshere' -X POST -H'Content-Type: application/json' --data-binary '{"name":"'"$NETWORK_NAME"'", "type":"appliance", "timeZone":"'"$TIME_ZONE"'"}' 'https://dashboard.meraki.com/api/v0/organizations/foobar/networks'
This is the response:
{"id":"N_596164000673190002","organizationId":"foo","type":"appliance","name":"bar","timeZone":"America/Chicago","tags":""}
How do I successfully read and grab the variable after id (without the double quotes), while also simutaneously assigning it to a variable, $NETWORK_ID? I imagine this can all be done in one line.
If this is successful, echo $NETWORK_ID should return N_596164000673190002
To parse json in bash, people usually use jq as it is installed by default on most Unix distributions.
Try the following :
NETWORK_ID=$(my_curl_command | jq -r '.id')
Here, '.id' is a filter indicating we want to retrieve the value for the key id, and the -r flag is used to remove double quotes from the output.
Pipe the JSON output to python json to grab the id value you need, and use bash command substitution to assign the result to your NETWORK_ID environment variable.
NETWORK_ID=$(curl -L -H 'X-Cisco-Meraki-API-Key: mykeygoeshere' -X POST \
-H'Content-Type: application/json' \
--data-binary '{"name":"'"$NETWORK_NAME"'", "type":"appliance", \
"timeZone":"'"$TIME_ZONE"'"}' \
'https://dashboard.meraki.com/api/v0/organizations/foobar/networks' \
| python -c "import sys, json; print json.load(sys.stdin)['id']")
Related
I'm trying to write a sample script where I'm generating names like 'student-101...student-160'. I need to post JSON data and when I do, I get a JSON parse error.
Here's my script:
name="student-10"
for i in {1..1}
do
r_name=$name$i
echo $r_name
curl -i -H 'Authorization: token <token>' -d '{"name": $r_name, "private": true}' "<URL>" >> create_repos_1.txt
echo created $r_name
done
I always get a "Problems parsing JSON" error. I've tried various combination of quotes, etc but nothing seems to work!
What am I doing wrong?
First, your name property is a string, so you need to add double quotes to it in your json.
Second, using single quotes, bash won't do variable expansion: it won't replace $r_name with the variable content (see Expansion of variable inside single quotes in a command in bash shell script for more information).
In summary, use:
-d '{"name": "'"$r_name"'", "private": true}'
Another option is to use printf to create the data string:
printf -v data '{"name": "%s", "private": true}' "$r_name"
curl -i -H 'Authorization: token <token>' -d "$data" "$url" >> create_repos_1.txt
Don't; use jq (or something similar) to build correctly quoted JSON using variable inputs.
name="student-10"
for i in {1..1}
do
r_name=$name$i
jq -n --arg r_name "$r_name" '{name: $r_name, private: true}' |
curl -i -H 'Authorization: token <token>' -d #- "<URL>" >> create_repos_1.txt
echo created $r_name
done
The #- argument tells curl to read data from standard input (via the pipe from jq) to use for -d.
Something like "{\"name\": \"$r_name\", \"private\": true}" may work, but it is ugly and will also fail if r_name contains any character which needs to be quoted in the resulting JSON, such as double quotes or ASCII control characters.
Using bash v4.3.48(1) on Ubuntu 16.04.2 LTS and bash v3.2.57 on macOS Sierra 10.12.5 I have this code which POSTs JSON data stored in a variable:
#1
JSON_DATA="Test"; \
curl -X POST "https://www.techotaku.com/json.php" \
-H "Content-Type: application/json" \
-d '{"data":"'$JSON_DATA'"}'
This returns:
Array
(
[data] => Test
)
However, if $JSON_DATA contains a space...
#2
JSON_DATA="A Test"; \
curl -X POST "https://www.techotaku.com/json.php" \
-H "Content-Type: application/json" \
-d '{"data":"'$JSON_DATA'"}'
...it throws an error:
curl: (3) [globbing] unmatched close brace/bracket in column 6
I thought this might be due to incorrect use of quotes, but if I echo the JSON data...
echo '{"data":"'$JSON_DATA'"}'
...it appears well-formed:
{"data":"A Test"}
Also, if I include the JSON data as a string and not a variable, it works.
#3
curl -X POST "https://www.techotaku.com/json.php" \
-H "Content-Type: application/json" \
-d '{"data":"A Test"}'
The only way I can get it to work if the variable contains a space is to enclose the JSON data in double-quotes and escape the literal double-quotes in the string:
#4
JSON_DATA="A Test"; \
curl -X POST "https://www.techotaku.com/json.php" \
-H "Content-Type: application/json" \
-d "{\"data\":\"$JSON_DATA\"}"
I would have thought that all 4 examples would have worked. So, can anyone throw any light on why only examples 1,3 and 4 work and example 2 doesn't?
Many thanks, Steve.
You should quote your variables to prevent word splitting by shell. Your second example:
'{"data":"'$JSON_DATA'"}'
doesn't work because it's being expanded by bash to two words:
{"data":"A Test"}
so -d option receives 2 arguments ({"data":"A and Test"}). If JSON_DATA contained globbing metacharacters, those would cause filename expansion.
To fix it, quote it:
'{"data":"'"$JSON_DATA"'"}'
Note the use of alternating single and double quotes. This will be expanded to a single word:
{"data":"A Test"}
Here's a more detailed answer I wrote some time ago which also mentions another convenient option, the use of a here-document.
I am using Gitbash, GNU bash, version 4.3.46(2)-release (x86_64-pc-msys).
I am running a bash script that looks like this
CODE CHUNK 1
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
'http://localhost:5000/api/123'
This works fine. Basically it posts a giant payload to an endpoint and all is well. The problem is when I attempt to substitute the url for a value from a variable, I get a curl error,
curl: (1) Protocol "'http" not supported or disabled in libcurl
CODE CHUNK 2
stuff=\'http://localhost:5000/api/123\'
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
$stuff
If I echo $stuff immediately after the stuff=\'http://localhost:5000/api/123\', I get 'http://localhost:5000/api/123'. This is the same value as I had hard-coded in code chunk 1, single ticks and all. There is something hiding behind the scenes in how that url is being evaluated after the variable has been expanded. I need to get the same behavior as a hard coded url.
Look closely at this error message:
curl: (1) Protocol "'http" not supported or disabled in libcurl
Notice the single-quote in front of http.
The curl command surely knows the http protocol, but not the 'http protocol!
The way you wrote it, the single-quotes are part of the value of stuff:
stuff=\'http://localhost:5000/api/123\'
Remove those, write like this:
stuff='http://localhost:5000/api/123'
If you have variables inside your real string,
and you want them expanded, then use double-quotes instead of single-quotes:
stuff="http://localhost:5000/api/123"
Equally important,
when you use $stuff as a parameter of curl,
you must double-quote it.
If you just write curl $stuff,
then the shell may interpret some characters in $stuff before passing to curl.
To protect from that,
you must write curl "$stuff".
The complete command:
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
"$stuff"
Finally, make sure that after each \ at the end of lines,
there's nothing after the \ on each line,
the \ must be at the very end.
Why do you define stuff with the ' ' ?
Try it like that:
stuff="http://localhost:5000/api/123"
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
"$stuff"
Also, don't put variables in single quotes, because bash is not available to understand them.
stuff="http://localhost:5000/api/123"
echo "$stuff"
>> http://localhost:5000/api/123
echo '$stuff'
>> $stuff
Looking for some advice on the direction I should take my project as I'm very new to scripting. My objectives are as follows:
Using cURL to get JSON data, parse a particular object value. I was intending to use jq to parse the JSON and then store the results as a variable.
Here is what I have to started with:
SAMPLE JSON -
{
"href": "http://localhost:8080//profiles",
"Profiles": [
{
"href": "http://localhost:8080/profiles/myprofile",
"id": "myprofile",
"isRecording": false
},
{
"href": "http://localhost:8080/profiles/yourprofile",
"id": "yourprofile",
"isRecording": false
}
]
}
BASH SCRIPT -
#!/bin/bash
read -p "Please enter downtime name (and press Enter): " downtimename
read -p "Please enter timestart:" timeStart
read -p "Please enter time duration (in minutes): " durationMinutes
#!/bin/bash
PROFILE="$(curl --user admin:admin -k -X GET https://xxx.xx.xx.xxx:8080/profiles.json | jq '.Profiles[].id')"
echo "$PROFILE"
RETURNS -
"myprofile"
"yourprofile"
Next I need to construct 1 or more cURL POST of xml data (example below) Each post will be a single curl post of the xml, for each line in the above echo "$PROFILE". I’m thinking this will be a for loop? What I'm struggling with is how to read each value/line from "$PROFILE" and utilize a for loop to post xml while replacing ${profile] in the below curl URL, for each result above.
curl -X POST https://xxx.xx.xx.xxx:8080/testprofiles/${profile}/time/${downtimename} --data-urlencode xml="<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit="MINUTES">${durationMinutes}</duration>
</gametime>
</time>" -H 'Accept: application/xml' \ -H 'Content-Type: application/xml' -u admin:admin
Thank you in advance for the help
Strings and quotes
The XML string you pass to --data-urlencode is interpreted without double quotes:
<?xml version=1.0 encoding=UTF-8 standalone=yes?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit=MINUTES>${durationMinutes}</duration>
</gametime>
</time>
Double quotes are the shell syntax elements that influence parsing. They are removed before the command is called. Prepend a backslash to a double quote, if you want its literal value in the string, e.g. "version=\"1.0\"".
However, here documents are more convenient in most cases:
xml=$(cat <<XML
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit="MINUTES">${durationMinutes}</duration>
</gametime>
</time>
XML
)
jq output
Use --raw-output option to skip the JSON formatting. (In particular, string values are wrapped with double quotes by default.).
Turn off color output with --monochrome-output option. With this option, jq will not print the ASCII escape codes.
Although not necessary, I recommend turning off the output buffering with --unbuffered option. With this option, jq will flush the output buffer after each JSON object is printed.
Reading jq output line-by-line
You don't need to store the output of jq command into a variable. It is possible to process the lines on-the-fly using a pipe:
curl --user admin:admin -k -X GET https://xxx.xx.xx.xxx:8080/profiles.json | \
jq --monochrome-output --unbuffered --raw-output \
'.Profiles[].id' | while read profile
do
echo "Processing profile $profile"
url="https://xxx.xx.xx.xxx:8080/testprofiles/${profile}/time/${downtimename}"
curl -X POST "$url" \
--data-urlencode "xml=$xml" \
-H 'Accept: application/xml' \
-H 'Content-Type: application/xml' \
-u admin:admin
done
In a bash script i want to get date dynamically and send it along with the
curl call.
i have got the date from the user in the bash script
and in the script am making the below curl call. am already passing the request params using a separate file as below. How do i pass the date?
i have tried like $date, but it is not working, even tried "'$date'".
The below is my curl call:
curl -O -X POST -H "Content-Type: application/json" -d#formparams.json --url http://test.com
Contents of form params json: it has more than 10 params for simplicity iam including only two
{"params":"{"HOSTS:":"1",date=$date}}
in the above i have added date.
But the date is not replaced.
Any help is appreciated.
Use a here document instead of a separate file for the parameters. Inside the here document, you can run the date command in a command substitution to provide the correct date when the document is read.
curl -O -X POST -H "Content-Type: application/json" -d#- --url http://test.com <<EOF
{"params":"{"HOSTS:":"1", "date": "$(date)"}}
EOF
use this json for pass dynamic parameter
{"params":"{"HOSTS:":"1",date="'$date'"}}