How to pass bash variable to JSON - json

I'm trying to write a sample script where I'm generating names like 'student-101...student-160'. I need to post JSON data and when I do, I get a JSON parse error.
Here's my script:
name="student-10"
for i in {1..1}
do
r_name=$name$i
echo $r_name
curl -i -H 'Authorization: token <token>' -d '{"name": $r_name, "private": true}' "<URL>" >> create_repos_1.txt
echo created $r_name
done
I always get a "Problems parsing JSON" error. I've tried various combination of quotes, etc but nothing seems to work!
What am I doing wrong?

First, your name property is a string, so you need to add double quotes to it in your json.
Second, using single quotes, bash won't do variable expansion: it won't replace $r_name with the variable content (see Expansion of variable inside single quotes in a command in bash shell script for more information).
In summary, use:
-d '{"name": "'"$r_name"'", "private": true}'

Another option is to use printf to create the data string:
printf -v data '{"name": "%s", "private": true}' "$r_name"
curl -i -H 'Authorization: token <token>' -d "$data" "$url" >> create_repos_1.txt

Don't; use jq (or something similar) to build correctly quoted JSON using variable inputs.
name="student-10"
for i in {1..1}
do
r_name=$name$i
jq -n --arg r_name "$r_name" '{name: $r_name, private: true}' |
curl -i -H 'Authorization: token <token>' -d #- "<URL>" >> create_repos_1.txt
echo created $r_name
done
The #- argument tells curl to read data from standard input (via the pipe from jq) to use for -d.
Something like "{\"name\": \"$r_name\", \"private\": true}" may work, but it is ugly and will also fail if r_name contains any character which needs to be quoted in the resulting JSON, such as double quotes or ASCII control characters.

Related

Cannot expand and substitute the contents of a variable in Bash

I am using Gitbash, GNU bash, version 4.3.46(2)-release (x86_64-pc-msys).
I am running a bash script that looks like this
CODE CHUNK 1
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
'http://localhost:5000/api/123'
This works fine. Basically it posts a giant payload to an endpoint and all is well. The problem is when I attempt to substitute the url for a value from a variable, I get a curl error,
curl: (1) Protocol "'http" not supported or disabled in libcurl
CODE CHUNK 2
stuff=\'http://localhost:5000/api/123\'
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
$stuff
If I echo $stuff immediately after the stuff=\'http://localhost:5000/api/123\', I get 'http://localhost:5000/api/123'. This is the same value as I had hard-coded in code chunk 1, single ticks and all. There is something hiding behind the scenes in how that url is being evaluated after the variable has been expanded. I need to get the same behavior as a hard coded url.
Look closely at this error message:
curl: (1) Protocol "'http" not supported or disabled in libcurl
Notice the single-quote in front of http.
The curl command surely knows the http protocol, but not the 'http protocol!
The way you wrote it, the single-quotes are part of the value of stuff:
stuff=\'http://localhost:5000/api/123\'
Remove those, write like this:
stuff='http://localhost:5000/api/123'
If you have variables inside your real string,
and you want them expanded, then use double-quotes instead of single-quotes:
stuff="http://localhost:5000/api/123"
Equally important,
when you use $stuff as a parameter of curl,
you must double-quote it.
If you just write curl $stuff,
then the shell may interpret some characters in $stuff before passing to curl.
To protect from that,
you must write curl "$stuff".
The complete command:
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
"$stuff"
Finally, make sure that after each \ at the end of lines,
there's nothing after the \ on each line,
the \ must be at the very end.
Why do you define stuff with the ' ' ?
Try it like that:
stuff="http://localhost:5000/api/123"
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
"$stuff"
Also, don't put variables in single quotes, because bash is not available to understand them.
stuff="http://localhost:5000/api/123"
echo "$stuff"
>> http://localhost:5000/api/123
echo '$stuff'
>> $stuff

Parsing an HTTP response from a curl POST [duplicate]

This question already has answers here:
A better way to extract JSON value in bash script
(2 answers)
Closed 6 years ago.
I'm currently trying to grab and assign the N_596164000673190002 to a variable from a curl command.
This is the command:
curl -L -H 'X-Cisco-Meraki-API-Key: mykeygoeshere' -X POST -H'Content-Type: application/json' --data-binary '{"name":"'"$NETWORK_NAME"'", "type":"appliance", "timeZone":"'"$TIME_ZONE"'"}' 'https://dashboard.meraki.com/api/v0/organizations/foobar/networks'
This is the response:
{"id":"N_596164000673190002","organizationId":"foo","type":"appliance","name":"bar","timeZone":"America/Chicago","tags":""}
How do I successfully read and grab the variable after id (without the double quotes), while also simutaneously assigning it to a variable, $NETWORK_ID? I imagine this can all be done in one line.
If this is successful, echo $NETWORK_ID should return N_596164000673190002
To parse json in bash, people usually use jq as it is installed by default on most Unix distributions.
Try the following :
NETWORK_ID=$(my_curl_command | jq -r '.id')
Here, '.id' is a filter indicating we want to retrieve the value for the key id, and the -r flag is used to remove double quotes from the output.
Pipe the JSON output to python json to grab the id value you need, and use bash command substitution to assign the result to your NETWORK_ID environment variable.
NETWORK_ID=$(curl -L -H 'X-Cisco-Meraki-API-Key: mykeygoeshere' -X POST \
-H'Content-Type: application/json' \
--data-binary '{"name":"'"$NETWORK_NAME"'", "type":"appliance", \
"timeZone":"'"$TIME_ZONE"'"}' \
'https://dashboard.meraki.com/api/v0/organizations/foobar/networks' \
| python -c "import sys, json; print json.load(sys.stdin)['id']")

Bash Script Project - Parse JSON and construct cURL POST of XML within loop

Looking for some advice on the direction I should take my project as I'm very new to scripting. My objectives are as follows:
Using cURL to get JSON data, parse a particular object value. I was intending to use jq to parse the JSON and then store the results as a variable.
Here is what I have to started with:
SAMPLE JSON -
{
    "href": "http://localhost:8080//profiles",
    "Profiles": [
        {
            "href": "http://localhost:8080/profiles/myprofile",
            "id": "myprofile",
            "isRecording": false
        },
        {
            "href": "http://localhost:8080/profiles/yourprofile",
            "id": "yourprofile",
            "isRecording": false
        }
    ]
}
BASH SCRIPT -
#!/bin/bash
read -p "Please enter downtime name (and press Enter): " downtimename
read -p "Please enter timestart:" timeStart
read -p "Please enter time duration (in minutes): " durationMinutes
#!/bin/bash
PROFILE="$(curl --user admin:admin -k -X GET https://xxx.xx.xx.xxx:8080/profiles.json | jq '.Profiles[].id')"
echo "$PROFILE"
RETURNS -
"myprofile"
"yourprofile"
Next I need to construct 1 or more cURL POST of xml data (example below) Each post will be a single curl post of the xml, for each line in the above echo "$PROFILE". I’m thinking this will be a for loop? What I'm struggling with is how to read each value/line from "$PROFILE" and utilize a for loop to post xml while replacing ${profile] in the below curl URL, for each result above.
curl -X POST https://xxx.xx.xx.xxx:8080/testprofiles/${profile}/time/${downtimename} --data-urlencode xml="<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit="MINUTES">${durationMinutes}</duration>
</gametime>
</time>" -H 'Accept: application/xml' \ -H 'Content-Type: application/xml' -u admin:admin
Thank you in advance for the help
Strings and quotes
The XML string you pass to --data-urlencode is interpreted without double quotes:
<?xml version=1.0 encoding=UTF-8 standalone=yes?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit=MINUTES>${durationMinutes}</duration>
</gametime>
</time>
Double quotes are the shell syntax elements that influence parsing. They are removed before the command is called. Prepend a backslash to a double quote, if you want its literal value in the string, e.g. "version=\"1.0\"".
However, here documents are more convenient in most cases:
xml=$(cat <<XML
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit="MINUTES">${durationMinutes}</duration>
</gametime>
</time>
XML
)
jq output
Use --raw-output option to skip the JSON formatting. (In particular, string values are wrapped with double quotes by default.).
Turn off color output with --monochrome-output option. With this option, jq will not print the ASCII escape codes.
Although not necessary, I recommend turning off the output buffering with --unbuffered option. With this option, jq will flush the output buffer after each JSON object is printed.
Reading jq output line-by-line
You don't need to store the output of jq command into a variable. It is possible to process the lines on-the-fly using a pipe:
curl --user admin:admin -k -X GET https://xxx.xx.xx.xxx:8080/profiles.json | \
jq --monochrome-output --unbuffered --raw-output \
'.Profiles[].id' | while read profile
do
echo "Processing profile $profile"
url="https://xxx.xx.xx.xxx:8080/testprofiles/${profile}/time/${downtimename}"
curl -X POST "$url" \
--data-urlencode "xml=$xml" \
-H 'Accept: application/xml' \
-H 'Content-Type: application/xml' \
-u admin:admin
done

How to echo variable inside single quotes using Bash?

users.I want to run a curl command with a bash script.
(below command works perfectly in terminal)
curl -i -H "Content-Type: application/json" -X POST -d '{"mountpoint":"/gua-la-autentica-1426251559"}' http://127.0.0.1:5000/connect
But unable to run this command in bash. When mountpoint value is given in a variable($final).
final="/gua-la-autentica-1426251559"
curl -i -H "Content-Type: application/json" -X POST -d '{"mountpoint":'$final'}' http://127.0.0.1:5000/connect
Could someone please help me, how to echo variable inside single quotes?
JSON string values should be quoted and so should parameter expansions. You can achieve this by using double quotes around the entire JSON string and escaping the inner double quotes, like this:
curl -i -H "Content-Type: application/json" -X POST -d "{\"mountpoint\":\"$final\"}" http://127.0.0.1:5000/connect
As mentioned in the comments, a more robust approach would be to use a tool such as jq to generate the JSON:
json=$(jq -n --arg final "$final" '{ mountpoint: $final }')
curl -i -H "Content-Type: application/json" -X POST -d "$json" http://127.0.0.1:5000/connect
I'd probably do it with a printf format. It makes it easier to see formatting errors, and gives you better control over your output:
final="/gua-la-autentica-1426251559"
fmt='{"mountpoint":"%s"}'
curl -i -H "Content-Type: application/json" -X POST \
-d "$(printf "$fmt" "$final")" \
http://127.0.0.1:5000/connect
I don't know where you're getting $final in your actual use case, but you might also want to consider checking it for content that would break your JSON. For example:
Portable version:
if expr "$final" : '[A-Za-z0-9./]*$'; then
curl ...
else
echo "ERROR"
fi
Bash-only version (perhaps better performance but less portable):
if [[ "$final" ]~ ^[A-Za-z0-9./]*$ ]]; then
curl ...
...
Checking your input is important if there's even the remote possibility that your $final variable will be something other than what you're expecting. You don't have valid JSON anymore if it somehow includes a double quote.
Salt to taste.

xargs: how to have literal double-quotes in replacement?

I have a source file with JSON-Objects one per line like this:
source:
{"_id":"1","name":"one"}
{"_id":"2","name":"two"}
{"_id":"3","name":"three"}
I want to send each line to a
curl -X POST -H "application/json" myURL -d '<REPLACEMENT>'
The double quotes do not make it to curl when I am trying
<source xargs -I % curl -X POST -H "application/json" myURL -d '%'
I tried escaping the quotes in the curl command and later I replaced all double-quotes in the source file with \". I found no version to work.
Another approach to use seq with sed to write each line into a temp file and curl -d #temp did not work out for me.
Is there an elegant solution or do I have to write a script with a loop?
That's an interesting problem. There must be a better solution (perhaps konsolebox is onto something), but substituting all " with \" would work:
$ echo '"hello"'
"hello"
$ echo '"hello"' | xargs echo
hello
$ echo '"hello"' | sed 's/"/\\"/g' | xargs echo
"hello"
GNU Parallel was built specifically to deal with xargs bad handling of special chars:
<source parallel curl -X POST -H "application/json" myURL -d {}
Not only will it quote " correctly, it will quote any string correctly, so it will be interpreted as a single argument by curl.
Added bonus: Your queries will run in parallel - one query per cpu.
Should do the trick:
cat source.json | xargs -0 -I {} curl {}
From man xargs:
-0, --null
Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (ev‐
ery character is taken literally). Disables the end of file string, which is treated like any other argument. Useful
when input items might contain white space, quote marks, or backslashes. The GNU find -print0 option produces input suit‐
able for this mode.
Try to use --data-urlencode:
<source xargs -I % curl -X POST -H "application/json" myURL --data-urlencode '%'
The option may be used with other formats. See the manual of curl. You can also try --data-binary.