I have a source file with JSON-Objects one per line like this:
source:
{"_id":"1","name":"one"}
{"_id":"2","name":"two"}
{"_id":"3","name":"three"}
I want to send each line to a
curl -X POST -H "application/json" myURL -d '<REPLACEMENT>'
The double quotes do not make it to curl when I am trying
<source xargs -I % curl -X POST -H "application/json" myURL -d '%'
I tried escaping the quotes in the curl command and later I replaced all double-quotes in the source file with \". I found no version to work.
Another approach to use seq with sed to write each line into a temp file and curl -d #temp did not work out for me.
Is there an elegant solution or do I have to write a script with a loop?
That's an interesting problem. There must be a better solution (perhaps konsolebox is onto something), but substituting all " with \" would work:
$ echo '"hello"'
"hello"
$ echo '"hello"' | xargs echo
hello
$ echo '"hello"' | sed 's/"/\\"/g' | xargs echo
"hello"
GNU Parallel was built specifically to deal with xargs bad handling of special chars:
<source parallel curl -X POST -H "application/json" myURL -d {}
Not only will it quote " correctly, it will quote any string correctly, so it will be interpreted as a single argument by curl.
Added bonus: Your queries will run in parallel - one query per cpu.
Should do the trick:
cat source.json | xargs -0 -I {} curl {}
From man xargs:
-0, --null
Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (ev‐
ery character is taken literally). Disables the end of file string, which is treated like any other argument. Useful
when input items might contain white space, quote marks, or backslashes. The GNU find -print0 option produces input suit‐
able for this mode.
Try to use --data-urlencode:
<source xargs -I % curl -X POST -H "application/json" myURL --data-urlencode '%'
The option may be used with other formats. See the manual of curl. You can also try --data-binary.
Related
I'm working on a bash script and have a curl cmd :
curl -u root:$pass -X PUT -H "Content-Type:application/json" http://$ip/nitro/ \
-d '{"user":{"username":"test","password":"test123" }}' -k
For password test123 in the above curl cmd, i want to pass it as an environment variable.
Have tried following ways but it isn't working.
curl -u root:$pass -X PUT -H "Content-Type:application/json" http://$ip/nitro/ \
-d '{"user":{"username":"test","password":$new_pswrd }}' -k
curl -u root:$pass -X PUT -H "Content-Type:application/json" http://$ip/nitro/ \
-d '{"user":{"username":"test","password":\"$new_pswrd\" }}' -k
Please suggest what could be wrong here, and how can I escape double quotes in post JSON parameter for curl cmd ?
You can write like this:
[STEP 101] $ pass=test123
[STEP 102] $ echo '{"user":{"username":"test","password":"'"$pass"'" }}'
{"user":{"username":"test","password":"test123" }}
[STEP 103] $
(This would not work if the password includes chars like ". I don't know the syntax for JSON to escape " and other special chars so I cannot comment more.)
You used single quotes. Bash allows two different quoting mechanisms:
Single quote: Everything enclosed is literal.
Double quotes: Interpret $ and \, pass the rest as literal.
Your bash environment variable $new_pswrd is in single quotes and will NOT be expanded, but will be passed as a dollar sign and the letters new_pswrd.
There are two way to achieve your goal:
Using single quotes: 'End the single quote '"and put your $variable in double quotes"' and bash will concatenate all of the strings '" because they lack white space between "" the open and close quotes "_or_lack_white_space_entirely.
Using double quotes: "A backslash will be interpreted when inside \"double quotes\" and the backslash will cause the following character, even it it is a double quote, to be pass through without closing the literal \". Dollar signs will be expanded and $shell_variable will be replaced with its value while \$shell_variable will not."
I'm trying to write a sample script where I'm generating names like 'student-101...student-160'. I need to post JSON data and when I do, I get a JSON parse error.
Here's my script:
name="student-10"
for i in {1..1}
do
r_name=$name$i
echo $r_name
curl -i -H 'Authorization: token <token>' -d '{"name": $r_name, "private": true}' "<URL>" >> create_repos_1.txt
echo created $r_name
done
I always get a "Problems parsing JSON" error. I've tried various combination of quotes, etc but nothing seems to work!
What am I doing wrong?
First, your name property is a string, so you need to add double quotes to it in your json.
Second, using single quotes, bash won't do variable expansion: it won't replace $r_name with the variable content (see Expansion of variable inside single quotes in a command in bash shell script for more information).
In summary, use:
-d '{"name": "'"$r_name"'", "private": true}'
Another option is to use printf to create the data string:
printf -v data '{"name": "%s", "private": true}' "$r_name"
curl -i -H 'Authorization: token <token>' -d "$data" "$url" >> create_repos_1.txt
Don't; use jq (or something similar) to build correctly quoted JSON using variable inputs.
name="student-10"
for i in {1..1}
do
r_name=$name$i
jq -n --arg r_name "$r_name" '{name: $r_name, private: true}' |
curl -i -H 'Authorization: token <token>' -d #- "<URL>" >> create_repos_1.txt
echo created $r_name
done
The #- argument tells curl to read data from standard input (via the pipe from jq) to use for -d.
Something like "{\"name\": \"$r_name\", \"private\": true}" may work, but it is ugly and will also fail if r_name contains any character which needs to be quoted in the resulting JSON, such as double quotes or ASCII control characters.
Looking for some advice on the direction I should take my project as I'm very new to scripting. My objectives are as follows:
Using cURL to get JSON data, parse a particular object value. I was intending to use jq to parse the JSON and then store the results as a variable.
Here is what I have to started with:
SAMPLE JSON -
{
"href": "http://localhost:8080//profiles",
"Profiles": [
{
"href": "http://localhost:8080/profiles/myprofile",
"id": "myprofile",
"isRecording": false
},
{
"href": "http://localhost:8080/profiles/yourprofile",
"id": "yourprofile",
"isRecording": false
}
]
}
BASH SCRIPT -
#!/bin/bash
read -p "Please enter downtime name (and press Enter): " downtimename
read -p "Please enter timestart:" timeStart
read -p "Please enter time duration (in minutes): " durationMinutes
#!/bin/bash
PROFILE="$(curl --user admin:admin -k -X GET https://xxx.xx.xx.xxx:8080/profiles.json | jq '.Profiles[].id')"
echo "$PROFILE"
RETURNS -
"myprofile"
"yourprofile"
Next I need to construct 1 or more cURL POST of xml data (example below) Each post will be a single curl post of the xml, for each line in the above echo "$PROFILE". I’m thinking this will be a for loop? What I'm struggling with is how to read each value/line from "$PROFILE" and utilize a for loop to post xml while replacing ${profile] in the below curl URL, for each result above.
curl -X POST https://xxx.xx.xx.xxx:8080/testprofiles/${profile}/time/${downtimename} --data-urlencode xml="<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit="MINUTES">${durationMinutes}</duration>
</gametime>
</time>" -H 'Accept: application/xml' \ -H 'Content-Type: application/xml' -u admin:admin
Thank you in advance for the help
Strings and quotes
The XML string you pass to --data-urlencode is interpreted without double quotes:
<?xml version=1.0 encoding=UTF-8 standalone=yes?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit=MINUTES>${durationMinutes}</duration>
</gametime>
</time>
Double quotes are the shell syntax elements that influence parsing. They are removed before the command is called. Prepend a backslash to a double quote, if you want its literal value in the string, e.g. "version=\"1.0\"".
However, here documents are more convenient in most cases:
xml=$(cat <<XML
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<time>
<gametime>
2016-11-23T05:30:00+02:00
<start>${timeStart}</start>
<duration unit="MINUTES">${durationMinutes}</duration>
</gametime>
</time>
XML
)
jq output
Use --raw-output option to skip the JSON formatting. (In particular, string values are wrapped with double quotes by default.).
Turn off color output with --monochrome-output option. With this option, jq will not print the ASCII escape codes.
Although not necessary, I recommend turning off the output buffering with --unbuffered option. With this option, jq will flush the output buffer after each JSON object is printed.
Reading jq output line-by-line
You don't need to store the output of jq command into a variable. It is possible to process the lines on-the-fly using a pipe:
curl --user admin:admin -k -X GET https://xxx.xx.xx.xxx:8080/profiles.json | \
jq --monochrome-output --unbuffered --raw-output \
'.Profiles[].id' | while read profile
do
echo "Processing profile $profile"
url="https://xxx.xx.xx.xxx:8080/testprofiles/${profile}/time/${downtimename}"
curl -X POST "$url" \
--data-urlencode "xml=$xml" \
-H 'Accept: application/xml' \
-H 'Content-Type: application/xml' \
-u admin:admin
done
users.I want to run a curl command with a bash script.
(below command works perfectly in terminal)
curl -i -H "Content-Type: application/json" -X POST -d '{"mountpoint":"/gua-la-autentica-1426251559"}' http://127.0.0.1:5000/connect
But unable to run this command in bash. When mountpoint value is given in a variable($final).
final="/gua-la-autentica-1426251559"
curl -i -H "Content-Type: application/json" -X POST -d '{"mountpoint":'$final'}' http://127.0.0.1:5000/connect
Could someone please help me, how to echo variable inside single quotes?
JSON string values should be quoted and so should parameter expansions. You can achieve this by using double quotes around the entire JSON string and escaping the inner double quotes, like this:
curl -i -H "Content-Type: application/json" -X POST -d "{\"mountpoint\":\"$final\"}" http://127.0.0.1:5000/connect
As mentioned in the comments, a more robust approach would be to use a tool such as jq to generate the JSON:
json=$(jq -n --arg final "$final" '{ mountpoint: $final }')
curl -i -H "Content-Type: application/json" -X POST -d "$json" http://127.0.0.1:5000/connect
I'd probably do it with a printf format. It makes it easier to see formatting errors, and gives you better control over your output:
final="/gua-la-autentica-1426251559"
fmt='{"mountpoint":"%s"}'
curl -i -H "Content-Type: application/json" -X POST \
-d "$(printf "$fmt" "$final")" \
http://127.0.0.1:5000/connect
I don't know where you're getting $final in your actual use case, but you might also want to consider checking it for content that would break your JSON. For example:
Portable version:
if expr "$final" : '[A-Za-z0-9./]*$'; then
curl ...
else
echo "ERROR"
fi
Bash-only version (perhaps better performance but less portable):
if [[ "$final" ]~ ^[A-Za-z0-9./]*$ ]]; then
curl ...
...
Checking your input is important if there's even the remote possibility that your $final variable will be something other than what you're expecting. You don't have valid JSON anymore if it somehow includes a double quote.
Salt to taste.
I have a BASH script that attempts to capture the output of build/deployment logs and insert them into a Jira ticket using Jira's REST API and CURL:
curl -v -X POST \
-H "Content-Type: application/json" \
--data "#header.json" \
--data "#log.txt" \
--data "#footer.json" \
-H "Authorization:Basic ABC123!##" \
https://companyname.jira.com/rest/api/latest/issue/FOO-1234/comment
My problem is that the logs contain all manner of JSON tokens, which causes the insert to fail. Is there a way from BASH to clean up the text blob before posting to escape out all the illegal characters? Or a way to say "don't parse anything in this block" or similar? Worst case, I'll write some really scary AWK.
Once upon a time I used this code snippet for sending POST data using curl.
urlquote() {
echo -ne "$1" | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g'
}
It works great with unicode stuff as well. Maybe this is going to help.
It turns out that all I needed to escape were the quotes and to convert newlines to \n. I used the following sed actions:
sed -inplace 's/\"/\\\"/g' log.txt
sed -inplace ':a;N;$!ba;s/\n/\\n/g' log.txt