I have a BASH script that attempts to capture the output of build/deployment logs and insert them into a Jira ticket using Jira's REST API and CURL:
curl -v -X POST \
-H "Content-Type: application/json" \
--data "#header.json" \
--data "#log.txt" \
--data "#footer.json" \
-H "Authorization:Basic ABC123!##" \
https://companyname.jira.com/rest/api/latest/issue/FOO-1234/comment
My problem is that the logs contain all manner of JSON tokens, which causes the insert to fail. Is there a way from BASH to clean up the text blob before posting to escape out all the illegal characters? Or a way to say "don't parse anything in this block" or similar? Worst case, I'll write some really scary AWK.
Once upon a time I used this code snippet for sending POST data using curl.
urlquote() {
echo -ne "$1" | xxd -plain | tr -d '\n' | sed 's/\(..\)/%\1/g'
}
It works great with unicode stuff as well. Maybe this is going to help.
It turns out that all I needed to escape were the quotes and to convert newlines to \n. I used the following sed actions:
sed -inplace 's/\"/\\\"/g' log.txt
sed -inplace ':a;N;$!ba;s/\n/\\n/g' log.txt
Related
Using bash v4.3.48(1) on Ubuntu 16.04.2 LTS and bash v3.2.57 on macOS Sierra 10.12.5 I have this code which POSTs JSON data stored in a variable:
#1
JSON_DATA="Test"; \
curl -X POST "https://www.techotaku.com/json.php" \
-H "Content-Type: application/json" \
-d '{"data":"'$JSON_DATA'"}'
This returns:
Array
(
[data] => Test
)
However, if $JSON_DATA contains a space...
#2
JSON_DATA="A Test"; \
curl -X POST "https://www.techotaku.com/json.php" \
-H "Content-Type: application/json" \
-d '{"data":"'$JSON_DATA'"}'
...it throws an error:
curl: (3) [globbing] unmatched close brace/bracket in column 6
I thought this might be due to incorrect use of quotes, but if I echo the JSON data...
echo '{"data":"'$JSON_DATA'"}'
...it appears well-formed:
{"data":"A Test"}
Also, if I include the JSON data as a string and not a variable, it works.
#3
curl -X POST "https://www.techotaku.com/json.php" \
-H "Content-Type: application/json" \
-d '{"data":"A Test"}'
The only way I can get it to work if the variable contains a space is to enclose the JSON data in double-quotes and escape the literal double-quotes in the string:
#4
JSON_DATA="A Test"; \
curl -X POST "https://www.techotaku.com/json.php" \
-H "Content-Type: application/json" \
-d "{\"data\":\"$JSON_DATA\"}"
I would have thought that all 4 examples would have worked. So, can anyone throw any light on why only examples 1,3 and 4 work and example 2 doesn't?
Many thanks, Steve.
You should quote your variables to prevent word splitting by shell. Your second example:
'{"data":"'$JSON_DATA'"}'
doesn't work because it's being expanded by bash to two words:
{"data":"A Test"}
so -d option receives 2 arguments ({"data":"A and Test"}). If JSON_DATA contained globbing metacharacters, those would cause filename expansion.
To fix it, quote it:
'{"data":"'"$JSON_DATA"'"}'
Note the use of alternating single and double quotes. This will be expanded to a single word:
{"data":"A Test"}
Here's a more detailed answer I wrote some time ago which also mentions another convenient option, the use of a here-document.
I am using Gitbash, GNU bash, version 4.3.46(2)-release (x86_64-pc-msys).
I am running a bash script that looks like this
CODE CHUNK 1
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
'http://localhost:5000/api/123'
This works fine. Basically it posts a giant payload to an endpoint and all is well. The problem is when I attempt to substitute the url for a value from a variable, I get a curl error,
curl: (1) Protocol "'http" not supported or disabled in libcurl
CODE CHUNK 2
stuff=\'http://localhost:5000/api/123\'
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
$stuff
If I echo $stuff immediately after the stuff=\'http://localhost:5000/api/123\', I get 'http://localhost:5000/api/123'. This is the same value as I had hard-coded in code chunk 1, single ticks and all. There is something hiding behind the scenes in how that url is being evaluated after the variable has been expanded. I need to get the same behavior as a hard coded url.
Look closely at this error message:
curl: (1) Protocol "'http" not supported or disabled in libcurl
Notice the single-quote in front of http.
The curl command surely knows the http protocol, but not the 'http protocol!
The way you wrote it, the single-quotes are part of the value of stuff:
stuff=\'http://localhost:5000/api/123\'
Remove those, write like this:
stuff='http://localhost:5000/api/123'
If you have variables inside your real string,
and you want them expanded, then use double-quotes instead of single-quotes:
stuff="http://localhost:5000/api/123"
Equally important,
when you use $stuff as a parameter of curl,
you must double-quote it.
If you just write curl $stuff,
then the shell may interpret some characters in $stuff before passing to curl.
To protect from that,
you must write curl "$stuff".
The complete command:
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
"$stuff"
Finally, make sure that after each \ at the end of lines,
there's nothing after the \ on each line,
the \ must be at the very end.
Why do you define stuff with the ' ' ?
Try it like that:
stuff="http://localhost:5000/api/123"
curl -i -X POST \
-H "Content-Type:application/json" \
-H "x-customheader:customstuff" \
-d \
'{
Gigantic json payload contents in here
}' \
"$stuff"
Also, don't put variables in single quotes, because bash is not available to understand them.
stuff="http://localhost:5000/api/123"
echo "$stuff"
>> http://localhost:5000/api/123
echo '$stuff'
>> $stuff
I have been trying to call the CloudFlare API v4, using an example provided in their own documentation.
This is the code of the example
curl -X PUT "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records/372e67954025e0ba6aaa6d586b9e0b59" \ -H "X-Auth-Email: user#example.com" \ -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \ -H "Content-Type: application/json" \ --data '{"id":"372e67954025e0ba6aaa6d586b9e0b59","type":"A","name":"example.com","content":"1.2.3.4","proxiable":true,"proxied":false,"ttl":120,"locked":false,"zone_id":"023e105f4ecef8ad9ca31a8372d0c353","zone_name":"example.com","created_on":"2014-01-01T05:20:00.12345Z","modified_on":"2014-01-01T05:20:00.12345Z","data":{}}'
Which can also be found at
Update DNS Records
Using Windows cmd.exe to run this command, I need to make it single line first, so I removed the "" and reformatted it (twice) making sure I altered no part in the process.
This is the same code in one line:
curl -X PUT "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records/372e67954025e0ba6aaa6d586b9e0b59" -H "X-Auth-Email: user#example.com" -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" -H "Content-Type: application/json" --data '{"id":"372e67954025e0ba6aaa6d586b9e0b59","type":"A","name":"example.com","content":"1.2.3.4","proxiable":true,"proxied":false,"ttl":120,"locked":false,"zone_id":"023e105f4ecef8ad9ca31a8372d0c353","zone_name":"example.com","created_on":"2014-01-01T05:20:00.12345Z","modified_on":"2014-01-01T05:20:00.12345Z","data":{}}'
When I run this single-liner in cmd, it works but I get a malformed JSON in request body, however, a visual check, formatting on Notepad++ and a run through the JSON validator are all positive, this JSON (copied from the CloudFlare documentation) is not malformed.
Error Message
{"success":false,"errors":[{"code":6007,"message":"Malformed JSON in request body"}],"messages":[],"result":null}
Googling this error message or the error code gives me nothing and this same command works on a PC running Linux.
Can someone tell me if this is a known bug, if the JSON really is malformed or if something else comes to mind?
I found the answer in the blog post: "Expecting to find valid JSON in request body..." curl for Windows.
For example, for Purge everything --data value will be:
# On Linux
--data '{"purge_everything":true}'
# On Windows
--data "{\"purge_everything\":true}"
For Windows:
Replace the single quotes with double quotes: ' --> "
Escape the double quotes with a backslash: " --> \"
cmd.exe doesn't support single quotes, to run those commands straight from the docs you can use Bash.
Bash can be enabled in Windows 10 : https://www.laptopmag.com/uk/articles/use-bash-shell-windows-10
or Git Bash comes with Git for windows: https://gitforwindows.org/
users.I want to run a curl command with a bash script.
(below command works perfectly in terminal)
curl -i -H "Content-Type: application/json" -X POST -d '{"mountpoint":"/gua-la-autentica-1426251559"}' http://127.0.0.1:5000/connect
But unable to run this command in bash. When mountpoint value is given in a variable($final).
final="/gua-la-autentica-1426251559"
curl -i -H "Content-Type: application/json" -X POST -d '{"mountpoint":'$final'}' http://127.0.0.1:5000/connect
Could someone please help me, how to echo variable inside single quotes?
JSON string values should be quoted and so should parameter expansions. You can achieve this by using double quotes around the entire JSON string and escaping the inner double quotes, like this:
curl -i -H "Content-Type: application/json" -X POST -d "{\"mountpoint\":\"$final\"}" http://127.0.0.1:5000/connect
As mentioned in the comments, a more robust approach would be to use a tool such as jq to generate the JSON:
json=$(jq -n --arg final "$final" '{ mountpoint: $final }')
curl -i -H "Content-Type: application/json" -X POST -d "$json" http://127.0.0.1:5000/connect
I'd probably do it with a printf format. It makes it easier to see formatting errors, and gives you better control over your output:
final="/gua-la-autentica-1426251559"
fmt='{"mountpoint":"%s"}'
curl -i -H "Content-Type: application/json" -X POST \
-d "$(printf "$fmt" "$final")" \
http://127.0.0.1:5000/connect
I don't know where you're getting $final in your actual use case, but you might also want to consider checking it for content that would break your JSON. For example:
Portable version:
if expr "$final" : '[A-Za-z0-9./]*$'; then
curl ...
else
echo "ERROR"
fi
Bash-only version (perhaps better performance but less portable):
if [[ "$final" ]~ ^[A-Za-z0-9./]*$ ]]; then
curl ...
...
Checking your input is important if there's even the remote possibility that your $final variable will be something other than what you're expecting. You don't have valid JSON anymore if it somehow includes a double quote.
Salt to taste.
I have a source file with JSON-Objects one per line like this:
source:
{"_id":"1","name":"one"}
{"_id":"2","name":"two"}
{"_id":"3","name":"three"}
I want to send each line to a
curl -X POST -H "application/json" myURL -d '<REPLACEMENT>'
The double quotes do not make it to curl when I am trying
<source xargs -I % curl -X POST -H "application/json" myURL -d '%'
I tried escaping the quotes in the curl command and later I replaced all double-quotes in the source file with \". I found no version to work.
Another approach to use seq with sed to write each line into a temp file and curl -d #temp did not work out for me.
Is there an elegant solution or do I have to write a script with a loop?
That's an interesting problem. There must be a better solution (perhaps konsolebox is onto something), but substituting all " with \" would work:
$ echo '"hello"'
"hello"
$ echo '"hello"' | xargs echo
hello
$ echo '"hello"' | sed 's/"/\\"/g' | xargs echo
"hello"
GNU Parallel was built specifically to deal with xargs bad handling of special chars:
<source parallel curl -X POST -H "application/json" myURL -d {}
Not only will it quote " correctly, it will quote any string correctly, so it will be interpreted as a single argument by curl.
Added bonus: Your queries will run in parallel - one query per cpu.
Should do the trick:
cat source.json | xargs -0 -I {} curl {}
From man xargs:
-0, --null
Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (ev‐
ery character is taken literally). Disables the end of file string, which is treated like any other argument. Useful
when input items might contain white space, quote marks, or backslashes. The GNU find -print0 option produces input suit‐
able for this mode.
Try to use --data-urlencode:
<source xargs -I % curl -X POST -H "application/json" myURL --data-urlencode '%'
The option may be used with other formats. See the manual of curl. You can also try --data-binary.