I'm trying to create a script that will use the Github API to post a comment containing the output of a command. This output has multiple lines.
Here's what I'm trying to do:
curl -H "Authorization: token oauthtoken" \
-H "Content-Type: application/json" \
-X POST -d#- \
https://api.github.com/repos/company/repo/issues/14/comments <<EOF
{
"body":"$OUTPUT"
}
EOF
How can I output the variable in such a way that it respects the multiple lines contained within? Now when I run that command, all of the newlines get squished on to one line.
I don't think that the basic cause of the problem are the newlines, the issues is that the value of $text is not properly formatted json.
Follow this simple example:
test="
Hello
World
"
curl -X POST -d '{"body": "'"$test"'"}' http://server.com/...
to see new lines working.
To make it possible to send the result of arbitrary commands using json, you need to json-encode the text before.
Related
I am trying to push some data to my Tiddlywiki via api so I can create a new tiddler.
I can pass almost all the data, but if the value "text" has spaces inside it breaks the curl command trying to pass multiple urls.
This is the actual command:
curl -X PUT -i "http://192.168.1.166:8080/recipes/default/tiddlers/$title" -H "X-Requested-With: TiddlyWiki" --data $(jq -nc --arg tags "$tag" --arg text "'${content}'" '{ $tags, $text }')
I tried at first using $variables, inside brackets, even inside multiple "'"$var"'" following others questions here. But most of them quickly recommended using jq so I gave it a try.
I learned how to create some keys with the bash¡s variables contents, and If I pipe all of this I can get it to work only if I replace spaces with other characters...I tried %20 or scaping the space \ with sed whitout success. (%20 is replaced literally so not helpful)
Any recommendations at all, I will follow any other path you could bring to it.
Thanks.
EDIT:
I tried using --data-urlencode "text=${var}" but it wasnt filled, only variable expanded was the title. The others didnt show at all.
API'S INFO:
https://tiddlywiki.com/static/WebServer%2520API%253A%2520Put%2520Tiddler.html
I forgot to mention that I am using zsh shell...
You need to quote the output of the command substitution: --data "$(jq ...)". Without this, curl thinks the words after the first one are individual URLs to connect to. You should also remove the single quotes from --arg text "'${content}'", otherwise they'll be added to the text itself.
within my bash script I have a line that looks like this:
JOB_RESULTS=$(curl --fail -k -H "Content-Type: application/xml" --write-out HTTP_CODE='%{http_code}' "${request_url}")
This line outputs the following:
{"name":"callCLF010Job","id":11693,"status":"STARTING","message":"The job has started.","exitCode":"exitCode=UNKNOWN;exitDescription="}HTTP_CODE=200
The problem is, I want to add the HTTP_CODE into the JSON message.
Is there a way to make the return look like {"name":"callCLF010Job","id":11693,"status":"STARTING","message":"The job has started.","exitCode":"exitCode=UNKNOWN;exitDescription=","HTTP_CODE":"200"}
EDIT:
With confettis changes my code looks like this:
http_code="${JOB_RESULTS:${#JOB_RESULTS}-17}"
http_body="${JOB_RESULTS:0:${#JOB_RESULTS}-17}"
http_code_json=", ${http_code}}"
my_result="${http_body/%\}/$http_code_json}"
when I run echo $my_result my output looks like this:
{"name":"callCLF010Job","id":11702,"status":"STARTING","message":"The job has started.","exitCode":"exitCode=UNKNOWN;exitDescriptio
One option is to use sed to format the output the way you want:
echo $JOB_RESULTS | sed 's/}HTTP_CODE=\([0-9]\+\)$/,"HTTP_CODE":"\1"}/'
Or inside your command:
JOB_RESULTS=$(curl --fail -k -H "Content-Type: application/xml" --write-out HTTP_CODE='%{http_code}' "${request_url}" | sed 's/}HTTP_CODE=\([0-9]\+\)$/,"HTTP_CODE":"\1"}/')
This is using shell parameter expansion to get the job done. (Pure bash, besides curl.)
curlout=$(curl -s --fail -k -H "Content-Type: application/xml" --write-out '"HTTP_CODE":"%{http_code}"' "http://example.com")
http_code="${curlout:${#curlout}-17}"
http_body="${curlout:0:${#curlout}-17}"
http_code_json=", ${http_code}}"
my_result="${http_body/%\}/$http_code_json}"
Replace http://example.com with the URL or variable of your choice.
In order to prevent errors in case of a wrong URL or (no) output, you should put the last four lines within an if-construct. E.g. if [[ $http_code != '"HTTP_CODE":"000"' ]].
Using GET I need to pass a json value to a URL via the command line within a bash script.
This works:
curl -i "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit&conf=\{\"filename\":\"myfile.csv\"\}"
If I want to expand on the json value, I would prefer to pass a variable via the URL parameter for readability. Somethig like ... but this doesn't appear to work correctly.
generate_post_data =
{
"filename": "myfile.csv"
}
curl -i "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit&conf=${generate_post_data}"
You need to properly set the variable and you should url encode it using the --data-urlencode option.
#!/bin/bash
generate_post_data="filename=myfile.csv"
curl -G "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit" --data-urlencode $generate_post_data
From the manpage:
--data-urlencode <data>
(HTTP) This posts data, similar to the other -d, --data options with
the exception that this performs URL-encoding.
To be CGI-compliant, the part should begin with a name followed
by a separator and a content specification. The part can be
passed to curl using one of the following syntaxes:
For more info you can use man curl and then /data-urlencode to jump to the section on it.
The two commands below work perfectly outside of a script if I execute them one after the other, however, if I put them inside a bash script. They fail.
DATA="{\"size\":500,\"sort\":{\"#timestamp\":\"desc\"},\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"tx.traceId: AIP1447283489-6 AND event: published_notification AND attempt: 1\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"#timestamp\":{\"gte\":1420099200000,\"lte\":1451635199999}}}],\"must_not\":[]}}}},\"highlight\":{\"pre_tags\":[\"#kibana-highlighted-field#\"],\"post_tags\":[\"#\/kibana-highlighted-field#\"],\"fields\":{\"*\":{}}},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"#timestamp\",\"interval\":\"1w\",\"pre_zone\":\"-08:00\",\"pre_zone_adjust_large_interval\":true,\"min_doc_count\":0,\"extended_bounds\":{\"min\":1420099200000,\"max\":1451635199999}}}},\"fields\":[\"*\",\"_source\"],\"script_fields\":{},\"fielddata_fields\":[\"log.timestamp\",\"#timestamp\",\"val\"]}"
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA | jq '.hits.hits[0].fields."log.timestamp"[0]'
By using set -x I've found that the final curl command that is executed is:
curl -s -XPOST 'http://username:password#url/elasticsearch/index-*/_search' -d '{"size":500,"sort":{"#timestamp":"desc"},"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"}},"filter":{"bool":{"must":[{"range":{"#timestamp":{"gte":1420099200000,"lte":1451635199999}}}],"must_not":[]}}}},"highlight":{"pre_tags":["#kibana-highlighted-field#"],"post_tags":["#\/kibana-highlighted-field#"],"fields":{"*":{}}},"aggs":{"2":{"date_histogram":{"field":"#timestamp","interval":"1w","pre_zone":"-08:00","pre_zone_adjust_large_interval":true,"min_doc_count":0,"extended_bounds":{"min":1420099200000,"max":1451635199999}}}},"fields":["*","_source"],"script_fields":{},"fielddata_fields":["log.timestamp","#timestamp","val"]}'
And if you notice there are extra single quotes like so: ' added around the value of "query". As you can see here:
"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"
What the heck is going on and how do I use these two commands with this json in a script?
It's much easier to read from a here document than to ensure you've quoted all the quotation marks properly.
url='http://username:password#url/elasticsearch/index-*/_search'
curl -s -X POST "$url" -d#- <<EOF | jq '.hits.hits[0].fields."log.timestamp"[0]'
{ "size":500,
"sort": {
...
}
EOF
You didn't quote $DATA in your command
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA
so $DATA is subject to word splitting (and more) after parameter expansion is performed. Your $DATA contains whitespace in the value of query, so it's split there and broken into multiple arguments. What you want is
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d "$DATA"
Also, my personal advice is to quote JSON string with single quotes, and use '\'' when you need a literal single quote. Quoting JSON with double quotes in shell just makes the result impossible to read...
I'm trying to POST data from a bash variable using curl, however, I'm unable to get this to work. This is the command that I'm using:
escape() { printf "%q" "$1"; }
curl -d "$(escape "$client")" -X POST -v https://$server/clients
The client variable look like this:
{"roles":["test"],"softwareName":"Some Soft","passwordSalt":"aaa","clientID":"full-client-2","contactPerson":"Test","contactPersonEmail":"a#b.org","description":"test","name":"Full Client-2","organization":"Some Org","passwordAlgorithm":"sha512","passwordHash":"bbb"}
And on the server I'm receiving the following:
{ '{"roles":': { '"test"': { '\"test\"\': '' } } }
I think its a problem with the escaping but I can't figure this out.
I've had a look at a number of other questions about this on here, but it seems most people need to insert variable into a literal that they are then trying to post. My problem is around using an entire variable as the json body. I've tried to use their answers to help me out but I haven't had any luck so far.
Don't try to quote it; use a here document:
curl -d#- -X POST -v https://"$server"/clients <<JSON
{"roles":["test"],"softwareName":"Some Soft","passwordSalt":"aaa","clientID":"full-client-2","contactPerson":"Test","contactPersonEmail":"a#b.org","description":"test","name":"Full Client-2","organization":"Some Org","passwordAlgorithm":"sha512","passwordHash":"bob"}
JSON
#- tells the -d option to look in standard input for the data, rather than using a hard-coded string.
If the text is in a variable, nothing more needs to be done; just quote the variable to prevent the shell from processing it:
curl -d "$client" -X POST -v https://"$server"/clients