The two commands below work perfectly outside of a script if I execute them one after the other, however, if I put them inside a bash script. They fail.
DATA="{\"size\":500,\"sort\":{\"#timestamp\":\"desc\"},\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"tx.traceId: AIP1447283489-6 AND event: published_notification AND attempt: 1\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"#timestamp\":{\"gte\":1420099200000,\"lte\":1451635199999}}}],\"must_not\":[]}}}},\"highlight\":{\"pre_tags\":[\"#kibana-highlighted-field#\"],\"post_tags\":[\"#\/kibana-highlighted-field#\"],\"fields\":{\"*\":{}}},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"#timestamp\",\"interval\":\"1w\",\"pre_zone\":\"-08:00\",\"pre_zone_adjust_large_interval\":true,\"min_doc_count\":0,\"extended_bounds\":{\"min\":1420099200000,\"max\":1451635199999}}}},\"fields\":[\"*\",\"_source\"],\"script_fields\":{},\"fielddata_fields\":[\"log.timestamp\",\"#timestamp\",\"val\"]}"
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA | jq '.hits.hits[0].fields."log.timestamp"[0]'
By using set -x I've found that the final curl command that is executed is:
curl -s -XPOST 'http://username:password#url/elasticsearch/index-*/_search' -d '{"size":500,"sort":{"#timestamp":"desc"},"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"}},"filter":{"bool":{"must":[{"range":{"#timestamp":{"gte":1420099200000,"lte":1451635199999}}}],"must_not":[]}}}},"highlight":{"pre_tags":["#kibana-highlighted-field#"],"post_tags":["#\/kibana-highlighted-field#"],"fields":{"*":{}}},"aggs":{"2":{"date_histogram":{"field":"#timestamp","interval":"1w","pre_zone":"-08:00","pre_zone_adjust_large_interval":true,"min_doc_count":0,"extended_bounds":{"min":1420099200000,"max":1451635199999}}}},"fields":["*","_source"],"script_fields":{},"fielddata_fields":["log.timestamp","#timestamp","val"]}'
And if you notice there are extra single quotes like so: ' added around the value of "query". As you can see here:
"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"
What the heck is going on and how do I use these two commands with this json in a script?
It's much easier to read from a here document than to ensure you've quoted all the quotation marks properly.
url='http://username:password#url/elasticsearch/index-*/_search'
curl -s -X POST "$url" -d#- <<EOF | jq '.hits.hits[0].fields."log.timestamp"[0]'
{ "size":500,
"sort": {
...
}
EOF
You didn't quote $DATA in your command
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA
so $DATA is subject to word splitting (and more) after parameter expansion is performed. Your $DATA contains whitespace in the value of query, so it's split there and broken into multiple arguments. What you want is
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d "$DATA"
Also, my personal advice is to quote JSON string with single quotes, and use '\'' when you need a literal single quote. Quoting JSON with double quotes in shell just makes the result impossible to read...
Related
I am providing a snippet of a bash script that's called at some regular interval. I cannot seem to get the JSON that must be submitted to be properly escaped (web server says required field is missing):
url="http://myurl"
json='{\"rollingRestartStrategy\":\"\"}'
request_cmd="$(curl -u ${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY} -X POST -H 'Accept: application/json' -d $json $url)"
output=$(echo "$request_cmd")
Error message:
{"id":"04723a42-c2b4-4181-839d-a8ca5180434b","type":"error","links":{},"actions":{},"status":422,"code":"MissingRequired","message":null,"detail":null,"fieldName":"rollingRestartStrategy","baseType":"error"}
Fixing The Assignment (As Literal Data)
Double quotes are already literal inside single quotes. You don't need to escape them further with backslashes (and can't, because when inside single quotes those backslashes are also literal -- which is to say, parsed as data rather than as syntax).
# THIS IS BAD; the value assigned is: {\"rollingRestartStrategy\":\"\"} -- not valid JSON
json='{\"rollingRestartStrategy\":\"\"}'
# THIS IS GOOD: the value assigned is: {"rollingRestartStrategy":""}
json='{"rollingRestartStrategy":""}'
Fixing The Assignment (If Passing Shell Variables)
Let's say that instead of being a literal value, you actually want to use a shell variable to set the strategy. In that case, the best practice is to use jq:
strategy="whatever"
json=$(jq -cn --arg strategy "$strategy" '{ "rollingRestartStrategy": $strategy }')
This will generate well-formed JSON for every possible strategy -- even values that contain literal quotes, literal newlines, or other surprising weirdness.
Fixing The Usage
Always, always quote expansions (except in a scenario that specifically doesn't require it -- such as assignments, which are immune from expansion-time string-splitting and glob expansion even when unquoted). And note that command substitution with $() creates a new quoting context, so quotes used inside a command substitution won't interact with quotes used outside it.
output=$(curl -u "${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY}" \
-X POST \
-H 'Accept: application/json' \
-d "$json" \
"$url")
Using GET I need to pass a json value to a URL via the command line within a bash script.
This works:
curl -i "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit&conf=\{\"filename\":\"myfile.csv\"\}"
If I want to expand on the json value, I would prefer to pass a variable via the URL parameter for readability. Somethig like ... but this doesn't appear to work correctly.
generate_post_data =
{
"filename": "myfile.csv"
}
curl -i "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit&conf=${generate_post_data}"
You need to properly set the variable and you should url encode it using the --data-urlencode option.
#!/bin/bash
generate_post_data="filename=myfile.csv"
curl -G "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit" --data-urlencode $generate_post_data
From the manpage:
--data-urlencode <data>
(HTTP) This posts data, similar to the other -d, --data options with
the exception that this performs URL-encoding.
To be CGI-compliant, the part should begin with a name followed
by a separator and a content specification. The part can be
passed to curl using one of the following syntaxes:
For more info you can use man curl and then /data-urlencode to jump to the section on it.
I'm trying to POST data from a bash variable using curl, however, I'm unable to get this to work. This is the command that I'm using:
escape() { printf "%q" "$1"; }
curl -d "$(escape "$client")" -X POST -v https://$server/clients
The client variable look like this:
{"roles":["test"],"softwareName":"Some Soft","passwordSalt":"aaa","clientID":"full-client-2","contactPerson":"Test","contactPersonEmail":"a#b.org","description":"test","name":"Full Client-2","organization":"Some Org","passwordAlgorithm":"sha512","passwordHash":"bbb"}
And on the server I'm receiving the following:
{ '{"roles":': { '"test"': { '\"test\"\': '' } } }
I think its a problem with the escaping but I can't figure this out.
I've had a look at a number of other questions about this on here, but it seems most people need to insert variable into a literal that they are then trying to post. My problem is around using an entire variable as the json body. I've tried to use their answers to help me out but I haven't had any luck so far.
Don't try to quote it; use a here document:
curl -d#- -X POST -v https://"$server"/clients <<JSON
{"roles":["test"],"softwareName":"Some Soft","passwordSalt":"aaa","clientID":"full-client-2","contactPerson":"Test","contactPersonEmail":"a#b.org","description":"test","name":"Full Client-2","organization":"Some Org","passwordAlgorithm":"sha512","passwordHash":"bob"}
JSON
#- tells the -d option to look in standard input for the data, rather than using a hard-coded string.
If the text is in a variable, nothing more needs to be done; just quote the variable to prevent the shell from processing it:
curl -d "$client" -X POST -v https://"$server"/clients
I'm trying to create a script that will use the Github API to post a comment containing the output of a command. This output has multiple lines.
Here's what I'm trying to do:
curl -H "Authorization: token oauthtoken" \
-H "Content-Type: application/json" \
-X POST -d#- \
https://api.github.com/repos/company/repo/issues/14/comments <<EOF
{
"body":"$OUTPUT"
}
EOF
How can I output the variable in such a way that it respects the multiple lines contained within? Now when I run that command, all of the newlines get squished on to one line.
I don't think that the basic cause of the problem are the newlines, the issues is that the value of $text is not properly formatted json.
Follow this simple example:
test="
Hello
World
"
curl -X POST -d '{"body": "'"$test"'"}' http://server.com/...
to see new lines working.
To make it possible to send the result of arbitrary commands using json, you need to json-encode the text before.
I am using the following curl command to post the following json data.
curl -d 'json_data={"operation":"core/update","comment":"Synchronization from blah...","class":"Incident","key":{"ref":"I-000060"},"fields":{"public_log":"$(pwd)"}}' 'http://172.27.220.46/itop/webservices/rest.php?version=1.1&auth_user=admin1123&auth_pwd=xxxxx'
I am using a command substitution in the form $(pwd), however this is not recognised, and the curl posts it in the explicit form $(pwd) instead of "root".
What am i doing wrong?
It's because the whole JSON string is surrounded by the single quote ' which stops Bash for expanding anything:
~/temp> export MY_VAR=Hello
~/temp>
~/temp> echo "$MY_VAR"
Hello
~/temp> echo '$MY_VAR'
$MY_VAR
You'll have to replace the single quote with double quotes and escape the other double quotes:
curl -d "json_data={\"operation\":...
Reference:
GNU Bash Reference - Single Quotes
GNU Bash Reference - Double Quotes