I am providing a snippet of a bash script that's called at some regular interval. I cannot seem to get the JSON that must be submitted to be properly escaped (web server says required field is missing):
url="http://myurl"
json='{\"rollingRestartStrategy\":\"\"}'
request_cmd="$(curl -u ${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY} -X POST -H 'Accept: application/json' -d $json $url)"
output=$(echo "$request_cmd")
Error message:
{"id":"04723a42-c2b4-4181-839d-a8ca5180434b","type":"error","links":{},"actions":{},"status":422,"code":"MissingRequired","message":null,"detail":null,"fieldName":"rollingRestartStrategy","baseType":"error"}
Fixing The Assignment (As Literal Data)
Double quotes are already literal inside single quotes. You don't need to escape them further with backslashes (and can't, because when inside single quotes those backslashes are also literal -- which is to say, parsed as data rather than as syntax).
# THIS IS BAD; the value assigned is: {\"rollingRestartStrategy\":\"\"} -- not valid JSON
json='{\"rollingRestartStrategy\":\"\"}'
# THIS IS GOOD: the value assigned is: {"rollingRestartStrategy":""}
json='{"rollingRestartStrategy":""}'
Fixing The Assignment (If Passing Shell Variables)
Let's say that instead of being a literal value, you actually want to use a shell variable to set the strategy. In that case, the best practice is to use jq:
strategy="whatever"
json=$(jq -cn --arg strategy "$strategy" '{ "rollingRestartStrategy": $strategy }')
This will generate well-formed JSON for every possible strategy -- even values that contain literal quotes, literal newlines, or other surprising weirdness.
Fixing The Usage
Always, always quote expansions (except in a scenario that specifically doesn't require it -- such as assignments, which are immune from expansion-time string-splitting and glob expansion even when unquoted). And note that command substitution with $() creates a new quoting context, so quotes used inside a command substitution won't interact with quotes used outside it.
output=$(curl -u "${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY}" \
-X POST \
-H 'Accept: application/json' \
-d "$json" \
"$url")
Related
I am trying to push some data to my Tiddlywiki via api so I can create a new tiddler.
I can pass almost all the data, but if the value "text" has spaces inside it breaks the curl command trying to pass multiple urls.
This is the actual command:
curl -X PUT -i "http://192.168.1.166:8080/recipes/default/tiddlers/$title" -H "X-Requested-With: TiddlyWiki" --data $(jq -nc --arg tags "$tag" --arg text "'${content}'" '{ $tags, $text }')
I tried at first using $variables, inside brackets, even inside multiple "'"$var"'" following others questions here. But most of them quickly recommended using jq so I gave it a try.
I learned how to create some keys with the bash¡s variables contents, and If I pipe all of this I can get it to work only if I replace spaces with other characters...I tried %20 or scaping the space \ with sed whitout success. (%20 is replaced literally so not helpful)
Any recommendations at all, I will follow any other path you could bring to it.
Thanks.
EDIT:
I tried using --data-urlencode "text=${var}" but it wasnt filled, only variable expanded was the title. The others didnt show at all.
API'S INFO:
https://tiddlywiki.com/static/WebServer%2520API%253A%2520Put%2520Tiddler.html
I forgot to mention that I am using zsh shell...
You need to quote the output of the command substitution: --data "$(jq ...)". Without this, curl thinks the words after the first one are individual URLs to connect to. You should also remove the single quotes from --arg text "'${content}'", otherwise they'll be added to the text itself.
I'm working on an API to create a tenant in QRadar. I have this basic script that I'm using and I've ran into essentially the same problem before but now it got a little more complex.
Here is the code I'm using:
$QRadarIP = 'xxx.xxx.xxx.xxx'
$authtoken = 'blahblahblah'
$SECHeader = #(
"SEC: $authtoken"
)
$name = 'test4'
$description = 'The fourth test'
$TenantInput = #('{"deleted": false,"description": $description,"event_rate_limit": 0,"flow_rate_limit": 0,"name": $name}')
curl.exe -S -X POST -k -H $SECHeader -H 'Content-Type: application/json' -H 'Version: 15.1' -H 'Accept: application/json' --data-binary $TenantInput $tenantURI
So the problem I'm running into is that $name and $description are getting passed as literals instead of the value from the variable. The string within the $TenantInput is JSON and those quotes are apparently necessary for the way that the JSON is interpreted. I'm trying to put that variable in the #() because that's what I used to fix my issue of passing the auth token into the SEC header and then into the curl command.
I've looked at other questions on here that were similar but there's always something about the scenario that is different than mine to where I can't decipher it. Any advice is appreciated.
The solution is:
$TenantInput = #("{`"deleted`": false,`"description`": ${description},`"event_rate_limit`": 0,`"flow_rate_limit`": 0,`"name`": ${name}}")
You need to use double quotes and define variables in ${}
To understand the quotes in String you can read this blog (with examples): https://www.computerperformance.co.uk/powershell/quotes/
So #Damien was right, the final-ish solution came down to me adding those backticks and quotes in front of the variable but also replacing my spaces with underscores soo like below:
$name6 = "test5"
$description6 = "The_fifth_test"
$TenantInput = #("{`"deleted`": false,`"description`": `"${description6}`",`"event_rate_limit`": 0,`"flow_rate_limit`": 0,`"name`": `"${name6}`"}")
it's definitely a bummer that I have to use the underscores in the string so I'm sure there's still a better answer than what I eventually stopped with.
I wrote a script to create sqs resource on local stack. I wanted to pass a value that I get from one cli command to the next but inside an inline json. Following is the section of the script in question.
arn=$(aws --endpoint-url=http://localhost:4576 sqs get-queue-attributes \
--queue-url http://localhost:4576/my_dead_letter_queue_url \
--query 'Attributes.QueueArn' \
--output text)
aws --endpoint-url=http://localhost:4576 sqs create-queue \
--queue-name my_queue \
--attributes \
'{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
So I'm trying to pass that "arn" variable but the cli is taking that as a string and trying to find a sqs with url "$arn" and fails. I also tried removing the quote. In that case, the error is malformed string.
Instead of the arn variable, if I use the arn value as string there, it works.
Can someone please show me how to pass that variable inside that inline json if it is possible?
Thank you for reading :)
Shahed
I was able to do the following with successful results, grant it it doesn't process the json (for that I'm just replacing tokens via sed), but I updated my example and tested it at least in bash with what I was doing:
#!/bin/bash
export awscmd="aws --region us-east-1 iam"
function setArn() {
${awscmd} list-policies --query 'Policies[?PolicyName==`'${1}'`].{ARN:Arn}' --output text
}
arn=$(setArn "some-policy-name")
echo '{"RedrivePolicy":"{"deadLetterTargetArn":"'$arn'", "maxReceiveCount":"5"}"}'
$ ./somearntest.sh
{"RedrivePolicy":"{"deadLetterTargetArn":"arn:aws:iam::############:policy/some-policy-name", "maxReceiveCount":"5"}"}
Notice the use of single tics to concatenate the output result outside of the string. This is in bash 4 and I removed the escaped \"s as I think that was added in error; ymmv.
The problem here is you are trying to expand a bash variable inside single quotes. Using single quotes like this is usually to pass a bunch of strings and unqoutable stuff as one argument. If you can't replace them with double quotes you'll have to resort to dirty eval hacks, which I do not recommend.
Here is an example:
$ arn=foobar
$ echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}
$ eval echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{RedrivePolicy:{"deadLetterTargetArn":"foobar", "maxReceiveCount":"5"}}
For more information I suggest to check How eval works and Expansion of variables inside single quotes
The two commands below work perfectly outside of a script if I execute them one after the other, however, if I put them inside a bash script. They fail.
DATA="{\"size\":500,\"sort\":{\"#timestamp\":\"desc\"},\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"tx.traceId: AIP1447283489-6 AND event: published_notification AND attempt: 1\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"#timestamp\":{\"gte\":1420099200000,\"lte\":1451635199999}}}],\"must_not\":[]}}}},\"highlight\":{\"pre_tags\":[\"#kibana-highlighted-field#\"],\"post_tags\":[\"#\/kibana-highlighted-field#\"],\"fields\":{\"*\":{}}},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"#timestamp\",\"interval\":\"1w\",\"pre_zone\":\"-08:00\",\"pre_zone_adjust_large_interval\":true,\"min_doc_count\":0,\"extended_bounds\":{\"min\":1420099200000,\"max\":1451635199999}}}},\"fields\":[\"*\",\"_source\"],\"script_fields\":{},\"fielddata_fields\":[\"log.timestamp\",\"#timestamp\",\"val\"]}"
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA | jq '.hits.hits[0].fields."log.timestamp"[0]'
By using set -x I've found that the final curl command that is executed is:
curl -s -XPOST 'http://username:password#url/elasticsearch/index-*/_search' -d '{"size":500,"sort":{"#timestamp":"desc"},"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"}},"filter":{"bool":{"must":[{"range":{"#timestamp":{"gte":1420099200000,"lte":1451635199999}}}],"must_not":[]}}}},"highlight":{"pre_tags":["#kibana-highlighted-field#"],"post_tags":["#\/kibana-highlighted-field#"],"fields":{"*":{}}},"aggs":{"2":{"date_histogram":{"field":"#timestamp","interval":"1w","pre_zone":"-08:00","pre_zone_adjust_large_interval":true,"min_doc_count":0,"extended_bounds":{"min":1420099200000,"max":1451635199999}}}},"fields":["*","_source"],"script_fields":{},"fielddata_fields":["log.timestamp","#timestamp","val"]}'
And if you notice there are extra single quotes like so: ' added around the value of "query". As you can see here:
"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"
What the heck is going on and how do I use these two commands with this json in a script?
It's much easier to read from a here document than to ensure you've quoted all the quotation marks properly.
url='http://username:password#url/elasticsearch/index-*/_search'
curl -s -X POST "$url" -d#- <<EOF | jq '.hits.hits[0].fields."log.timestamp"[0]'
{ "size":500,
"sort": {
...
}
EOF
You didn't quote $DATA in your command
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA
so $DATA is subject to word splitting (and more) after parameter expansion is performed. Your $DATA contains whitespace in the value of query, so it's split there and broken into multiple arguments. What you want is
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d "$DATA"
Also, my personal advice is to quote JSON string with single quotes, and use '\'' when you need a literal single quote. Quoting JSON with double quotes in shell just makes the result impossible to read...
I am using the following curl command to post the following json data.
curl -d 'json_data={"operation":"core/update","comment":"Synchronization from blah...","class":"Incident","key":{"ref":"I-000060"},"fields":{"public_log":"$(pwd)"}}' 'http://172.27.220.46/itop/webservices/rest.php?version=1.1&auth_user=admin1123&auth_pwd=xxxxx'
I am using a command substitution in the form $(pwd), however this is not recognised, and the curl posts it in the explicit form $(pwd) instead of "root".
What am i doing wrong?
It's because the whole JSON string is surrounded by the single quote ' which stops Bash for expanding anything:
~/temp> export MY_VAR=Hello
~/temp>
~/temp> echo "$MY_VAR"
Hello
~/temp> echo '$MY_VAR'
$MY_VAR
You'll have to replace the single quote with double quotes and escape the other double quotes:
curl -d "json_data={\"operation\":...
Reference:
GNU Bash Reference - Single Quotes
GNU Bash Reference - Double Quotes