I am using the following curl command to post the following json data.
curl -d 'json_data={"operation":"core/update","comment":"Synchronization from blah...","class":"Incident","key":{"ref":"I-000060"},"fields":{"public_log":"$(pwd)"}}' 'http://172.27.220.46/itop/webservices/rest.php?version=1.1&auth_user=admin1123&auth_pwd=xxxxx'
I am using a command substitution in the form $(pwd), however this is not recognised, and the curl posts it in the explicit form $(pwd) instead of "root".
What am i doing wrong?
It's because the whole JSON string is surrounded by the single quote ' which stops Bash for expanding anything:
~/temp> export MY_VAR=Hello
~/temp>
~/temp> echo "$MY_VAR"
Hello
~/temp> echo '$MY_VAR'
$MY_VAR
You'll have to replace the single quote with double quotes and escape the other double quotes:
curl -d "json_data={\"operation\":...
Reference:
GNU Bash Reference - Single Quotes
GNU Bash Reference - Double Quotes
Related
I am writing a JSON template for AWS SSM document.
Once of the commands I am trying to run looks like this:
"ssh-keygen -q -t rsa -N '' <<< ""$'\n'"y" 2>&1 >/dev/null",
However, due to the "" after the <<< JSON thinks that this is a new line and is expecting a comma.
Is there a way I can tell JSON that that is a single command and need to be treated as a single line?
You need to escape the double quotes. So " inside your json string becomes \".
I wrote a script to create sqs resource on local stack. I wanted to pass a value that I get from one cli command to the next but inside an inline json. Following is the section of the script in question.
arn=$(aws --endpoint-url=http://localhost:4576 sqs get-queue-attributes \
--queue-url http://localhost:4576/my_dead_letter_queue_url \
--query 'Attributes.QueueArn' \
--output text)
aws --endpoint-url=http://localhost:4576 sqs create-queue \
--queue-name my_queue \
--attributes \
'{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
So I'm trying to pass that "arn" variable but the cli is taking that as a string and trying to find a sqs with url "$arn" and fails. I also tried removing the quote. In that case, the error is malformed string.
Instead of the arn variable, if I use the arn value as string there, it works.
Can someone please show me how to pass that variable inside that inline json if it is possible?
Thank you for reading :)
Shahed
I was able to do the following with successful results, grant it it doesn't process the json (for that I'm just replacing tokens via sed), but I updated my example and tested it at least in bash with what I was doing:
#!/bin/bash
export awscmd="aws --region us-east-1 iam"
function setArn() {
${awscmd} list-policies --query 'Policies[?PolicyName==`'${1}'`].{ARN:Arn}' --output text
}
arn=$(setArn "some-policy-name")
echo '{"RedrivePolicy":"{"deadLetterTargetArn":"'$arn'", "maxReceiveCount":"5"}"}'
$ ./somearntest.sh
{"RedrivePolicy":"{"deadLetterTargetArn":"arn:aws:iam::############:policy/some-policy-name", "maxReceiveCount":"5"}"}
Notice the use of single tics to concatenate the output result outside of the string. This is in bash 4 and I removed the escaped \"s as I think that was added in error; ymmv.
The problem here is you are trying to expand a bash variable inside single quotes. Using single quotes like this is usually to pass a bunch of strings and unqoutable stuff as one argument. If you can't replace them with double quotes you'll have to resort to dirty eval hacks, which I do not recommend.
Here is an example:
$ arn=foobar
$ echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}
$ eval echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{RedrivePolicy:{"deadLetterTargetArn":"foobar", "maxReceiveCount":"5"}}
For more information I suggest to check How eval works and Expansion of variables inside single quotes
I am providing a snippet of a bash script that's called at some regular interval. I cannot seem to get the JSON that must be submitted to be properly escaped (web server says required field is missing):
url="http://myurl"
json='{\"rollingRestartStrategy\":\"\"}'
request_cmd="$(curl -u ${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY} -X POST -H 'Accept: application/json' -d $json $url)"
output=$(echo "$request_cmd")
Error message:
{"id":"04723a42-c2b4-4181-839d-a8ca5180434b","type":"error","links":{},"actions":{},"status":422,"code":"MissingRequired","message":null,"detail":null,"fieldName":"rollingRestartStrategy","baseType":"error"}
Fixing The Assignment (As Literal Data)
Double quotes are already literal inside single quotes. You don't need to escape them further with backslashes (and can't, because when inside single quotes those backslashes are also literal -- which is to say, parsed as data rather than as syntax).
# THIS IS BAD; the value assigned is: {\"rollingRestartStrategy\":\"\"} -- not valid JSON
json='{\"rollingRestartStrategy\":\"\"}'
# THIS IS GOOD: the value assigned is: {"rollingRestartStrategy":""}
json='{"rollingRestartStrategy":""}'
Fixing The Assignment (If Passing Shell Variables)
Let's say that instead of being a literal value, you actually want to use a shell variable to set the strategy. In that case, the best practice is to use jq:
strategy="whatever"
json=$(jq -cn --arg strategy "$strategy" '{ "rollingRestartStrategy": $strategy }')
This will generate well-formed JSON for every possible strategy -- even values that contain literal quotes, literal newlines, or other surprising weirdness.
Fixing The Usage
Always, always quote expansions (except in a scenario that specifically doesn't require it -- such as assignments, which are immune from expansion-time string-splitting and glob expansion even when unquoted). And note that command substitution with $() creates a new quoting context, so quotes used inside a command substitution won't interact with quotes used outside it.
output=$(curl -u "${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY}" \
-X POST \
-H 'Accept: application/json' \
-d "$json" \
"$url")
I'm trying to make a GET request with curl in a bash file.
So I call curl, and get a JSON response from my server, and I try to get the object line which is part of the response, and I do so with jq, so I can print the value.
But its not working.
My response is something like this:
{"line":"Azul", "id":"j3453j45n35", "lat":"39.091937", "long":"-9.265441"}
My bash script:
#!/bin/bash
while :
do
cmd=$(curl -i -H "Content-Type: application/json" http://localhost:8080/cards/exists/1 | jq ‘.”line”’)
sleep 2
done
The result:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at , line 1:
‘.”line”’
jq: 1 compile error
(23) Failed writing body
Whats going wrong?
I followed plenty of responses people advised online, but I keep getting the same response.
My json object seems to be correct.
I'm running this file on MacOS Sierra
There are two problems with your code:
You're using non-ASCII quotes (‘ ” ” ’), which jq doesn't understand:
Replace them with their ASCII equivalents: ' " " '
On macOS, avoid a text editor such as the standard TextEdit.app that has the "smart quotes" feature turned on, or turn that feature off via the menu system: Edit > Substitutions > Smart Quotes
It is this feature that replaces ASCII quoting chars. with their non-ASCII, typographical equivalents, as you type.
Generally, consider using a dedicated source-code editor for editing your code, such as Visual Studio Code (free) or Atom (free) or Sublime Text (paid).
Your curl command includes header output (-i), which jq chokes on - simply omit the -i:
#!/bin/bash
while :; do
cmd=$(curl -H 'Content-Type: application/json' 'http://localhost:8080/cards/exists/1' |
jq '.line')
sleep 2
done
While '."line"' would have worked too, there's no need for the extra layer of quoting for simple property names like that.
Also, it's better to enclose strings that you want to be interpreted literally in '...' in the shell.
The two commands below work perfectly outside of a script if I execute them one after the other, however, if I put them inside a bash script. They fail.
DATA="{\"size\":500,\"sort\":{\"#timestamp\":\"desc\"},\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"tx.traceId: AIP1447283489-6 AND event: published_notification AND attempt: 1\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"#timestamp\":{\"gte\":1420099200000,\"lte\":1451635199999}}}],\"must_not\":[]}}}},\"highlight\":{\"pre_tags\":[\"#kibana-highlighted-field#\"],\"post_tags\":[\"#\/kibana-highlighted-field#\"],\"fields\":{\"*\":{}}},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"#timestamp\",\"interval\":\"1w\",\"pre_zone\":\"-08:00\",\"pre_zone_adjust_large_interval\":true,\"min_doc_count\":0,\"extended_bounds\":{\"min\":1420099200000,\"max\":1451635199999}}}},\"fields\":[\"*\",\"_source\"],\"script_fields\":{},\"fielddata_fields\":[\"log.timestamp\",\"#timestamp\",\"val\"]}"
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA | jq '.hits.hits[0].fields."log.timestamp"[0]'
By using set -x I've found that the final curl command that is executed is:
curl -s -XPOST 'http://username:password#url/elasticsearch/index-*/_search' -d '{"size":500,"sort":{"#timestamp":"desc"},"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"}},"filter":{"bool":{"must":[{"range":{"#timestamp":{"gte":1420099200000,"lte":1451635199999}}}],"must_not":[]}}}},"highlight":{"pre_tags":["#kibana-highlighted-field#"],"post_tags":["#\/kibana-highlighted-field#"],"fields":{"*":{}}},"aggs":{"2":{"date_histogram":{"field":"#timestamp","interval":"1w","pre_zone":"-08:00","pre_zone_adjust_large_interval":true,"min_doc_count":0,"extended_bounds":{"min":1420099200000,"max":1451635199999}}}},"fields":["*","_source"],"script_fields":{},"fielddata_fields":["log.timestamp","#timestamp","val"]}'
And if you notice there are extra single quotes like so: ' added around the value of "query". As you can see here:
"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"
What the heck is going on and how do I use these two commands with this json in a script?
It's much easier to read from a here document than to ensure you've quoted all the quotation marks properly.
url='http://username:password#url/elasticsearch/index-*/_search'
curl -s -X POST "$url" -d#- <<EOF | jq '.hits.hits[0].fields."log.timestamp"[0]'
{ "size":500,
"sort": {
...
}
EOF
You didn't quote $DATA in your command
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA
so $DATA is subject to word splitting (and more) after parameter expansion is performed. Your $DATA contains whitespace in the value of query, so it's split there and broken into multiple arguments. What you want is
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d "$DATA"
Also, my personal advice is to quote JSON string with single quotes, and use '\'' when you need a literal single quote. Quoting JSON with double quotes in shell just makes the result impossible to read...