I am trying to push some data to my Tiddlywiki via api so I can create a new tiddler.
I can pass almost all the data, but if the value "text" has spaces inside it breaks the curl command trying to pass multiple urls.
This is the actual command:
curl -X PUT -i "http://192.168.1.166:8080/recipes/default/tiddlers/$title" -H "X-Requested-With: TiddlyWiki" --data $(jq -nc --arg tags "$tag" --arg text "'${content}'" '{ $tags, $text }')
I tried at first using $variables, inside brackets, even inside multiple "'"$var"'" following others questions here. But most of them quickly recommended using jq so I gave it a try.
I learned how to create some keys with the bash¡s variables contents, and If I pipe all of this I can get it to work only if I replace spaces with other characters...I tried %20 or scaping the space \ with sed whitout success. (%20 is replaced literally so not helpful)
Any recommendations at all, I will follow any other path you could bring to it.
Thanks.
EDIT:
I tried using --data-urlencode "text=${var}" but it wasnt filled, only variable expanded was the title. The others didnt show at all.
API'S INFO:
https://tiddlywiki.com/static/WebServer%2520API%253A%2520Put%2520Tiddler.html
I forgot to mention that I am using zsh shell...
You need to quote the output of the command substitution: --data "$(jq ...)". Without this, curl thinks the words after the first one are individual URLs to connect to. You should also remove the single quotes from --arg text "'${content}'", otherwise they'll be added to the text itself.
Related
I wrote a script to create sqs resource on local stack. I wanted to pass a value that I get from one cli command to the next but inside an inline json. Following is the section of the script in question.
arn=$(aws --endpoint-url=http://localhost:4576 sqs get-queue-attributes \
--queue-url http://localhost:4576/my_dead_letter_queue_url \
--query 'Attributes.QueueArn' \
--output text)
aws --endpoint-url=http://localhost:4576 sqs create-queue \
--queue-name my_queue \
--attributes \
'{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
So I'm trying to pass that "arn" variable but the cli is taking that as a string and trying to find a sqs with url "$arn" and fails. I also tried removing the quote. In that case, the error is malformed string.
Instead of the arn variable, if I use the arn value as string there, it works.
Can someone please show me how to pass that variable inside that inline json if it is possible?
Thank you for reading :)
Shahed
I was able to do the following with successful results, grant it it doesn't process the json (for that I'm just replacing tokens via sed), but I updated my example and tested it at least in bash with what I was doing:
#!/bin/bash
export awscmd="aws --region us-east-1 iam"
function setArn() {
${awscmd} list-policies --query 'Policies[?PolicyName==`'${1}'`].{ARN:Arn}' --output text
}
arn=$(setArn "some-policy-name")
echo '{"RedrivePolicy":"{"deadLetterTargetArn":"'$arn'", "maxReceiveCount":"5"}"}'
$ ./somearntest.sh
{"RedrivePolicy":"{"deadLetterTargetArn":"arn:aws:iam::############:policy/some-policy-name", "maxReceiveCount":"5"}"}
Notice the use of single tics to concatenate the output result outside of the string. This is in bash 4 and I removed the escaped \"s as I think that was added in error; ymmv.
The problem here is you are trying to expand a bash variable inside single quotes. Using single quotes like this is usually to pass a bunch of strings and unqoutable stuff as one argument. If you can't replace them with double quotes you'll have to resort to dirty eval hacks, which I do not recommend.
Here is an example:
$ arn=foobar
$ echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}
$ eval echo '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"$arn\", \"maxReceiveCount\":\"5\"}"}'
{RedrivePolicy:{"deadLetterTargetArn":"foobar", "maxReceiveCount":"5"}}
For more information I suggest to check How eval works and Expansion of variables inside single quotes
So I know there is easier ways to do this, however I was given the code and asked to attempt to make it work. Rather than rewrite the entire thing I'd simply like to get this working.
So what it does is download the source code for the web page that displays when a person searches the app store. Once that is done I am attempting to pull the version of the app which comes across as the first line below
Once I get the code from the downloaded file I'd like it be placed in another file to be called for later use, however if this is an unnecessary step I am willing to remove it
I have a feeling I am missing something simple.
<span class="htlgb">4.72</span>
# connects to iTunes website with Casino/Manufacturers id
curl https://play.google.com/store/apps/details?id=${address[$a]} > json
# puts just the version from the json file into version file
grep -Po '(?<=<span class="htlgb"> ).*?(?=</span>)' json > version
# cuts out some data so we have just a version number
current_Version=`cat version | tr -d '"' | tr -d ',' | tr -d 'version:'`
Please don't use regex to parse HTML! Use a true HTML parser like Xidel instead:
echo '<span class="htlgb">4.72</span>' | xidel -s - -e '//span[#class="htlgb"]'
4.72
I wouldn't use this path expression for the playstore website however, because there are a lot of these. I've used the Spotify page as an example.
xidel -s https://play.google.com/store/apps/details?id=com.spotify.music -e '//div[#class="hAyfc"][div
="Current Version"]/outer-html()'
<div class="hAyfc"><div class="BgcNfc">Current Version</div><span class="htlgb"><div class="IQ1z0d"><s
pan class="htlgb">8.5.40.195</span></div></span></div>
The version string 8.5.40.195 can be found within a div (with an attribute class="hAyfc") that has a child div with the text Current Version.
Then it's as simple as selecting the span (text)node:
xidel -s https://play.google.com/store/apps/details?id=com.spotify.music -e '//div[#class="hAyfc"][div="Current Version"]/span'
8.5.40.195
# or with your variable:
xidel -s https://play.google.com/store/apps/details?id=${address[$a]} -e '//div[#class="hAyfc"][div="Current Version"]/span'
I am not a bash pro, but this matches 3 groups to your desired html tag. All you need to add now, it so select the value from group 2.
(<span class=\"htlgb\">)(.*?)(</span>)
You can test it here: https://regex101.com/r/9RPycf/1
I am providing a snippet of a bash script that's called at some regular interval. I cannot seem to get the JSON that must be submitted to be properly escaped (web server says required field is missing):
url="http://myurl"
json='{\"rollingRestartStrategy\":\"\"}'
request_cmd="$(curl -u ${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY} -X POST -H 'Accept: application/json' -d $json $url)"
output=$(echo "$request_cmd")
Error message:
{"id":"04723a42-c2b4-4181-839d-a8ca5180434b","type":"error","links":{},"actions":{},"status":422,"code":"MissingRequired","message":null,"detail":null,"fieldName":"rollingRestartStrategy","baseType":"error"}
Fixing The Assignment (As Literal Data)
Double quotes are already literal inside single quotes. You don't need to escape them further with backslashes (and can't, because when inside single quotes those backslashes are also literal -- which is to say, parsed as data rather than as syntax).
# THIS IS BAD; the value assigned is: {\"rollingRestartStrategy\":\"\"} -- not valid JSON
json='{\"rollingRestartStrategy\":\"\"}'
# THIS IS GOOD: the value assigned is: {"rollingRestartStrategy":""}
json='{"rollingRestartStrategy":""}'
Fixing The Assignment (If Passing Shell Variables)
Let's say that instead of being a literal value, you actually want to use a shell variable to set the strategy. In that case, the best practice is to use jq:
strategy="whatever"
json=$(jq -cn --arg strategy "$strategy" '{ "rollingRestartStrategy": $strategy }')
This will generate well-formed JSON for every possible strategy -- even values that contain literal quotes, literal newlines, or other surprising weirdness.
Fixing The Usage
Always, always quote expansions (except in a scenario that specifically doesn't require it -- such as assignments, which are immune from expansion-time string-splitting and glob expansion even when unquoted). And note that command substitution with $() creates a new quoting context, so quotes used inside a command substitution won't interact with quotes used outside it.
output=$(curl -u "${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY}" \
-X POST \
-H 'Accept: application/json' \
-d "$json" \
"$url")
Using GET I need to pass a json value to a URL via the command line within a bash script.
This works:
curl -i "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit&conf=\{\"filename\":\"myfile.csv\"\}"
If I want to expand on the json value, I would prefer to pass a variable via the URL parameter for readability. Somethig like ... but this doesn't appear to work correctly.
generate_post_data =
{
"filename": "myfile.csv"
}
curl -i "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit&conf=${generate_post_data}"
You need to properly set the variable and you should url encode it using the --data-urlencode option.
#!/bin/bash
generate_post_data="filename=myfile.csv"
curl -G "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit" --data-urlencode $generate_post_data
From the manpage:
--data-urlencode <data>
(HTTP) This posts data, similar to the other -d, --data options with
the exception that this performs URL-encoding.
To be CGI-compliant, the part should begin with a name followed
by a separator and a content specification. The part can be
passed to curl using one of the following syntaxes:
For more info you can use man curl and then /data-urlencode to jump to the section on it.
I'm trying to POST data from a bash variable using curl, however, I'm unable to get this to work. This is the command that I'm using:
escape() { printf "%q" "$1"; }
curl -d "$(escape "$client")" -X POST -v https://$server/clients
The client variable look like this:
{"roles":["test"],"softwareName":"Some Soft","passwordSalt":"aaa","clientID":"full-client-2","contactPerson":"Test","contactPersonEmail":"a#b.org","description":"test","name":"Full Client-2","organization":"Some Org","passwordAlgorithm":"sha512","passwordHash":"bbb"}
And on the server I'm receiving the following:
{ '{"roles":': { '"test"': { '\"test\"\': '' } } }
I think its a problem with the escaping but I can't figure this out.
I've had a look at a number of other questions about this on here, but it seems most people need to insert variable into a literal that they are then trying to post. My problem is around using an entire variable as the json body. I've tried to use their answers to help me out but I haven't had any luck so far.
Don't try to quote it; use a here document:
curl -d#- -X POST -v https://"$server"/clients <<JSON
{"roles":["test"],"softwareName":"Some Soft","passwordSalt":"aaa","clientID":"full-client-2","contactPerson":"Test","contactPersonEmail":"a#b.org","description":"test","name":"Full Client-2","organization":"Some Org","passwordAlgorithm":"sha512","passwordHash":"bob"}
JSON
#- tells the -d option to look in standard input for the data, rather than using a hard-coded string.
If the text is in a variable, nothing more needs to be done; just quote the variable to prevent the shell from processing it:
curl -d "$client" -X POST -v https://"$server"/clients