Curl - Encode file as url query
I'm trying to send message to Amazon SQS queue with payload read from file using curl
So far I've got working curl to send message to queue but with harcoded value like:
curl --request POST \
--url 'http://localhost:9324/queue/price-import?Action=SendMessage&MessageBody=%7B%0A%09%22id%22%3A%20%22f23399af-a384-4ed3-bac0-386a01804d83%22%2C%0A%09%22de%22%3A%20%7B%0A%09%09%22price%22%3A%2010.59%0A%09%7D%2C%0A%09%22nl%22%3A%20%7B%0A%09%09%22price%22%3A%20107.99%0A%09%7D%2C%0A%09%22pl%22%3A%20%7B%0A%09%09%22price%22%3A%2012.15%0A%09%7D%0A%7D'
And payload I send is:
{
"id": "f23399af-a384-4ed3-bac0-386a01804d83",
"de": {
"price": 10.59
},
"nl": {
"price": 107.99
},
"pl": {
"price": 12.15
}
}
How can I modify this curl to omit harcoded value and instead put path to file with .json extension?
Desired output
curl --request POST \
--data-urlencode http://localhost:9324/queue/price-import?Action=SendMessage&MessageBody=#input.json
What I already tried
curl --request POST -data-urlencode http://localhost:9324/queue/price-import?Action=SendMessage&MessageBody=#input.json
curl --request POST -data-urlencode input.json http://localhost:9324/queue/price-import
curl --request POST -data-urlencode "sendbody#input.json" http://localhost:9324/queue/price-import
curl --request POST --data-urlencode SendBody#input.json --url http://localhost:9324/queue/price-import?Action=SendMessage
Resources I found
Unix cURL POST to specific variable using contents from a file
POSTing a file's contents with CURL
curl.1 the man page
Answering to my own question based on this answer:
How to urlencode data for curl command?
This worked for me:
curl \
--data-urlencode MessageBody#input.json \
--data-urlencode "Action=SendMessage" \
http://localhost:9324/queue/price-import
and my input.json file:
{
"id": "f23399af-a384-4ed3-bac0-386a01804d83",
"de": {
"price": 10.59
},
"nl": {
"price": 107.99
},
"pl": {
"price": 12.15
}
}
Related
So I'm writing a script that needs to create a json object, and post it with curl.
This is working:
curl --header "Content-Type: application/json" --request POST --data '{ "_type": "_test", "_device": "123.123.123.123", "_system": "web-services", "result": "success", "_time": "123", "error": "" }' $data_pipeline
$data_pipeline contains the URL for the post request
if $json_string contains the string including single quotes:
'{ "_type": "_test", "_device": "123.123.123.123", "_system": "web-services", "result": "success", "_time": "123", "error": "" }'
This should work but it doesn't:
curl --header "Content-Type: application/json" --request POST --data $json_string $data_pipeline
First I was creating the $json_object without the single quotes, and tried to add them on the command line for CURL. If I don't escape the single quotes, $json_string is sent as a literal, instead of expanding the variable. I escaped the single quotes, and it did not help, I even tried a double escape in case that was needed, and still it is not working. It only works if I put the entire json string by hand, but not if I put it in a variable. How can I fix this? The json is dynamically created by the script, using jq, and the json is valid as I can successfully run the post by hand, I just need it to work with a variable. It won't work without the single quotes, but the single quotes don't work when I use a variable to hold the json, it doesn't matter if I put the single quotes in the variable, or try to do it outside of the variable... How do I fix this?
The json object is built with this code:
json_string=\'$( jq -n \
--arg _type "$_type" \
--arg _device "$_device" \
--arg _system "$_system" \
--arg result "$result" \
--arg _time "$_time" \
--arg error "$error" \
'{_type: $_type, _device: $_device, _system: $_system, result: $result, _time: $_time, error: $error}' )\'
Originally I was creating the json_string without the ' but I added that in attempt to get the single quotes wrapped around the json.
Thanks!
#!/bin/bash
_type="hello"
# let's have some fun
_device="single'quote"
_system='double"quote'
error='multi
line'
encode()
{
printf "%s" "$1" | sed 's/"/%22/' | tr '\n' ' '
}
# you don't need encode() if your strings are not json-offensive
json_string="{
_type: \"$_type\"
_device: \"$_device\"
_system: \"$(encode "$_system")\"
error: \"$(encode "$error")\"
}"
set -x
curl \
-X POST \
-d "$json_string" \
-H 'content-type: application/json' \
http://httpbin.org/anything
Output:
+ curl -X POST -d '{
_type: "hello"
_device: "single'\''quote"
_system: "double%22quote"
error: "multi line"
}' -H 'content-type: application/json' http://httpbin.org/anything
{
"args": {},
"data": "{\n\t_type: \"hello\"\n\t_device: \"single'quote\"\n\t_system: \"double%22quote\"\n\terror: \"multi line\"\n}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "92",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.58.0",
},
"json": null,
"method": "POST",
"url": "http://httpbin.org/anything"
}
When running this list files there is not a nextPageToken there is just files[].
curl \
'https://www.googleapis.com/drive/v3/files?fields=files(id%2C%20name)&key=[YOUR_API_KEY]' \
--header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept: application/json' \
--compressed
result:
{
"files": [
{
"id": "1",
"name": "1"
},
...
{
"id": "2",
"name": "2"
}
]
}
Leaving the fields parameter empty, the nextPageToken is returned.
curl \
'https://www.googleapis.com/drive/v3/files?key=[YOUR_API_KEY]' \
--header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept: application/json' \
--compressed
result:
{
"kind": "drive#fileList",
"nextPageToken": "~!!~AI9FV7TN...",
"incompleteSearch": false,
"files": [
{
"kind": "drive#file",
"id": "1",
"name": "1",
"mimeType": "application/vnd.google-apps.spreadsheet"
},
...
Is this a bug or is there some way to get the nextPageToken and limit the fields returned? The doc page for fields implies that it should work:
Note: The Drive API supports query parameters for data pagination (maxResults and nextPageToken). For APIs that support these parameters, use these parameters to reduce the results of each query to a manageable size. Otherwise, the performance gains possible with partial response might not be realized.
When fields=files(id,name) is used for the method of "Files: list", the file ID and filename are returned. In this case, the values of fields are files.id and files.name. nextPageToken is not included in fields. By this, the page token is not returned. When fields is not used, it seems that fields of nextPageToken,incompleteSearch,kind,files(id,name,kind,mimeType) is the default value. So I thought that this is not a bug, and it might be the current specification.
So when you want to retrieve nextPageToken when you use https://www.googleapis.com/drive/v3/files?fields=files(id%2C%20name), please include nextPageToken in fields as follows.
Modified curl command:
curl \
'https://www.googleapis.com/drive/v3/files?fields=nextPageToken%2Cfiles%28id%2Cname%29&key=[YOUR_API_KEY]' \
--header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept: application/json' \
--compressed
In this case, fields is nextPageToken,files(id,name).
Result:
When above curl command is run, the following result is returned.
{
"nextPageToken": "###",
"files": [
{"id": "###", "name": "###"},
{"id": "###", "name": "###"},
,
,
,
]
}
Reference:
Files: list
Hello I want to use the API approach to show the jenkins build status with curl as stated here: https://developer.atlassian.com/server/bitbucket/how-tos/updating-build-status-for-commits/
Assembled that:
curl -u username:'password' -H "Content-Type: application/json" -X POST https://tools.my.repo.com/bitbucket/rest/build-status/1.0/commits/620d293ed88412e45c58aae537b8765be2e8d148 -d #build0.json`
Where my build json contains:
{
"state": "SUCCESSFUL",
"key": "8",
"name": "feature/branch",
"url": "https://my.jenkins.placeholder/job/Plugintest/",
"description": "Changes by Umbert Eco"
}
I need to have that in one line...Is it possible?
Yes!! Try framing the curl in this way
curl -u username:password -H "Content-Type: application/json" -X POST https://tools.my.repo.com/bitbucket/rest/build-status/1.0/commits/620d293ed88412e45c58aae537b8765be2e8d148 -d '{\"state\": \"SUCCESSFUL\",\"key\": \"8\",\"name\": \"feature/branch\",\"url\": \"https://my.jenkins.placeholder/job/Plugintest\",\"description\": \"Changes by Umbert Eco\"}'
This question already has answers here:
Curl command issue in bash script because of string variable containing quotes [duplicate]
(2 answers)
Closed 3 years ago.
I am trying to insert a variable into my json payload (working in shell script) but I am unsure of how to escape the characters properly
I've tried many different escape methods but I'm a total noob at it and I either return the literal string or it doesn't run
SLACK_ALERT_WEBHOOK=desiredurl
curl -X POST -H 'Content-type: application/json' --data '{"text": "*Daily Webhook Verification*", "attachments": [
{
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Slack post failed for webhook, please investigate: $SLACK_ALERT_WEBHOOK"
}
}
]
}
]}' "$SLACK_ALERT_WEBHOOK"
I just want to insert the value of SLACK_ALERT_WEBHOOK into this portion of the code "text": "Slack post failed for webhook, please investigate: $SLACK_ALERT_WEBHOOK but it either doesn't run or returns the literal string. I have the bottom "$SLACK_ALERT_WEBHOOK" working successfully at the bottom to send to my desired slack channel so I'm not worried about that.
I've got it working thanks to tripleee:
curl -X POST -H 'Content-type: application/json' --data "{\"text\": \"*Verification*\", \"attachments\": [{\"blocks\": [{\"type\": \"section\",\"text\": {\"type\": \"mrkdwn\",\"text\": \"$SLACK_ALERT_WEBHOOK\"}}]}]}" $SLACK_ALERT_WEBHOOK
Try this:
SLACK_ALERT_WEBHOOK=desiredurl
$(curl -X POST -H 'Content-type: application/json' --data '{"text": "*Daily Webhook Verification*", "attachments": [
{
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Slack post failed for webhook, please investigate: `echo $SLACK_ALERT_WEBHOOK`"
}
}
]
}
]}' "`echo $SLACK_ALERT_WEBHOOK`")
Using the echo <<variable>> it should just run that as a command. I tested it on my own server, but might work differently on yours. Hope it helps.
I've been trying to use the Post references (https://developer.autodesk.com/en/docs/model-derivative/v2/reference/http/urn-references-POST/) to set up the reference between two files in forge but although I get a message "success" as result when I try it on the forge viewer I still see the files separately even after I translate the models. Has someone been through the same issue?
Without seeing you code it is hard to tell what is happening. Below I copied my bash script code which references/translate an obj with material and texture.
Au.obj
+- Au.mtl
+- Au.jpg
After upload, I got these
idObj="urn:adsk.objects:os.object:cyrillejcrja/Au.obj"
idMtl="urn:adsk.objects:os.object:cyrillejcrja/Au.mtl"
idJpg="urn:adsk.objects:os.object:cyrillejcrja/Au.jpg"
the code to set references, now
urn=$(xbase64encode $idObj)
job='{
"urn": "'${idObj}'",
"filename": "Au.obj",
"references": [{
"urn": "'${idMtl}'",
"relativePath": "./Au.mtl",
"filename": "Au.mtl",
"references": [{
"urn": "'${idJpg}'",
"relativePath": "./Au.jpg"
}]
}]
}'
response=$(curl -H "Content-Type: application/json" \
-H "Authorization: ${bearer}" \
-X POST ${ForgeHost}/modelderivative/v2/designdata/${urn}/references \
-k -s -d "${job}")
Here is got a reply like below which only means that the references are registered.
{
"result": "success"
}
Now, I do this to translate the obj and use the references
urn=$(xbase64encode $idObj)
job='{
"input": {
"urn": "'${urn}'",
"checkReferences": true
},
"output": {
"formats": [
{
"type": "svf",
"views": [
"2d",
"3d"
]
}
]
}
}'
response=$(curl -H "Content-Type: application/json" \
-H "Authorization: ${bearer}" \
-H "x-ads-force: true" \
-X POST ${ForgeHost}/modelderivative/v2/designdata/job \
-k -s -d "${job}")
Note the "checkReferences": true, as documented here.
Now, I can wait the translation to complete and see the result in the Viewer.
For reference the xbase64safeencode function used above
function xbase64safeencode () { local id64=$(echo -ne $1 | base64 $wrap_arg | tr -d '=' | tr '+/' '-_'); echo $id64; }
#Cyrille, this is my request:
curl -X 'POST' -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsI' -H 'Content-Type: application/json' -v 'https://developer.api.autodesk.com/modelderivative/v2/designdata/{urn}/references' -d
'{
"urn": "urn:adsk.objects:os.object:bucket/non-existent.rvt",
"filename": "",
"references": [
{
"urn": "urn:adsk.objects:os.object:bucket/non-existent.rvt",
"relativePath": "",
"filename": ""
}
]
}'
I got as result:
'{
"result": "success"
}'
The point is I'm getting success as a result even when I do not have the specified file on the server, so I'd suggest few server-side validations, for example, when a model has been translated once we can't set as reference right so it should at least return an error. Thank you and I hope this helps.