execute in bash command stored in variable [duplicate] - json

This question already has answers here:
Bash script: Use string variable in curl JSON Post data
(3 answers)
Closed 2 years ago.
I'm trying to script a mailing using a curl api (this is the base API, in mine the html part is changed with "xmessage":
curl -s \
-X POST \
--user "$MJ_APIKEY_PUBLIC:$MJ_APIKEY_PRIVATE" \
https://api.mailjet.com/v3.1/send \
-H 'Content-Type: application/json' \
-d '{
"Messages":[
{
"From": {
"Email": "pilot#mailjet.com",
"Name": "Mailjet Pilot"
},
"To": [
{
"Email": "passenger1#mailjet.com",
"Name": "passenger 1"
}
],
"Subject": "Your email flight plan!",
"TextPart": "Dear passenger 1, welcome to Mailjet! May the delivery force be with you!",
"HTMLPart": "<h3>Dear passenger 1, welcome to Mailjet!</h3><br />May the delivery force be with you!",
"CustomCampaign": "SendAPI_campaign",
"DeduplicateCampaign": true
}
]
}'
My script look like this :
...
message=$(cat ./message.txt)
message=${message//"xdate"/$courseDate}
message=${message//"xcoursecode"/$courseCode}
message=${message//"xsubtitle"/$subtitle}
message=${message//"\r"/""}
message=${message//"\r\n"/""}
message=${message//"\n"/""}
message=${message//"\""/"\\\""}
message=${message//"'"/"&apos;"}
mailJet=$(cat ./mailjet.txt) # containing my API as described as above
mailJet=${mailJet//"xmessage"/$message}
echo $mailJet
eval $mailJet
the command "eval $mailJet" does not works but if I do a copy paste in the terminal of the "echo $mailJet" output my command works
The eval $mailJet give the following error :
{"ErrorIdentifier":"5cce36c5-373c-48ca-90b8-2b6bfc5df526","ErrorCode":"mj-0031","StatusCode":400,"ErrorMessage":"Request payload contains not valid UTF-8 encoded characters"}
Something that partially worked, was to put directly the mailJet.txt content in the script
but I'm struggling to find the syntax to replace the xmessage by what's in $message.
Like this it did not worked :
...
message=$(cat ./message.txt)
message=${message//"xdate"/$courseDate}
message=${message//"xcoursecode"/$courseCode}
message=${message//"xsubtitle"/$subtitle}
message=${message//"\r"/""}
message=${message//"\r\n"/""}
message=${message//"\n"/""}
message=${message//"\""/"\\\""}
message=${message//"'"/"&apos;"}
curl -s \
-X POST \
--user "$MJ_APIKEY_PUBLIC:$MJ_APIKEY_PRIVATE" \
https://api.mailjet.com/v3.1/send \
-H 'Content-Type: application/json' \
-d '{
"Messages":[
{
"From": {
"Email": "pilot#mailjet.com",
"Name": "Mailjet Pilot"
},
"To": [
{
"Email": "passenger1#mailjet.com",
"Name": "passenger 1"
}
],
"Subject": "Your email flight plan!",
"TextPart": "Dear passenger 1, welcome to Mailjet! May the delivery force be with you!",
"HTMLPart": "$message", ## neither like this : "HTMLPart": "'$message'",
"CustomCampaign": "SendAPI_campaign",
"DeduplicateCampaign": true
}
]
}'
Whereas if I put any html stuff instead of $message in the curl api, the script run without any issue.
I'm stuck (and not a great bash coder or even coder at all)
Many thanks by advance for your help.

I think your problems start at
-d '{
This use of a single quote means that nothing in this section is interpreted by bash. "$message" later on will be treated as the text $ and message.
If this is the issue, then what you need to do is unquote around the variable names, but also double-quote arround them, so you write '"$message"' Or, if you need the double-quotes to appear inside the curl command "'"$message"'".
Note you can't have the ## comment either, but I assume you put that in for our benefit.

Related

Trying to include a dynamic variable within JSON data of a Bash / Shell POST request to API endpoint

I am trying to submit a POST request using Bash that includes JSON data with a variable equal to that of a random string. I get the string dynamically by parsing a file which can vary in content... and may contain regex characters and also new lines.
I will provide an example string here with a curl request that works successfully with the API I am posting this request to. I can only get the request to go through with the string hardcoded into the JSON data and not while assigning a variable to the string like for instance stringVar and using the variable in the JSON data. I could sure use some help where I am messing this up
my working shell script looks something like this
#!/bin/bash
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' -d '{
"name": "my-project",
"files": [
{
"data": "helloWorldFunction() {\n echo \"hello world\" \n}"
},
],
}'
This works, however I need to change data's value from the string
helloWorldFunction() {\n echo "hello world" \n}
to a variable
I have tried settings the variable in different ways in the JSON content from reading suggestions on other questions. For instance I have tried tried changing my shell script to
#!/bin/bash
stringVar="helloWorldFunction() {\n echo \"hello world\" \n}"
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' -d '{
"name": "my-project",
"files": [
{
"data": "${stringVar}"
},
],
}'
i have tried setting the variable like this
"${stringVar}" | turns into ${stringVar}
"$stringVar" | turns into "$stringVar"
and
"'${stringVar}'"
"'$stringVar'"
both seem to return this error
{"error":{"code":"bad_request","message":"Invalid JSON"}}curl: (3) unmatched brace in URL position 1: {\n
and
stringVar
stringVar
$stringVar
"'"$stringVar"'"
""$stringVar""
${stringVar}
all seem to return this error
{"error":{"code":"bad_request","message":"Invalid JSON"}}
Ahh any help on what I am doing wrong would be great.
Thanks in advance y'all
In order to interpolate the value of a variable, you need to enclose the string in double-quotes("), but the JSON also requires literal double-quotes.
The easiest solution is probably to use a here-document to feed the data into curl's standard input, as in #Gilles Quénot's answer. But you can still pass it in via the command line; you just have to be careful with the quotes.
This is one way:
curl ... -d '{
"name": "my-project",
"files": [
{
"data": "'"$stringVar"'"
}
]
}'
The JSON here is mostly contained within single quotation marks ('...'). But right after opening the pair of literal "s that will enclose the value of data, we close the single quotes and switch our shell quotation mode to double quotes in order to include the value of $stringVar. After closing the double quotes around that expansion, we go back into single quotes for the rest of the JSON, starting with closing the literal double-quotes around the value of data.
In a language that used + for string concatenation, it would look like '... "data": "' + "$stringVar" + '"... ', but in the shell you just put the strings next to each other with no operator to concatenate them.
As an alternative, you could put the whole thing in double-quotes, but then you need backslashes to include the literal double quotes that are part of the JSON:
curl ... -d "{
\"name\": \"my-project\",
\"files\": [
{
\"data\": \"$stringVar\"
}
]
}"
So that requires a lot more changes if you're starting from plain JSON; it also looks messier, IMO.
You can also use a tool that knows how to build JSON and let it worry about quoting etc. Here's one way to build it with jq:
jq -n --arg data "$stringVar" '{
"name": "my-project",
"files": [
{
"data": $data
}
]
}'
Using --arg creates a variable inside jq – I named it data – which can then be included in an expression with the syntax $varname ($data in this case). Despite the similarity of syntax, that's not a shell interpolation; we're passing the literal text $data to jq, and jq itself is what replaces it with the value of the variable (which was passed as the second argument to --arg).
There's another tool called jo, which doesn't manipulate JSON but rather produces it, from input that is easier to generate in the shell. Here's one way to construct the desired object with it:
jo name=my-project files="$(jo -a "$(jo data="$stringVar")")"
Either way you can include the constructed JSON in your curl command line like this:
curl ... -d "$(jo or jq command goes here)"
Do not generate such JSON by hand. Use a tool like jq to do it for you.
#!/bin/bash
stringVar="helloWorldFunction() {\n echo \"hello world\" \n}"
jq -n --arg s "$stringVar" '{name: "my-project", files: [{data: $s}]}' |
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-X POST \
-d #- \
'https://api.example.com/v1/endpoint'
Like this:
#!/bin/bash
stringVar="..."
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' \
-d#/dev/stdin <<EOF
{
"name": "my-project",
"files": [
{
"data": $stringVar
},
],
}
EOF
You then should take care about what you fed in the variable, this have to be valid JSON
As an alternative to jq and curl you could use xidel to generate the JSON and submit the POST-request.
With command-line options:
#!/bin/bash
stringVar='helloWorldFunction() {\n echo "hello world" \n}'
xidel -s --variable var="$stringVar" \
-H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json"
-d '{serialize(
{"name":"my-project","files":array{{"data":$var}}},
{"method":"json"}
)}' \
"https://api.example.com/v1/endpoint" \
-e '$raw'
Or with the x:request() function in-query:
#!/bin/bash
stringVar='helloWorldFunction() {\n echo "hello world" \n}'
xidel -s --variable var="$stringVar" -e '
x:request({
"headers":(
"Authorization: Bearer <MY_API_TOKEN>",
"Content-Type: application/json",
"Accept: application/json"
),
"post":serialize(
{"name":"my-project","files":array{{"data":$var}}},
{"method":"json"}
),
"url":"https://api.example.com/v1/endpoint"
})/raw
'
$raw / /raw returns the raw output, like curl. If the API-endpoint returns JSON, then you can use $json / /json to parse it.

Non variable recognition on Syspass API Bash Script

When i input a variable on the curl with the json indexed, it takes the string value of the variable, if i scaped the double-quotes, it returns me a sintax error because of in a json request u have to input the data with double-quotes.
Example:
#!/bin/bash
token="x"
tokenPass="x"
name="prueba"
url="https://prueba.prueba.prueba"
user="prueba"
pass="prueba"
notes="prueba"
curl -k -H "Content-Type:Application/json" -d '{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "$token", "tokenPass": "$tokenPass", "name": "$name" , "categoryId": "7", "clientId": "9","login":"$user","url": "$url" ,"pass": "$pass","notes": "$notes"}, "id": 1}' -o- https://syspass.prueba.es/api.php
The curl json request works if i input the data manually, but with the variables, when i create the account, the account is named as the string value of the variable, i mean, if the variable is named $name , the account created is name $name. Any help please?
Also i tried to input the variable: "${variable}" and either works
Try this:
curl -k -H "Content-Type:Application/json" -d '{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "'${token}'", "tokenPass": "'${tokenPass}'", "name": "'${name}'" , "categoryId": "7", "clientId": "9","login":"'${user}'","url": "'${url}'" ,"pass": "'${pass}'","notes": "'${notes}'"}, "id": 1}' -o- https://syspass.prueba.es/api.php
A bit of explanation:
Text that is between single quotes ' will not be interpreted by bash and that is why the variables will not be inserted. You have to break the string up...
An other suggestion is that you should use curly braces around variables: When do we need curly braces around shell variables?
I explain it here for the variable token. The other variables can be treated in the same way.
Assuming that the variable token contains the string foo bar baz, then you want curl to receive as its 5th argument the following string:
{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "foo bar baz", "tokenPass":... rest omitted for brevity}
Since this has to become a single argument, you have to quote the hole string somehow, and because this string contains many double-quotes, it is easier to use single quotes for wrapping. However, this would prevent parameter expansion. To make this happen, the variables must be outside the single quotes. However, they must also be inside double quotes, lest you become a victim of word splitting, if the variables contain spaces. This yields the following solution:
curl .... '{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "'"$token"'", "tokenPass": ... rest omitted for brevity}'
Note the "'" in front of $token: The first double quote is the one which will be part of the expanded "foo bar baz". It must be literal, because curl wants to see it. The next single quote terminates the initial single quote. The next double quote tells the shell to expand $token, but without attempting word splitting.
If there are many parameters to expand, the resulting expression is difficult to debug and maintain. An alternative is to set up a HERE-document:
# Set up Parameter 5 for curl
read p5 <<END
{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "$token", "tokenPass":... rest omitted for brevity}
END
and then use it as
curl .... -d "$p5"
This requires the separated step of setting up the argument, but in the end is much more readable, and you don't have to worry about word splitting either.
I'd use jq to generate the JSON string (mostly, to handle the quoting for you, JSON strings are not trivial), and for readability (and for the sanity of the next maintainer) add some newlines and indentation:
data=$(
jq -n \
--arg token "$token" \
--arg tokenPass "$tokenPass" \
--arg name "$name" \
--arg url "$url" \
--arg user "$user" \
--arg pass "$pass" \
--arg notes "$notes" \
'{
jsonrpc: "2.0",
method: "account/create",
params: {
authToken: $token,
tokenPass: $tokenPass,
name: $name,
categoryId: "7",
clientId: "9",
login: $user,
url: $url ,
pass: $pass,
notes: $notes
},
id: 1
}'
)
curl -k \
-H "Content-Type:Application/json" \
-d "$data" \
-o- \
https://syspass.prueba.es/api.php

Interpolate command output into GitHub REST request

I am trying to create a pull request comment automatically whenever CI is run. The output of a given command is written to a file (could also just be stored inside an environment variable though). The problem is, I usually get the following response:
curl -XPOST -d "{'body':'$RESULT'}" https://api.github.com/repo/name/issues/number/comment
{
"message": "Problems parsing JSON",
"documentation_url": "https://developer.github.com/v3/issues/comments/#create-a-comment"
}
This is usually due to unescpaed characters, like \n, \t, " etc.
Is there any easy way to achieve this on the command line or in bash, sh, with jq or Python? Using the Octokit.rb library is works straight away, but I don't want to install Ruby in the build environment.
You can use jq to create your JSON object. Supposing you have your comment content in RESULT variable, the full request would be :
DATA=$(echo '{}' | jq --arg val "$RESULT" '.| {"body": $val}')
curl -s -H 'Content-Type: application/json' \
-H 'Authorization: token YOUR_TOKEN' \
-d "$DATA" \
"https://api.github.com/repos/:owner/:repo/issues/:number/comments"
The post "Using curl POST with variables defined in bash script functions" proposes multiple techniques for passing a varialbe like $RESULT in a curl POST parameter.
generate_post_data()
{
cat <<EOF
{
"body": "$RESULT"
}
EOF
}
Then, following "A curl tutorial using GitHub's API ":
curl -X POST \
-H "authToken: <yourToken" \
-H "Content-Type: application/json" \
--data "$(generate_post_data)" https://api.github.com/repo/name/issues/number/comment

JSON data load containing single quote loses the single quote when sent through curl, how to escape/keep single quotes [duplicate]

Here is an example of command line that fit this description :
curl http://dumbdomain.com/solr/collection2/update/json -H
'Content-type:application/json' -d ' { "add": { "doc": { "uid":
"79729", "text" : "I''ve got your number"} } }'
I already tried \' (not escaped), url encoded (not urldecoded at this other end!) and '' (quote disappear!), without success.
If you replace ' by unicode encoded ' (which is \u0027), then it works:
curl http://dumbdomain.com/solr/collection2/update/json -H 'Content-type:application/json' -d ' { "add": { "doc": { "uid": "79729", "text" : "I\u0027ve got your number"} } }'
Strange, but worth to know!
An usual workaround in such cases is to put the data in a file and post.
$ cat post.json
{ "add": { "doc": { "uid": "79729", "text" : "I've got your number"} } }
And then invoke:
curl -H "Content-type:application/json" --data #post.json http://dumbdomain.com/solr/collection2/update/json
This would obviate the need of escaping any quotes in the json.
In case you're using Windows (this problem typically doesn't occur on *nix), you can pipe the output from echo to curl to avoid the escaping altogether:
echo {"foo": "bar", "xyzzy": "fubar"} | curl -X POST -H "Content-Type: application/json" -d #- localhost:4444/api/foo
Do you mean how to get the JSON passed via the command line correctly? If you're using Windows then you need to be careful how you escape your string. It works if you use double quotes around the whole data string and then escape the double quotes for the JSON. For example:
curl http://dumbdomain.com/solr/collection2/update/json -H 'Content-type:application/json' -d "{ \"add\": { \"doc\": { \"uid\": \"79729\", \"text\" : \"I've got your number\"} } }"

Powershell curl double quotes

I am trying to invoke a curl command in powershell and pass some JSON information.
Here is my command:
curl -X POST -u username:password -H "Content-Type: application/json" -d "{ "fields": { "project": { "key": "key" }, "summary": "summary", "description": "description - here", "type": { "name": "Task" }}}"
I was getting globbing errors and "unmatched braces" and host could not be resolved, etc.
Then I tried prefixing the double quotes in the string with the backtick character, but it could not recognize the - character in the description json field
thanks
EDIT 1:
When I wrote the curl command in a regular batch file, I used double quotes and no single quotes. Also, in the -d string, I escaped all the double quotes with \ and the command worked.
In this case, my curl is actually pointing to curl.exe. I specified the path, just didn't list it here. Also I tried adding single quotes around -d and I got:
curl: option -: is unknown curl: try 'curl --help' or 'curl --manual' for more information
Seems like it cannot recognize the - character in the JSON
Pipe the data into curl.exe, instead of trying to escape it.
$data = #{
fields = #{
project = #{
key = "key"
}
summary = "summary"
description = "description - here"
type = #{
name = "Task"
}
}
}
$data | ConvertTo-Json -Compress | curl.exe -X POST -u username:password -H "Content-Type: application/json" -d "#-"
curl.exe reads stdin if you use #- as your data parameter.
P.S.: I strongly suggest you use a proper data structure and ConvertTo-Json, as shown, instead of building the JSON string manually.
Easy way (for simple testing):
curl -X POST -H "Content-Type: application/json" -d '{ \"field\": \"value\"}'