Github api not parsing multi line shell variables - json

I've got a text param (from jenkins job dsl plugin) in jenkins configuration which allows you to enter a multi line comment. I'm using that variable for the body value when posting a release to a github repository from a shell script. I'm getting this error that says problem parsing json and I can't find a workaround. I'll try to give you an example below. Please help.
PERSONAL_ACCESS_TOKEN="random"
TAG_NAME="12.0.0"
VERSION_BUMP="major"
MIGRATION_DOCUMENT="This is first line
This is second line"
curl -i \
-H "Authorization: token ${PERSONAL_ACCESS_TOKEN}" \
-d '{"tag_name": "'"${TAG_NAME}"'", "name": "'"${VERSION_BUMP}"'", \
"body": "'"${MIGRATION_DOCUMENT}"'"}' \
https://github.deere.com/api/v3/repos/randomOrg/testRepo/releases

This
{ "a": "b
c" }
is invalid JSON because a string must not contain control characters such as newlines.
If you have a string containing newlines, you can convert them to \n using shell parameter expansion:
$ var='a
b'
$ echo "$var"
a
b
$ echo "${var//$'\n'/'\n'}"
a\nb
So, to feed your string into your JSON object, use
"body": "'"${MIGRATION_DOCUMENT//$'\n'/'\n'}"'"
at the end of your JSON object.
Also, if you use line continuation in single quotes such as
var='abc \
def'
then the backslash and the linebreak are literal:
$ echo "$var"
abc \
def
Don't use line continuation like that in single quoted strings.
All in all:
curl -i \
-H "Authorization: token ${PERSONAL_ACCESS_TOKEN}" \
-d '{"tag_name": "'"${TAG_NAME}"'", "name": "'"${VERSION_BUMP}"'", "body": "'"${MIGRATION_DOCUMENT//$'\n'/'\n'}"'"}' \
https://github.deere.com/api/v3/repos/randomOrg/testRepo/releases
If you really want to, you can still use line continuation, but it has to be in a double quoted context:
curl -i \
-H "Authorization: token ${PERSONAL_ACCESS_TOKEN}" \
-d '{"tag_name": "'"${TAG_NAME}"'", "name": "'"${VERSION_BUMP}"'", '"\
"'"body": "'"${MIGRATION_DOCUMENT//$'\n'/'\n'}"'"}' \
https://github.deere.com/api/v3/repos/randomOrg/testRepo/releases
As a side note, you shouldn't use all uppercase names for variables; those are reserved for environment variables, see the POSIX spec (fourth paragraph).

Related

Trying to include a dynamic variable within JSON data of a Bash / Shell POST request to API endpoint

I am trying to submit a POST request using Bash that includes JSON data with a variable equal to that of a random string. I get the string dynamically by parsing a file which can vary in content... and may contain regex characters and also new lines.
I will provide an example string here with a curl request that works successfully with the API I am posting this request to. I can only get the request to go through with the string hardcoded into the JSON data and not while assigning a variable to the string like for instance stringVar and using the variable in the JSON data. I could sure use some help where I am messing this up
my working shell script looks something like this
#!/bin/bash
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' -d '{
"name": "my-project",
"files": [
{
"data": "helloWorldFunction() {\n echo \"hello world\" \n}"
},
],
}'
This works, however I need to change data's value from the string
helloWorldFunction() {\n echo "hello world" \n}
to a variable
I have tried settings the variable in different ways in the JSON content from reading suggestions on other questions. For instance I have tried tried changing my shell script to
#!/bin/bash
stringVar="helloWorldFunction() {\n echo \"hello world\" \n}"
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' -d '{
"name": "my-project",
"files": [
{
"data": "${stringVar}"
},
],
}'
i have tried setting the variable like this
"${stringVar}" | turns into ${stringVar}
"$stringVar" | turns into "$stringVar"
and
"'${stringVar}'"
"'$stringVar'"
both seem to return this error
{"error":{"code":"bad_request","message":"Invalid JSON"}}curl: (3) unmatched brace in URL position 1: {\n
and
stringVar
stringVar
$stringVar
"'"$stringVar"'"
""$stringVar""
${stringVar}
all seem to return this error
{"error":{"code":"bad_request","message":"Invalid JSON"}}
Ahh any help on what I am doing wrong would be great.
Thanks in advance y'all
In order to interpolate the value of a variable, you need to enclose the string in double-quotes("), but the JSON also requires literal double-quotes.
The easiest solution is probably to use a here-document to feed the data into curl's standard input, as in #Gilles Quénot's answer. But you can still pass it in via the command line; you just have to be careful with the quotes.
This is one way:
curl ... -d '{
"name": "my-project",
"files": [
{
"data": "'"$stringVar"'"
}
]
}'
The JSON here is mostly contained within single quotation marks ('...'). But right after opening the pair of literal "s that will enclose the value of data, we close the single quotes and switch our shell quotation mode to double quotes in order to include the value of $stringVar. After closing the double quotes around that expansion, we go back into single quotes for the rest of the JSON, starting with closing the literal double-quotes around the value of data.
In a language that used + for string concatenation, it would look like '... "data": "' + "$stringVar" + '"... ', but in the shell you just put the strings next to each other with no operator to concatenate them.
As an alternative, you could put the whole thing in double-quotes, but then you need backslashes to include the literal double quotes that are part of the JSON:
curl ... -d "{
\"name\": \"my-project\",
\"files\": [
{
\"data\": \"$stringVar\"
}
]
}"
So that requires a lot more changes if you're starting from plain JSON; it also looks messier, IMO.
You can also use a tool that knows how to build JSON and let it worry about quoting etc. Here's one way to build it with jq:
jq -n --arg data "$stringVar" '{
"name": "my-project",
"files": [
{
"data": $data
}
]
}'
Using --arg creates a variable inside jq – I named it data – which can then be included in an expression with the syntax $varname ($data in this case). Despite the similarity of syntax, that's not a shell interpolation; we're passing the literal text $data to jq, and jq itself is what replaces it with the value of the variable (which was passed as the second argument to --arg).
There's another tool called jo, which doesn't manipulate JSON but rather produces it, from input that is easier to generate in the shell. Here's one way to construct the desired object with it:
jo name=my-project files="$(jo -a "$(jo data="$stringVar")")"
Either way you can include the constructed JSON in your curl command line like this:
curl ... -d "$(jo or jq command goes here)"
Do not generate such JSON by hand. Use a tool like jq to do it for you.
#!/bin/bash
stringVar="helloWorldFunction() {\n echo \"hello world\" \n}"
jq -n --arg s "$stringVar" '{name: "my-project", files: [{data: $s}]}' |
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-X POST \
-d #- \
'https://api.example.com/v1/endpoint'
Like this:
#!/bin/bash
stringVar="..."
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' \
-d#/dev/stdin <<EOF
{
"name": "my-project",
"files": [
{
"data": $stringVar
},
],
}
EOF
You then should take care about what you fed in the variable, this have to be valid JSON
As an alternative to jq and curl you could use xidel to generate the JSON and submit the POST-request.
With command-line options:
#!/bin/bash
stringVar='helloWorldFunction() {\n echo "hello world" \n}'
xidel -s --variable var="$stringVar" \
-H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json"
-d '{serialize(
{"name":"my-project","files":array{{"data":$var}}},
{"method":"json"}
)}' \
"https://api.example.com/v1/endpoint" \
-e '$raw'
Or with the x:request() function in-query:
#!/bin/bash
stringVar='helloWorldFunction() {\n echo "hello world" \n}'
xidel -s --variable var="$stringVar" -e '
x:request({
"headers":(
"Authorization: Bearer <MY_API_TOKEN>",
"Content-Type: application/json",
"Accept: application/json"
),
"post":serialize(
{"name":"my-project","files":array{{"data":$var}}},
{"method":"json"}
),
"url":"https://api.example.com/v1/endpoint"
})/raw
'
$raw / /raw returns the raw output, like curl. If the API-endpoint returns JSON, then you can use $json / /json to parse it.

Passing variables (containing spaces) to curl --data field

Arguments containing spaces will not properly pass to the curl command.
Quotes are not passed correctly in the --data field.
If I just echo the variable 'curlData' that I use in the curl command I get everything as it should be; ex :
$echo $curlData
'{"name":"jason","description","service"}'
I don't understand why curl dont expend this 'curlData' variable as expected:
curl --data '{"name":"jason","description","service"}'
Here's a sample of my code:
read -p "Name : " repoName
read -p "Description []: " repoDescription
curlData="'"{'"'name'"':'"'$repoName'"','"'descripton'"':'"'$repoDescription'"'}"'"
curl --data $curlData $apiURL
And the error:
curl: (3) [globbing] unmatched close brace/bracket in column 26
Thank your for your help, I feel i'm in Quote-ception right now.
Quote all variable expansions,
To make sure that curlData is a valid JSON value with properly escaped special-characters etc., use jq for producing it.
curlData="$(jq --arg name "$repoName" --arg desc "$repoDescription" -nc '{name:$name,description:$desc}')"
curl --data "$curlData" "$apiURL"
If you have access to any form of package management, I highly recommend jo.
curlData=$(jo name="$repoName" description="$repoDescription")
curl -d "$curlData" "$apiURL"
I had similar issue, which was very difficult to even understand. I used the below construct in a number of curl commands present in my shell script. It always worked like a charm. Until one fine day I had to pass a variable which was string containing spaces (eg. modelName="Abc def").
curl -X 'PUT' \
'http://localhost:43124/api/v1/devices/'$Id'' \
-H 'accept:*/*' \
-H 'Authorization:Bearer '$token'' \
-H 'Content-Type:application/json' \
-d '{
"modelName":"'$modelName'",
"serialNumber":"'$childSN'"
}'
Worked for me after the below change
curl -X 'PUT' \
'http://localhost:43124/api/v1/devices/'$Id'' \
-H 'accept:*/*' \
-H 'Authorization:Bearer '$token'' \
-H 'Content-Type:application/json' \
-d '{
"modelName":'\""$modelName"\"',
"serialNumber":"'$childSN'"
}'
I took help from the accepted answer by #oguz. Pasting this response , just in case anyone is in similar situation

Interpolate command output into GitHub REST request

I am trying to create a pull request comment automatically whenever CI is run. The output of a given command is written to a file (could also just be stored inside an environment variable though). The problem is, I usually get the following response:
curl -XPOST -d "{'body':'$RESULT'}" https://api.github.com/repo/name/issues/number/comment
{
"message": "Problems parsing JSON",
"documentation_url": "https://developer.github.com/v3/issues/comments/#create-a-comment"
}
This is usually due to unescpaed characters, like \n, \t, " etc.
Is there any easy way to achieve this on the command line or in bash, sh, with jq or Python? Using the Octokit.rb library is works straight away, but I don't want to install Ruby in the build environment.
You can use jq to create your JSON object. Supposing you have your comment content in RESULT variable, the full request would be :
DATA=$(echo '{}' | jq --arg val "$RESULT" '.| {"body": $val}')
curl -s -H 'Content-Type: application/json' \
-H 'Authorization: token YOUR_TOKEN' \
-d "$DATA" \
"https://api.github.com/repos/:owner/:repo/issues/:number/comments"
The post "Using curl POST with variables defined in bash script functions" proposes multiple techniques for passing a varialbe like $RESULT in a curl POST parameter.
generate_post_data()
{
cat <<EOF
{
"body": "$RESULT"
}
EOF
}
Then, following "A curl tutorial using GitHub's API ":
curl -X POST \
-H "authToken: <yourToken" \
-H "Content-Type: application/json" \
--data "$(generate_post_data)" https://api.github.com/repo/name/issues/number/comment

Passing a shell variable to a JSON request to curl?

Let's take the following example:
curl -i -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":false}}' \
http://example.com/jsonrpc
Now I want to have the boolean value of "item" be set in a shell script variable such as:
PRIVATE=false
read -p "Is this a private? (y/[n]) " -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]; then
PRIVATE=true
fi
And I want to pass in the value of PRIVATE to item. I have tried every which way but no luck. Can anyone shed some light?
You can do it this way:
curl -i -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":'"$PRIVATE"'}}' \
http://example.com/jsonrpc
Instead of your existing -d ... line above, you could try the following:
-d "{\"jsonrpc\": \"2.0\", \"method\": \"Player.Open\", \"params\":{\"item\":$PRIVATE}}" \
That is: when using double quote speechmarks ("), bash substitutes values for variables referenced $LIKE_THIS (not the case for single quotes you were using). The downside is that you then need to escape any double-quotes in the string itself (using the backslash, as above).
This abomination works too.
$ npm run script -- madrid
# script
json='{"city":"'"$1"'"}'
curl -X POST -d $json http://localhost:5678/api/v1/weather

curl error when using url string from string

I'm now testing some restful APIs using bash shell script.
I want to read url from file then make a json data string with the url in file.
For the test, below codes work fine. It's not reading from file.
#!/bin/bash
URL=http://test.com/test.jpg
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$URL"'"}' \
http://api.test.com/test
But, it returns some error when I'm using the codes like below.
#!/bin/bash
FILE=./url.txt
cat $FILE | while read line; do
echo $line # or whaterver you want to do with the $line variable
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$line"'"}' \
http://api.test.com/test
done
But, it returns error when I use the string from reading file.
This is error message.
Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in string value
at [Source: org.apache.catalina.connector.CoyoteInputStream#27eb679c; line: 1, column: 237]
How to solve this issue?
Why it returns error when I use the string from file reading?
It appears that your file is in dos format with \n\r line terminators. Try running dos2unix on it to strip the \rs. Additionally, no need to cat the file, use redirection, like so
while read -r line; do
echo $line # or whaterver you want to do with the $line variable
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$line"'"}' \
http://api.test.com/test
done < "$FILE"
Also, pass -r to read to prevent backslash escaping