I've got a text param (from jenkins job dsl plugin) in jenkins configuration which allows you to enter a multi line comment. I'm using that variable for the body value when posting a release to a github repository from a shell script. I'm getting this error that says problem parsing json and I can't find a workaround. I'll try to give you an example below. Please help.
PERSONAL_ACCESS_TOKEN="random"
TAG_NAME="12.0.0"
VERSION_BUMP="major"
MIGRATION_DOCUMENT="This is first line
This is second line"
curl -i \
-H "Authorization: token ${PERSONAL_ACCESS_TOKEN}" \
-d '{"tag_name": "'"${TAG_NAME}"'", "name": "'"${VERSION_BUMP}"'", \
"body": "'"${MIGRATION_DOCUMENT}"'"}' \
https://github.deere.com/api/v3/repos/randomOrg/testRepo/releases
This
{ "a": "b
c" }
is invalid JSON because a string must not contain control characters such as newlines.
If you have a string containing newlines, you can convert them to \n using shell parameter expansion:
$ var='a
b'
$ echo "$var"
a
b
$ echo "${var//$'\n'/'\n'}"
a\nb
So, to feed your string into your JSON object, use
"body": "'"${MIGRATION_DOCUMENT//$'\n'/'\n'}"'"
at the end of your JSON object.
Also, if you use line continuation in single quotes such as
var='abc \
def'
then the backslash and the linebreak are literal:
$ echo "$var"
abc \
def
Don't use line continuation like that in single quoted strings.
All in all:
curl -i \
-H "Authorization: token ${PERSONAL_ACCESS_TOKEN}" \
-d '{"tag_name": "'"${TAG_NAME}"'", "name": "'"${VERSION_BUMP}"'", "body": "'"${MIGRATION_DOCUMENT//$'\n'/'\n'}"'"}' \
https://github.deere.com/api/v3/repos/randomOrg/testRepo/releases
If you really want to, you can still use line continuation, but it has to be in a double quoted context:
curl -i \
-H "Authorization: token ${PERSONAL_ACCESS_TOKEN}" \
-d '{"tag_name": "'"${TAG_NAME}"'", "name": "'"${VERSION_BUMP}"'", '"\
"'"body": "'"${MIGRATION_DOCUMENT//$'\n'/'\n'}"'"}' \
https://github.deere.com/api/v3/repos/randomOrg/testRepo/releases
As a side note, you shouldn't use all uppercase names for variables; those are reserved for environment variables, see the POSIX spec (fourth paragraph).
I am giving the user the possibility to answers some questions so that I can perform the according cURL requests and create what is needed accordingly.
In this way:
~functions.sh
echo "Question number 1?"
read question1
echo "Question number 2?"
read question2
The script is organised in a couple of files:
functions.sh, where I store all the functions
flow.sh, where the questions are asked and the previous functions are called in order
credentials.sh, where I store the api key credentials, api url etc.
env.sh, where I am hoping to store the env configuration that needs to be dynamically generated...
Unfortunately I am pretty much new to that as I come from a Frontend Background so I Am not aware or can't figure out/find on here, how to achieve the following:
I would need to send a cURL request based on one of the users answers, and that's easily done just by calling ie {$question1} inside one of the functions of functions.sh that is later called in the main flow.sh after putting the file as a source...
Unfortunately, when it comes to populating a multiline environment configuration in a separate file to be later be sent as a json in the cURL it gets out of my comfort zone...
Id need so send something along the lines of
curl -X POST "https//api.url"
-H "headers"
--data '{"content":"'{$env}'"}'
Where $env is the env.sh file, json valid, and populated with the variables read before from user input accordingly...
To solve the multi line json problem maybe I could use the following:
$( awk '{printf "%s\\n", $0}' ./env.sh )
Still I am clueless on how to populate such file form user read and echo it inside that json part in the cURL...
You could use awk to directly save correct JSON data to env.sh (partly taken from Dr.K's answer to this other question):
echo "Question number 1?"
read question1
echo "Question number 2?"
read question2
echo "Question number 3?"
read question3
awk ' BEGIN { print "[" ; } \
{ print " {" \
print " \"Question1\" : \"" $question1 "\"," \
print " \"Question2\" : \"" $question2 "\"," \
print " \"Question3\" : \"" $question3 "\"" \
print " }" \
} \
END { print "]" } ' env.sh
And then this should work correctly as JSON:
curl -X POST "https//api.url"
-H "headers"
--data '{"content":"{$env}"}'
and the data payload should be like this:
{
content: [
{
"Question1" : "User answer 1",
"Question2" : "User answer 2",
"Question3" : "User answer 3"
}
]
}
Is this what you had in mind?
For anyone who ends up here what I did was:
Creating a separate file where I set a variable and I cat the EOF inside of that.
env=$( cat <<EOF
multi
line
content
goes
here
EOF
)
export env
Then I generate a valid json through jq
generate_json() {
newenv="$( jq -nc --arg str "$env" '{"content": $str}' )"
}
and finally I use $newenv in the curl straight away:
example_curl() {
curl -X PUT "https://apiendpoint.co.uk" \
-H "Authorization: Bearer 123123" \
--data "${newenv}"
}
I need to read these bash variables into my JSON string and I am not familiar with bash. any help is appreciated.
#!/bin/sh
BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar
JSON_STRING='{"bucketname":"$BUCKET_NAME"","objectname":"$OBJECT_NAME","targetlocation":"$TARGET_LOCATION"}'
echo $JSON_STRING
You are better off using a program like jq to generate the JSON, if you don't know ahead of time if the contents of the variables are properly escaped for inclusion in JSON. Otherwise, you will just end up with invalid JSON for your trouble.
BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar
JSON_STRING=$( jq -n \
--arg bn "$BUCKET_NAME" \
--arg on "$OBJECT_NAME" \
--arg tl "$TARGET_LOCATION" \
'{bucketname: $bn, objectname: $on, targetlocation: $tl}' )
You can use printf:
JSON_FMT='{"bucketname":"%s","objectname":"%s","targetlocation":"%s"}\n'
printf "$JSON_FMT" "$BUCKET_NAME" "$OBJECT_NAME" "$TARGET_LOCATION"
much clear and simpler
A possibility:
#!/bin/bash
BUCKET_NAME="testbucket"
OBJECT_NAME="testworkflow-2.0.1.jar"
TARGET_LOCATION="/opt/test/testworkflow-2.0.1.jar
# one line
JSON_STRING='{"bucketname":"'"$BUCKET_NAME"'","objectname":"'"$OBJECT_NAME"'","targetlocation":"'"$TARGET_LOCATION"'"}'
# multi-line
JSON_STRING="{
\"bucketname\":\"${BUCKET_NAME}\",
\"objectname\":\"${OBJECT_NAME}\",
\"targetlocation\":\"${TARGET_LOCATION}\"
}"
# [optional] validate the string is valid json
echo "${JSON_STRING}" | jq
In addition to chepner's answer, it's also possible to construct the object completely from args with this simple recipe:
BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar
JSON_STRING=$(jq -n \
--arg bucketname "$BUCKET_NAME" \
--arg objectname "$OBJECT_NAME" \
--arg targetlocation "$TARGET_LOCATION" \
'$ARGS.named')
Explanation:
--null-input | -n disabled reading input. From the man page: Don't read any input at all! Instead, the filter is run once using null as the input. This is useful when using jq as a simple calculator or to construct JSON data from scratch.
--arg name value passes values to the program as predefined variables: value is available as $name. All named arguments are also available as $ARGS.named
Because the format of $ARGS.named is already an object, jq can output it as is.
First, don't use ALL_CAPS_VARNAMES: it's too easy to accidentally overwrite a crucial shell variable (like PATH)
Mixing single and double quotes in shell strings can be a hassle. In this case, I'd use printf:
bucket_name=testbucket
object_name=testworkflow-2.0.1.jar
target_location=/opt/test/testworkflow-2.0.1.jar
template='{"bucketname":"%s","objectname":"%s","targetlocation":"%s"}'
json_string=$(printf "$template" "$BUCKET_NAME" "$OBJECT_NAME" "$TARGET_LOCATION")
echo "$json_string"
For homework, read this page carefully: Security implications of forgetting to quote a variable in bash/POSIX shells
A note on creating JSON with string concatenation: there are edge cases. For example, if any of your strings contain double quotes, you can broken JSON:
$ bucket_name='a "string with quotes"'
$ printf '{"bucket":"%s"}\n' "$bucket_name"
{"bucket":"a "string with quotes""}
Do do this more safely with bash, we need to escape that string's double quotes:
$ printf '{"bucket":"%s"}\n' "${bucket_name//\"/\\\"}"
{"bucket":"a \"string with quotes\""}
I had to work out all possible ways to deal json strings in a command request, Please look at the following code to see why using single quotes can fail if used incorrectly.
# Create Release and Tag commit in Github repository
# returns string with in-place substituted variables
json=$(cat <<-END
{
"tag_name": "${version}",
"target_commitish": "${branch}",
"name": "${title}",
"body": "${notes}",
"draft": ${is_draft},
"prerelease": ${is_prerelease}
}
END
)
# returns raw string without any substitutions
# single or double quoted delimiter - check HEREDOC specs
json=$(cat <<-!"END" # or 'END'
{
"tag_name": "${version}",
"target_commitish": "${branch}",
"name": "${title}",
"body": "${notes}",
"draft": ${is_draft},
"prerelease": ${is_prerelease}
}
END
)
# prints fully formatted string with substituted variables as follows:
echo "${json}"
{
"tag_name" : "My_tag",
"target_commitish":"My_branch"
....
}
Note 1: Use of single vs double quotes
# enclosing in single quotes means no variable substitution
# (treats everything as raw char literals)
echo '${json}'
${json}
echo '"${json}"'
"${json}"
# enclosing in single quotes and outer double quotes causes
# variable expansion surrounded by single quotes(treated as raw char literals).
echo "'${json}'"
'{
"tag_name" : "My_tag",
"target_commitish":"My_branch"
....
}'
Note 2: Caution with Line terminators
Note the json string is formatted with line terminators such as LF \n
or carriage return \r(if its encoded on windows it contains CRLF \r\n)
using (translate) tr utility from shell we can remove the line terminators if any
# following code serializes json and removes any line terminators
# in substituted value/object variables too
json=$(echo "$json" | tr -d '\n' | tr -d '\r' )
# string enclosed in single quotes are still raw literals
echo '${json}'
${json}
echo '"${json}"'
"${json}"
# After CRLF/LF are removed
echo "'${json}'"
'{ "tag_name" : "My_tag", "target_commitish":"My_branch" .... }'
Note 3: Formatting
while manipulating json string with variables, we can use combination of ' and " such as following, if we want to protect some raw literals using outer double quotes to have in place substirution/string interpolation:
# mixing ' and "
username=admin
password=pass
echo "$username:$password"
admin:pass
echo "$username"':'"$password"
admin:pass
echo "$username"'[${delimiter}]'"$password"
admin[${delimiter}]pass
Note 4: Using in a command
Following curl request already removes existing \n (ie serializes json)
response=$(curl -i \
--user ${username}:${api_token} \
-X POST \
-H 'Accept: application/vnd.github.v3+json' \
-d "$json" \
"https://api.github.com/repos/${username}/${repository}/releases" \
--output /dev/null \
--write-out "%{http_code}" \
--silent
)
So when using it for command variables, validate if it is properly formatted before using it :)
If you need to build a JSON representation where members mapped to undefined or empty variables should be ommited, then jo can help.
#!/bin/bash
BUCKET_NAME=testbucket
OBJECT_NAME=""
JO_OPTS=()
if [[ ! "${BUCKET_NAME}x" = "x" ]] ; then
JO_OPTS+=("bucketname=${BUCKET_NAME}")
fi
if [[ ! "${OBJECT_NAME}x" = "x" ]] ; then
JO_OPTS+=("objectname=${OBJECT_NAME}")
fi
if [[ ! "${TARGET_LOCATION}x" = "x" ]] ; then
JO_OPTS+=("targetlocation=${TARGET_LOCATION}")
fi
jo "${JO_OPTS[#]}"
The output of the commands above would be just (note the absence of objectname and targetlocation members):
{"bucketname":"testbucket"}
can be done following way:
JSON_STRING='{"bucketname":"'$BUCKET_NAME'","objectname":"'$OBJECT_NAME'","targetlocation":"'$TARGET_LOCATION'"}'
For Node.js Developer, or if you have node environment installed, you can try this:
JSON_STRING=$(node -e "console.log(JSON.stringify({bucketname: $BUCKET_NAME, objectname: $OBJECT_NAME, targetlocation: $TARGET_LOCATION}))")
Advantage of this method is you can easily convert very complicated JSON Object (like object contains array, or if you need int value instead of string) to JSON String without worrying about invalid json error.
Disadvantage is it's relying on Node.js environment.
These solutions come a little late but I think they are inherently simpler that previous suggestions (avoiding the complications of quoting and escaping).
BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar
# Initial unsuccessful solution
JSON_STRING='{"bucketname":"$BUCKET_NAME","objectname":"$OBJECT_NAME","targetlocation":"$TARGET_LOCATION"}'
echo $JSON_STRING
# If your substitution variables have NO whitespace this is sufficient
JSON_STRING=$(tr -d [:space:] <<JSON
{"bucketname":"$BUCKET_NAME","objectname":"$OBJECT_NAME","targetlocation":"$TARGET_LOCATION"}
JSON
)
echo $JSON_STRING
# If your substitution variables are more general and maybe have whitespace this works
JSON_STRING=$(jq -c . <<JSON
{"bucketname":"$BUCKET_NAME","objectname":"$OBJECT_NAME","targetlocation":"$TARGET_LOCATION"}
JSON
)
echo $JSON_STRING
#... A change in layout could also make it more maintainable
JSON_STRING=$(jq -c . <<JSON
{
"bucketname" : "$BUCKET_NAME",
"objectname" : "$OBJECT_NAME",
"targetlocation" : "$TARGET_LOCATION"
}
JSON
)
echo $JSON_STRING
To build upon Hao's answer using NodeJS: you can split up the lines, and use the -p option which saves having to use console.log.
JSON_STRING=$(node -pe "
JSON.stringify({
bucketname: process.env.BUCKET_NAME,
objectname: process.env.OBJECT_NAME,
targetlocation: process.env.TARGET_LOCATION
});
")
An inconvenience is that you need to export the variables beforehand, i.e.
export BUCKET_NAME=testbucket
# etc.
Note: You might be thinking, why use process.env? Why not just use single quotes and have bucketname: '$BUCKET_NAME', etc so bash inserts the variables? The reason is that using process.env is safer - if you don't have control over the contents of $TARGET_LOCATION it could inject JavaScript into your node command and do malicious things (by closing the single quote, e.g. the $TARGET_LOCATION string contents could be '}); /* Here I can run commands to delete files! */; console.log({'a': 'b. On the other hand, process.env takes care of sanitising the input.
You could use envsubst:
export VAR="some_value_here"
echo '{"test":"$VAR"}' | envsubst > json.json
also it might be a "template" file:
//json.template
{"var": "$VALUE", "another_var":"$ANOTHER_VALUE"}
So after you could do:
export VALUE="some_value_here"
export ANOTHER_VALUE="something_else"
cat json.template | envsubst > misha.json
For a general case of building JSON from bash with arbitrary inputs, many of the previous responses (even the high voted ones with jq) omit cases when the variables contain " double quote, or \n newline escape string, and you need complex string concatenation of the inputs.
When using jq you need to printf %b the input first to get the \n converted to real newlines, so that once you pass through jq you get \n back and not \\n.
I found this with version with nodejs to be quite easy to reason about if you know javascript/nodejs well:
TITLE='Title'
AUTHOR='Bob'
JSON=$( TITLE="$TITLE" AUTHOR="$AUTHOR" node -p 'JSON.stringify( {"message": `Title: ${process.env.TITLE}\n\nAuthor: ${process.env.AUTHOR}`} )' )
It's a bit verbose due to process.env. but allows to properly pass the variables from shell, and then format things inside (nodejs) backticks in a safe way.
This outputs:
printf "%s\n" "$JSON"
{"message":"Title: Title\n\nAuthor: Bob"}
(Note: when having a variable with \n always use printf "%s\n" "$VAR" and not echo "$VAR", whose output is platform-dependent! See here for details)
Similar thing with jq would be
TITLE='Title'
AUTHOR='Bob'
MESSAGE="Title: ${TITLE}\n\nAuthor: ${AUTHOR}"
MESSAGE_ESCAPED_FOR_JQ=$(printf %b "${MESSAGE}")
JSON=$( jq '{"message": $jq_msg}' --arg jq_msg "$MESSAGE_ESCAPED_FOR_JQ" --null-input --compact-output --raw-output --monochrome-output )
(the last two params are not necessary when running in a subshell, but I just added them so that the output is then same when you run the jq command in a top-level shell).
Bash will not insert variables into a single-quote string. In order to get the variables bash needs a double-quote string.
You need to use double-quote string for the JSON and just escape double-quote characters inside JSON string.
Example:
#!/bin/sh
BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar
JSON_STRING="{\"bucketname\":\"$BUCKET_NAME\",\"objectname\":\"$OBJECT_NAME\",\"targetlocation\":\"$TARGET_LOCATION\"}"
echo $JSON_STRING
if you have node.js and get minimist installed in global:
jc() {
node -p "JSON.stringify(require('minimist')(process.argv), (k,v) => k=='_'?undefined:v)" -- "$#"
}
jc --key1 foo --number 12 --boolean \
--under_score 'abc def' --'white space' ' '
# {"key1":"foo","number":12,"boolean":true,"under_score":"abc def","white space":" "}
you can post it with curl or what:
curl --data "$(jc --type message --value 'hello world!')" \
--header 'content-type: application/json' \
http://server.ip/api/endpoint
be careful that minimist will parse dot:
jc --m.room.member #gholk:ccns.io
# {"m":{"room":{"member":"#gholk:ccns.io"}}}
Used this for AWS Macie configuration:
JSON_CONFIG=$( jq -n \
--arg bucket_name "$BUCKET_NAME" \
--arg kms_key_arn "$KMS_KEY_ARN" \
'{"s3Destination":{"bucketName":$bucket_name,"kmsKeyArn":$kms_key_arn}}'
)
aws macie2 put-classification-export-configuration --configuration "$JSON_CONFIG"
You can simply make a call like this to print the JSON.
#!/bin/sh
BUCKET_NAME=testbucket
OBJECT_NAME=testworkflow-2.0.1.jar
TARGET_LOCATION=/opt/test/testworkflow-2.0.1.jar
echo '{ "bucketName": "'"$BUCKET_NAME"'", "objectName": "'"$OBJECT_NAME"'", "targetLocation": "'"$TARGET_LOCATION"'" }'
or
JSON_STRING='{ "bucketName": "'"$BUCKET_NAME"'", "objectName": "'"$OBJECT_NAME"'", "targetLocation": "'"$TARGET_LOCATION"'" }'
echo $JOSN_STRING
I am trying to invoke a curl command in powershell and pass some JSON information.
Here is my command:
curl -X POST -u username:password -H "Content-Type: application/json" -d "{ "fields": { "project": { "key": "key" }, "summary": "summary", "description": "description - here", "type": { "name": "Task" }}}"
I was getting globbing errors and "unmatched braces" and host could not be resolved, etc.
Then I tried prefixing the double quotes in the string with the backtick character, but it could not recognize the - character in the description json field
thanks
EDIT 1:
When I wrote the curl command in a regular batch file, I used double quotes and no single quotes. Also, in the -d string, I escaped all the double quotes with \ and the command worked.
In this case, my curl is actually pointing to curl.exe. I specified the path, just didn't list it here. Also I tried adding single quotes around -d and I got:
curl: option -: is unknown curl: try 'curl --help' or 'curl --manual' for more information
Seems like it cannot recognize the - character in the JSON
Pipe the data into curl.exe, instead of trying to escape it.
$data = #{
fields = #{
project = #{
key = "key"
}
summary = "summary"
description = "description - here"
type = #{
name = "Task"
}
}
}
$data | ConvertTo-Json -Compress | curl.exe -X POST -u username:password -H "Content-Type: application/json" -d "#-"
curl.exe reads stdin if you use #- as your data parameter.
P.S.: I strongly suggest you use a proper data structure and ConvertTo-Json, as shown, instead of building the JSON string manually.
Easy way (for simple testing):
curl -X POST -H "Content-Type: application/json" -d '{ \"field\": \"value\"}'
I'm using curl to send JSON to an API endpoint. However, somewhere in the bash chain it is getting messed up.
Is there something special to know about encoding with curl?
If I construct the payload like this:
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "'
PAYLOAD+=$1
PAYLOAD+=' " }'
echo $PAYLOAD
curl -X POST --data-urlencode "$PAYLOAD" $SLACKPOSTURL
echo "sent"
I'll get back an error
Payload was not valid JSONsent
however if i just hardwire to assign a variable with the output
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "LAST_COMMIT Merge pull request #558 from dcsan/boteditor Boteditor " }'
then it will go through fine.
Is there something that a simple assignment is doing differently vs. concatenating strings? In the console the output looks identical.
FWIW some messages go through but content like this:
LAST_COMMIT Merge pull request #558 from dcsan/boteditor Boteditor
will only go through if hardcoded in. so its not the other end afaican see, its something to do with the way messages are built.
I guess you want to concatenate values into your variable. But += is not the way to do so.
To concatenate strings in a variable you need to say:
PAYLOAD="$PAYLOAD $1"
All together it would be something like the following. Note the need to use " so that the variable $PAYLOAD is expanded and the usage of \" to store a literal double quote:
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "'
PAYLOAD="$PAYLOAD $1 \" }"
echo "$PAYLOAD"
curl -X POST --data-urlencode "$PAYLOAD" $SLACKPOSTURL
echo "sent"
This is what worked from me from a bash script:
curl -X POST --data-urlencode "payload={\"text\": \"$2\"}" https://hooks.slack.com/services/$KEY
Notice the inner quotes are escaped, but the outer quotes are not.
FYI, adding:
set -x
at the beginning of a bash script will show you the actual commands being executed, and save a lot of guesswork.