i'm trying to build some script that i will need to check json via curl and print the output, but somehow it's not working. My other API JSON are working fine using this script
Script :
#!/bin/bash
OUT=$(curl -X POST -s -w "|%{time_total}|%{http_code}"\
-H "Content-Type: application/json" \
--ciphers 'AES256-GCM-SHA384' \
-d '{
"Username": "van_user1test",
"Password" : "123MM4566"}' \
https://127.0.0.1/api/call/login)
echo "THIS IS MASTER OUT" $OUT
out2="$(echo $OUT | cut -d\| -f 1)"
out_response="$(echo ${OUT} | cut -d\| -f 2)"
out_httpcode="$(echo ${OUT} | cut -d\| -f 3)"
echo "This is out2" $out2
echo "this is response" $out_response
echo "this is httpcode" $out_httpcode
And i'm getting as result :
}|0.437357|200ccess at 07/03/2022 11:23:11"
}Message": "Success at 07/03/2022 11:23:11"
this is response 0.437357
this is httpcode 200
But when i run curl command(above) manually, i get fine response :
{
"Successmessage": true,
"Message_status": "Success at 07/03/2022 11:27:28"
}|0.483469|200
I guess the problem can be in the tags somewhere because the first line SUccessmessage is missing and i'm worried why, also the format is not displayed correctly
I want to get fully response from json output and printing all the lines
Update : HOW to display in one line the latest result ?
{
"Successmessage": true,
"Message_status": "Success at 07/03/2022 11:27:28"
}|0.483469|200
Thank you!
Related
I am trying to submit a POST request using Bash that includes JSON data with a variable equal to that of a random string. I get the string dynamically by parsing a file which can vary in content... and may contain regex characters and also new lines.
I will provide an example string here with a curl request that works successfully with the API I am posting this request to. I can only get the request to go through with the string hardcoded into the JSON data and not while assigning a variable to the string like for instance stringVar and using the variable in the JSON data. I could sure use some help where I am messing this up
my working shell script looks something like this
#!/bin/bash
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' -d '{
"name": "my-project",
"files": [
{
"data": "helloWorldFunction() {\n echo \"hello world\" \n}"
},
],
}'
This works, however I need to change data's value from the string
helloWorldFunction() {\n echo "hello world" \n}
to a variable
I have tried settings the variable in different ways in the JSON content from reading suggestions on other questions. For instance I have tried tried changing my shell script to
#!/bin/bash
stringVar="helloWorldFunction() {\n echo \"hello world\" \n}"
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" -H "Content-Type: application/json" -H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' -d '{
"name": "my-project",
"files": [
{
"data": "${stringVar}"
},
],
}'
i have tried setting the variable like this
"${stringVar}" | turns into ${stringVar}
"$stringVar" | turns into "$stringVar"
and
"'${stringVar}'"
"'$stringVar'"
both seem to return this error
{"error":{"code":"bad_request","message":"Invalid JSON"}}curl: (3) unmatched brace in URL position 1: {\n
and
stringVar
stringVar
$stringVar
"'"$stringVar"'"
""$stringVar""
${stringVar}
all seem to return this error
{"error":{"code":"bad_request","message":"Invalid JSON"}}
Ahh any help on what I am doing wrong would be great.
Thanks in advance y'all
In order to interpolate the value of a variable, you need to enclose the string in double-quotes("), but the JSON also requires literal double-quotes.
The easiest solution is probably to use a here-document to feed the data into curl's standard input, as in #Gilles Quénot's answer. But you can still pass it in via the command line; you just have to be careful with the quotes.
This is one way:
curl ... -d '{
"name": "my-project",
"files": [
{
"data": "'"$stringVar"'"
}
]
}'
The JSON here is mostly contained within single quotation marks ('...'). But right after opening the pair of literal "s that will enclose the value of data, we close the single quotes and switch our shell quotation mode to double quotes in order to include the value of $stringVar. After closing the double quotes around that expansion, we go back into single quotes for the rest of the JSON, starting with closing the literal double-quotes around the value of data.
In a language that used + for string concatenation, it would look like '... "data": "' + "$stringVar" + '"... ', but in the shell you just put the strings next to each other with no operator to concatenate them.
As an alternative, you could put the whole thing in double-quotes, but then you need backslashes to include the literal double quotes that are part of the JSON:
curl ... -d "{
\"name\": \"my-project\",
\"files\": [
{
\"data\": \"$stringVar\"
}
]
}"
So that requires a lot more changes if you're starting from plain JSON; it also looks messier, IMO.
You can also use a tool that knows how to build JSON and let it worry about quoting etc. Here's one way to build it with jq:
jq -n --arg data "$stringVar" '{
"name": "my-project",
"files": [
{
"data": $data
}
]
}'
Using --arg creates a variable inside jq – I named it data – which can then be included in an expression with the syntax $varname ($data in this case). Despite the similarity of syntax, that's not a shell interpolation; we're passing the literal text $data to jq, and jq itself is what replaces it with the value of the variable (which was passed as the second argument to --arg).
There's another tool called jo, which doesn't manipulate JSON but rather produces it, from input that is easier to generate in the shell. Here's one way to construct the desired object with it:
jo name=my-project files="$(jo -a "$(jo data="$stringVar")")"
Either way you can include the constructed JSON in your curl command line like this:
curl ... -d "$(jo or jq command goes here)"
Do not generate such JSON by hand. Use a tool like jq to do it for you.
#!/bin/bash
stringVar="helloWorldFunction() {\n echo \"hello world\" \n}"
jq -n --arg s "$stringVar" '{name: "my-project", files: [{data: $s}]}' |
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-X POST \
-d #- \
'https://api.example.com/v1/endpoint'
Like this:
#!/bin/bash
stringVar="..."
curl -i -H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json" -X POST 'https://api.example.com/v1/endpoint' \
-d#/dev/stdin <<EOF
{
"name": "my-project",
"files": [
{
"data": $stringVar
},
],
}
EOF
You then should take care about what you fed in the variable, this have to be valid JSON
As an alternative to jq and curl you could use xidel to generate the JSON and submit the POST-request.
With command-line options:
#!/bin/bash
stringVar='helloWorldFunction() {\n echo "hello world" \n}'
xidel -s --variable var="$stringVar" \
-H "Authorization: Bearer <MY_API_TOKEN>" \
-H "Content-Type: application/json" \
-H "Accept: application/json"
-d '{serialize(
{"name":"my-project","files":array{{"data":$var}}},
{"method":"json"}
)}' \
"https://api.example.com/v1/endpoint" \
-e '$raw'
Or with the x:request() function in-query:
#!/bin/bash
stringVar='helloWorldFunction() {\n echo "hello world" \n}'
xidel -s --variable var="$stringVar" -e '
x:request({
"headers":(
"Authorization: Bearer <MY_API_TOKEN>",
"Content-Type: application/json",
"Accept: application/json"
),
"post":serialize(
{"name":"my-project","files":array{{"data":$var}}},
{"method":"json"}
),
"url":"https://api.example.com/v1/endpoint"
})/raw
'
$raw / /raw returns the raw output, like curl. If the API-endpoint returns JSON, then you can use $json / /json to parse it.
Edit: This question has been edited for context. 1.sh and 2.sh scripts in the original question are replaced by their real-world implementations: monitor.sh and api-check.sh respectively.
Function:
function checkAlive() {
script="$1"
args="${#:2}"
startCheck=$(date +%s)
echo "$#"
tempResult="$( "$#" )"
T_RC="$?"
if [ "$T_RC" -ne 0 ]; then
if [ "$result" = "" ]; then
result="ERRORS: $tempResult"
else
result="$result ,, $tempResult"
fi
fi
RC="$((RC > T_RC ? RC : T_RC))"
timePassed
}
api-check.sh:
method="$1"
url="https://$2"
jwt="$3"
body="$4"
echo "$body"
if ! response="$(curl -s --request "$method" "$url" -w "%{http_code}" --header \
'Authorization: Bearer '"$jwt" --header 'Content-Type: application/json' --header 'Accept: application/json' --data-raw "$body")"; then
echo "$url curl error getting code: $response"
exit 2
fi
monitor.sh:
api_url="foo"
oauth_token"bar"
body='{"address":"Test address"}'
checkAlive "api-check.sh" '"POST" "'$api_url'" "'$oauth_token'" '"'$body'"''
Execution Flow:
monitor.sh passes the string json body as an argument to checkAlive function (defined above).
Within checkAlive, the echo "$#" returns:
api-check.sh "POST" "foo" "bar" '{"address":"Test address"}'
The api-check.sh script never executes properly due to some escaping issue again
Question:
What json body should I pass as an argument to checkAlive to escape correctly OR Which line of code and in which script do I need to modify to evaluate the json body correctly?
do func() not function func()
just "$var". not anything else. Just make sure any $var is inside " ". Nothing more required.
Do:
api_url="foo"
oauth_token"bar"
body='{"address":"Test address"}'
checkAlive "api-check.sh" "POST" "$api_url" "$oauth_token" "$body"
I am giving the user the possibility to answers some questions so that I can perform the according cURL requests and create what is needed accordingly.
In this way:
~functions.sh
echo "Question number 1?"
read question1
echo "Question number 2?"
read question2
The script is organised in a couple of files:
functions.sh, where I store all the functions
flow.sh, where the questions are asked and the previous functions are called in order
credentials.sh, where I store the api key credentials, api url etc.
env.sh, where I am hoping to store the env configuration that needs to be dynamically generated...
Unfortunately I am pretty much new to that as I come from a Frontend Background so I Am not aware or can't figure out/find on here, how to achieve the following:
I would need to send a cURL request based on one of the users answers, and that's easily done just by calling ie {$question1} inside one of the functions of functions.sh that is later called in the main flow.sh after putting the file as a source...
Unfortunately, when it comes to populating a multiline environment configuration in a separate file to be later be sent as a json in the cURL it gets out of my comfort zone...
Id need so send something along the lines of
curl -X POST "https//api.url"
-H "headers"
--data '{"content":"'{$env}'"}'
Where $env is the env.sh file, json valid, and populated with the variables read before from user input accordingly...
To solve the multi line json problem maybe I could use the following:
$( awk '{printf "%s\\n", $0}' ./env.sh )
Still I am clueless on how to populate such file form user read and echo it inside that json part in the cURL...
You could use awk to directly save correct JSON data to env.sh (partly taken from Dr.K's answer to this other question):
echo "Question number 1?"
read question1
echo "Question number 2?"
read question2
echo "Question number 3?"
read question3
awk ' BEGIN { print "[" ; } \
{ print " {" \
print " \"Question1\" : \"" $question1 "\"," \
print " \"Question2\" : \"" $question2 "\"," \
print " \"Question3\" : \"" $question3 "\"" \
print " }" \
} \
END { print "]" } ' env.sh
And then this should work correctly as JSON:
curl -X POST "https//api.url"
-H "headers"
--data '{"content":"{$env}"}'
and the data payload should be like this:
{
content: [
{
"Question1" : "User answer 1",
"Question2" : "User answer 2",
"Question3" : "User answer 3"
}
]
}
Is this what you had in mind?
For anyone who ends up here what I did was:
Creating a separate file where I set a variable and I cat the EOF inside of that.
env=$( cat <<EOF
multi
line
content
goes
here
EOF
)
export env
Then I generate a valid json through jq
generate_json() {
newenv="$( jq -nc --arg str "$env" '{"content": $str}' )"
}
and finally I use $newenv in the curl straight away:
example_curl() {
curl -X PUT "https://apiendpoint.co.uk" \
-H "Authorization: Bearer 123123" \
--data "${newenv}"
}
I am trying to create a pull request comment automatically whenever CI is run. The output of a given command is written to a file (could also just be stored inside an environment variable though). The problem is, I usually get the following response:
curl -XPOST -d "{'body':'$RESULT'}" https://api.github.com/repo/name/issues/number/comment
{
"message": "Problems parsing JSON",
"documentation_url": "https://developer.github.com/v3/issues/comments/#create-a-comment"
}
This is usually due to unescpaed characters, like \n, \t, " etc.
Is there any easy way to achieve this on the command line or in bash, sh, with jq or Python? Using the Octokit.rb library is works straight away, but I don't want to install Ruby in the build environment.
You can use jq to create your JSON object. Supposing you have your comment content in RESULT variable, the full request would be :
DATA=$(echo '{}' | jq --arg val "$RESULT" '.| {"body": $val}')
curl -s -H 'Content-Type: application/json' \
-H 'Authorization: token YOUR_TOKEN' \
-d "$DATA" \
"https://api.github.com/repos/:owner/:repo/issues/:number/comments"
The post "Using curl POST with variables defined in bash script functions" proposes multiple techniques for passing a varialbe like $RESULT in a curl POST parameter.
generate_post_data()
{
cat <<EOF
{
"body": "$RESULT"
}
EOF
}
Then, following "A curl tutorial using GitHub's API ":
curl -X POST \
-H "authToken: <yourToken" \
-H "Content-Type: application/json" \
--data "$(generate_post_data)" https://api.github.com/repo/name/issues/number/comment
I am trying to invoke a curl command in powershell and pass some JSON information.
Here is my command:
curl -X POST -u username:password -H "Content-Type: application/json" -d "{ "fields": { "project": { "key": "key" }, "summary": "summary", "description": "description - here", "type": { "name": "Task" }}}"
I was getting globbing errors and "unmatched braces" and host could not be resolved, etc.
Then I tried prefixing the double quotes in the string with the backtick character, but it could not recognize the - character in the description json field
thanks
EDIT 1:
When I wrote the curl command in a regular batch file, I used double quotes and no single quotes. Also, in the -d string, I escaped all the double quotes with \ and the command worked.
In this case, my curl is actually pointing to curl.exe. I specified the path, just didn't list it here. Also I tried adding single quotes around -d and I got:
curl: option -: is unknown curl: try 'curl --help' or 'curl --manual' for more information
Seems like it cannot recognize the - character in the JSON
Pipe the data into curl.exe, instead of trying to escape it.
$data = #{
fields = #{
project = #{
key = "key"
}
summary = "summary"
description = "description - here"
type = #{
name = "Task"
}
}
}
$data | ConvertTo-Json -Compress | curl.exe -X POST -u username:password -H "Content-Type: application/json" -d "#-"
curl.exe reads stdin if you use #- as your data parameter.
P.S.: I strongly suggest you use a proper data structure and ConvertTo-Json, as shown, instead of building the JSON string manually.
Easy way (for simple testing):
curl -X POST -H "Content-Type: application/json" -d '{ \"field\": \"value\"}'