Perl shell curl execution handle json string - json

Just curious to know the difference between below three executions, If I use a variable to hold the json data and pass the same to curl command, json msg become corrupt and server returns incorrect result, What is the best way to pass json from a variable while invoking curl command ? Trying to make the last line in the snippet work.
my $tsFormatted = '2021-06-10';
my $metaLoad = '{"validDate": "'.$tsFormatted.'"}';
my $curl_out_1 = `curl -s -X POST -H 'Content-Type: application/json' $mw_meta_load_api --key $mw_access_key --cert $mw_access_cert --data '{"validDate": "2021-06-10"}'`;
my $curl_out_2 = `curl -s -X POST -H 'Content-Type: application/json' $mw_meta_load_api --key $mw_access_key --cert $mw_access_cert --data '{"validDate": "'.$tsFormatted.'"}'`;
my $curl_out_3 = `curl -s -X POST -H 'Content-Type: application/json' $mw_meta_load_api --key $mw_access_key --cert $mw_access_cert --data $metaLoad`;

Backticks interpolate as double-quoted strings, so we can easily view the difference by using double-quoted string literals instead of backticks.
my $tsFormatted = '2021-06-10';
my $metaLoad = '{"validDate": "'.$tsFormatted.'"}';
# qq`...`, qq"..." and "..." are all the same thing.
say qq`... --data '{"validDate": "2021-06-10"}'`;
say qq`... --data '{"validDate": "'.$tsFormatted.'"}'`;
say qq`... --data $metaLoad`;
Output:
... --data '{"validDate": "2021-06-10"}'
... --data '{"validDate": "'.2021-06-10.'"}'
... --data {"validDate": "2021-06-10"}
The contents of string literals (including backticks) aren't treated as Perl code.
Of those three, only the first is correct.
The first passes the following two arguments to curl:
--data
{"validDate": "2021-06-10"}
The second passes the following two arguments to curl:
--data
{"validDate": ".2021-06-10."}
The third passes the following three arguments to curl:
--data
{validDate:
2021-06-10}
What you want:
use Cpanel::JSON::XS qw( encode_json );
use String::ShellQuote qw( shell_quote );
my $ts_formatted = '2021-06-10';
my $data = { validDate => $ts_formatted };
my $data_formatted = encode_json($data);
my $cmd = shell_quote(
curl => (
$mw_meta_load_api,
'-s',
-X => 'POST',
-H => 'Content-Type: application/json',
'--key' => $mw_access_key,
'--cert' => $mw_access_cert,
'--data' => $data_formatted,
)
);
my $out = `$cmd`;
die("Can't execute shell: $!\n") if $? == -1;
die("curl killed by signal ".( $? & 0x7F )."\n") if $? & 0x7F;
die("curl exited with error ".( $? >> 8 )."\n") if $? >> 8;
Also, have you considered using Net::Curl::Easy instead of shelling out?

Related

Shell script whitespace escaping

Edit: This question has been edited for context. 1.sh and 2.sh scripts in the original question are replaced by their real-world implementations: monitor.sh and api-check.sh respectively.
Function:
function checkAlive() {
script="$1"
args="${#:2}"
startCheck=$(date +%s)
echo "$#"
tempResult="$( "$#" )"
T_RC="$?"
if [ "$T_RC" -ne 0 ]; then
if [ "$result" = "" ]; then
result="ERRORS: $tempResult"
else
result="$result ,, $tempResult"
fi
fi
RC="$((RC > T_RC ? RC : T_RC))"
timePassed
}
api-check.sh:
method="$1"
url="https://$2"
jwt="$3"
body="$4"
echo "$body"
if ! response="$(curl -s --request "$method" "$url" -w "%{http_code}" --header \
'Authorization: Bearer '"$jwt" --header 'Content-Type: application/json' --header 'Accept: application/json' --data-raw "$body")"; then
echo "$url curl error getting code: $response"
exit 2
fi
monitor.sh:
api_url="foo"
oauth_token"bar"
body='{"address":"Test address"}'
checkAlive "api-check.sh" '"POST" "'$api_url'" "'$oauth_token'" '"'$body'"''
Execution Flow:
monitor.sh passes the string json body as an argument to checkAlive function (defined above).
Within checkAlive, the echo "$#" returns:
api-check.sh "POST" "foo" "bar" '{"address":"Test address"}'
The api-check.sh script never executes properly due to some escaping issue again
Question:
What json body should I pass as an argument to checkAlive to escape correctly OR Which line of code and in which script do I need to modify to evaluate the json body correctly?
do func() not function func()
just "$var". not anything else. Just make sure any $var is inside " ". Nothing more required.
Do:
api_url="foo"
oauth_token"bar"
body='{"address":"Test address"}'
checkAlive "api-check.sh" "POST" "$api_url" "$oauth_token" "$body"

CURL ERROR: parse error: Invalid numeric literal at line 1, column 8

I'm doing a bash script, when working out such an error ... I can't figure out what the problem is. I'm not good at scripting yet.
#!/bin/bash
data=LOGIN
password=123PASSWD
note_link=$(curl -s 'https://cryptgeon.nicco.io' \
-H 'X-Requested-With: XMLHttpRequest' \
--data-urlencode "data=$data" \
--data "has_manual_pass=false&duration_hours=0&dont_ask=false&data_type=T&notify_email=&notify_ref=" \
| jq -r --arg arg $password '.note_link + "#" + $arg')
echo "note URL is $note_link"
curl's -s option is silencing the errors as well, but you want to see the errors in this case to be able to understand what is going wrong, so use -sS instead.
Also, jq can only parse json. If the input is not json, it will fail with the error you get. You should first try to parse the output with jq, and if it fails, display it.
#!/bin/bash
data=LOGIN
password=123PASSWD
curl_output=$(curl -sS 'https://cryptgeon.nicco.io' \
-H 'X-Requested-With: XMLHttpRequest' \
--data-urlencode "data=$data" \
--data "has_manual_pass=false&duration_hours=0&dont_ask=false&data_type=T&notify_email=&notify_ref=")
if note_link=$(jq -r --arg pass "$password" '.note_link + "#" + $pass' <<<"$curl_output" 2>/dev/null); then
echo "note URL is $note_link"
else
printf >&2 %s\\n "Could not parse the curl output:" "$curl_output"
fi

Shell script not working for curl request

I want to make a curl request with Databox to push somemetrix and want to do it in shell script.
Here is the databox POST request example (which works like a charm)
curl https://push.test \
-u token
: \
-X POST \
-H 'Content-Type: application/json' \
-H 'Accept: application/vnd.databox.v2+json' \
-d '{
"data":[
{
"$testcount": 50,
"test_name": "test"
}
]
}'
When I form the json body as a separate json string and try to pass as parameter, it doesn't work and gives a json parsing error. I am not sure what am I am doing wrong here. can someone help? I am new to shell scripts
#!/bin/bash
JSON_STRING= '{"data" : [{"$testcount":50,"testname":"test"}]}'
echo "$JSON_STRING"
curl https://testpush \
-u token
: \
-X POST \
-H 'Content-Type: application/json' \
-H 'Accept: application/vnd.databox.v2+json' \
-d '$JSON_STRING'
error :
{"type":"invalid_json","message":"Invalid request body - JSON parse error"}
I have added my token for the request, so the authorisation should work.
You have excess whitespace around the =.
Also, $JSON_STRING in the last line of the second script should be in double quotes instead of the single quotes, to get it expanded into what you just set it to.
Btw., if data gets out of hand or is sensitive, you might want to look into the possibility to start the data with the letter # and then have the rest be a file name that contains the data.

Passing a shell variable to a JSON request to curl?

Let's take the following example:
curl -i -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":false}}' \
http://example.com/jsonrpc
Now I want to have the boolean value of "item" be set in a shell script variable such as:
PRIVATE=false
read -p "Is this a private? (y/[n]) " -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]; then
PRIVATE=true
fi
And I want to pass in the value of PRIVATE to item. I have tried every which way but no luck. Can anyone shed some light?
You can do it this way:
curl -i -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":'"$PRIVATE"'}}' \
http://example.com/jsonrpc
Instead of your existing -d ... line above, you could try the following:
-d "{\"jsonrpc\": \"2.0\", \"method\": \"Player.Open\", \"params\":{\"item\":$PRIVATE}}" \
That is: when using double quote speechmarks ("), bash substitutes values for variables referenced $LIKE_THIS (not the case for single quotes you were using). The downside is that you then need to escape any double-quotes in the string itself (using the backslash, as above).
This abomination works too.
$ npm run script -- madrid
# script
json='{"city":"'"$1"'"}'
curl -X POST -d $json http://localhost:5678/api/v1/weather

curl error when using url string from string

I'm now testing some restful APIs using bash shell script.
I want to read url from file then make a json data string with the url in file.
For the test, below codes work fine. It's not reading from file.
#!/bin/bash
URL=http://test.com/test.jpg
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$URL"'"}' \
http://api.test.com/test
But, it returns some error when I'm using the codes like below.
#!/bin/bash
FILE=./url.txt
cat $FILE | while read line; do
echo $line # or whaterver you want to do with the $line variable
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$line"'"}' \
http://api.test.com/test
done
But, it returns error when I use the string from reading file.
This is error message.
Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in string value
at [Source: org.apache.catalina.connector.CoyoteInputStream#27eb679c; line: 1, column: 237]
How to solve this issue?
Why it returns error when I use the string from file reading?
It appears that your file is in dos format with \n\r line terminators. Try running dos2unix on it to strip the \rs. Additionally, no need to cat the file, use redirection, like so
while read -r line; do
echo $line # or whaterver you want to do with the $line variable
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$line"'"}' \
http://api.test.com/test
done < "$FILE"
Also, pass -r to read to prevent backslash escaping