curl error when using url string from string - json

I'm now testing some restful APIs using bash shell script.
I want to read url from file then make a json data string with the url in file.
For the test, below codes work fine. It's not reading from file.
#!/bin/bash
URL=http://test.com/test.jpg
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$URL"'"}' \
http://api.test.com/test
But, it returns some error when I'm using the codes like below.
#!/bin/bash
FILE=./url.txt
cat $FILE | while read line; do
echo $line # or whaterver you want to do with the $line variable
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$line"'"}' \
http://api.test.com/test
done
But, it returns error when I use the string from reading file.
This is error message.
Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in string value
at [Source: org.apache.catalina.connector.CoyoteInputStream#27eb679c; line: 1, column: 237]
How to solve this issue?
Why it returns error when I use the string from file reading?

It appears that your file is in dos format with \n\r line terminators. Try running dos2unix on it to strip the \rs. Additionally, no need to cat the file, use redirection, like so
while read -r line; do
echo $line # or whaterver you want to do with the $line variable
curl -X POST \
-H "Content-Type:application/json" \
-H "accept:application/json" \
--data '{"url":"'"$line"'"}' \
http://api.test.com/test
done < "$FILE"
Also, pass -r to read to prevent backslash escaping

Related

CURL ERROR: parse error: Invalid numeric literal at line 1, column 8

I'm doing a bash script, when working out such an error ... I can't figure out what the problem is. I'm not good at scripting yet.
#!/bin/bash
data=LOGIN
password=123PASSWD
note_link=$(curl -s 'https://cryptgeon.nicco.io' \
-H 'X-Requested-With: XMLHttpRequest' \
--data-urlencode "data=$data" \
--data "has_manual_pass=false&duration_hours=0&dont_ask=false&data_type=T&notify_email=&notify_ref=" \
| jq -r --arg arg $password '.note_link + "#" + $arg')
echo "note URL is $note_link"
curl's -s option is silencing the errors as well, but you want to see the errors in this case to be able to understand what is going wrong, so use -sS instead.
Also, jq can only parse json. If the input is not json, it will fail with the error you get. You should first try to parse the output with jq, and if it fails, display it.
#!/bin/bash
data=LOGIN
password=123PASSWD
curl_output=$(curl -sS 'https://cryptgeon.nicco.io' \
-H 'X-Requested-With: XMLHttpRequest' \
--data-urlencode "data=$data" \
--data "has_manual_pass=false&duration_hours=0&dont_ask=false&data_type=T&notify_email=&notify_ref=")
if note_link=$(jq -r --arg pass "$password" '.note_link + "#" + $pass' <<<"$curl_output" 2>/dev/null); then
echo "note URL is $note_link"
else
printf >&2 %s\\n "Could not parse the curl output:" "$curl_output"
fi

Convert log files to base64 and upload it using Curl to Github

I am trying to upload a file to Github using its API.
Following code works, but only with smaller size content which is approx less than 1MB.
tar -czvf logs.tar.gz a.log b.log
base64_logs=$(base64 logs.tar.gz | tr -d \\n)
content_response=$(curl \
-X PUT \
-u :"$GIT_TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
"$content_url" \
-d '{"message": "Log files", "content": "'"$base64_logs"'"}')
For content that is a bit large, I get the following error:
/usr/bin/curl: Argument list too long
Now, there is already a question on SO about this error message, and it says that to upload a file directly. See here: curl: argument list too long
When I try this, I get a problem parsing JSON error message.
tar -czvf logs.tar.gz a.log b.log
base64_logs=$( base64 logs.tar.gz | tr -d \\ ) > base64_logs.txt
content_response=$(curl \
-X PUT \
-u :"$GIT_TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
"$content_url" \
-d '{"message": "Log files", "content": #base64_logs.txt}')
Can anyone point me out where I am making mistake here? Thanks!
Use the base64 command rather than the #base64 filter from jq, because the later can only encode textual data and not binary data as from a .gz archive.
Pipe the base64 stream to jq to format it into a JSON data stream.
Curl will read the JSON data stream and send it.
# use base64 to encode binary data and -w0 all in one-line stream
base64 --wrap=0 logs.tar.gz |
# JSON format from raw input
jq \
--raw-input \
--compact-output \
'{"message": "Log files", "content": . }' |
# Pipe JSON to curl
curl \
--request PUT \
--user ":$GIT_TOKEN" \
--header "Accept: application/vnd.github.v3+json" \
--header 'Content-Type: application/json' \
--data-binary #- \
--url "$content_url"

Shell script not working for curl request

I want to make a curl request with Databox to push somemetrix and want to do it in shell script.
Here is the databox POST request example (which works like a charm)
curl https://push.test \
-u token
: \
-X POST \
-H 'Content-Type: application/json' \
-H 'Accept: application/vnd.databox.v2+json' \
-d '{
"data":[
{
"$testcount": 50,
"test_name": "test"
}
]
}'
When I form the json body as a separate json string and try to pass as parameter, it doesn't work and gives a json parsing error. I am not sure what am I am doing wrong here. can someone help? I am new to shell scripts
#!/bin/bash
JSON_STRING= '{"data" : [{"$testcount":50,"testname":"test"}]}'
echo "$JSON_STRING"
curl https://testpush \
-u token
: \
-X POST \
-H 'Content-Type: application/json' \
-H 'Accept: application/vnd.databox.v2+json' \
-d '$JSON_STRING'
error :
{"type":"invalid_json","message":"Invalid request body - JSON parse error"}
I have added my token for the request, so the authorisation should work.
You have excess whitespace around the =.
Also, $JSON_STRING in the last line of the second script should be in double quotes instead of the single quotes, to get it expanded into what you just set it to.
Btw., if data gets out of hand or is sensitive, you might want to look into the possibility to start the data with the letter # and then have the rest be a file name that contains the data.

Passing variables (containing spaces) to curl --data field

Arguments containing spaces will not properly pass to the curl command.
Quotes are not passed correctly in the --data field.
If I just echo the variable 'curlData' that I use in the curl command I get everything as it should be; ex :
$echo $curlData
'{"name":"jason","description","service"}'
I don't understand why curl dont expend this 'curlData' variable as expected:
curl --data '{"name":"jason","description","service"}'
Here's a sample of my code:
read -p "Name : " repoName
read -p "Description []: " repoDescription
curlData="'"{'"'name'"':'"'$repoName'"','"'descripton'"':'"'$repoDescription'"'}"'"
curl --data $curlData $apiURL
And the error:
curl: (3) [globbing] unmatched close brace/bracket in column 26
Thank your for your help, I feel i'm in Quote-ception right now.
Quote all variable expansions,
To make sure that curlData is a valid JSON value with properly escaped special-characters etc., use jq for producing it.
curlData="$(jq --arg name "$repoName" --arg desc "$repoDescription" -nc '{name:$name,description:$desc}')"
curl --data "$curlData" "$apiURL"
If you have access to any form of package management, I highly recommend jo.
curlData=$(jo name="$repoName" description="$repoDescription")
curl -d "$curlData" "$apiURL"
I had similar issue, which was very difficult to even understand. I used the below construct in a number of curl commands present in my shell script. It always worked like a charm. Until one fine day I had to pass a variable which was string containing spaces (eg. modelName="Abc def").
curl -X 'PUT' \
'http://localhost:43124/api/v1/devices/'$Id'' \
-H 'accept:*/*' \
-H 'Authorization:Bearer '$token'' \
-H 'Content-Type:application/json' \
-d '{
"modelName":"'$modelName'",
"serialNumber":"'$childSN'"
}'
Worked for me after the below change
curl -X 'PUT' \
'http://localhost:43124/api/v1/devices/'$Id'' \
-H 'accept:*/*' \
-H 'Authorization:Bearer '$token'' \
-H 'Content-Type:application/json' \
-d '{
"modelName":'\""$modelName"\"',
"serialNumber":"'$childSN'"
}'
I took help from the accepted answer by #oguz. Pasting this response , just in case anyone is in similar situation

Passing a shell variable to a JSON request to curl?

Let's take the following example:
curl -i -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":false}}' \
http://example.com/jsonrpc
Now I want to have the boolean value of "item" be set in a shell script variable such as:
PRIVATE=false
read -p "Is this a private? (y/[n]) " -n 1 -r
if [[ $REPLY =~ ^[Yy]$ ]]; then
PRIVATE=true
fi
And I want to pass in the value of PRIVATE to item. I have tried every which way but no luck. Can anyone shed some light?
You can do it this way:
curl -i -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "method": "Player.Open", "params":{"item":'"$PRIVATE"'}}' \
http://example.com/jsonrpc
Instead of your existing -d ... line above, you could try the following:
-d "{\"jsonrpc\": \"2.0\", \"method\": \"Player.Open\", \"params\":{\"item\":$PRIVATE}}" \
That is: when using double quote speechmarks ("), bash substitutes values for variables referenced $LIKE_THIS (not the case for single quotes you were using). The downside is that you then need to escape any double-quotes in the string itself (using the backslash, as above).
This abomination works too.
$ npm run script -- madrid
# script
json='{"city":"'"$1"'"}'
curl -X POST -d $json http://localhost:5678/api/v1/weather