I have to check the json data from an url using curl
curl -H "User-agent: 'your bot 0.1'" url.json | jq
this code is working
i wanted try this for a .txt file containing 200 url
like these
https://www.reddit.com/user/wanderer_007_.json
https://www.reddit.com/....
https://www.reddit.com/....
https://www.reddit.com/....
https://www.reddit.com/....
these are just examples. but whenever I give the text file as an input
#!/usr/bin/bash
while read -r line; do
name="$line"
curl -H "User-agent: 'your bot 0.1'" $name | jq
done < test001.txt
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
curl: (3) URL using bad/illegal format or missing URL
but if I try to use the url individually it is working as intended
Try to wrap the filename in curl command with double quotes:
curl -H "User-agent: 'your bot 0.1'" "$name" | jq
The error may be due to Bash interpreting some wierd characters in URLs specially. For example space in the URL may cause $name to split into 2 command-line arguments making curl unable to parse its inputs.
Related
I want to execute a bash script after i make a POST request.So far i am using Postman for sending the request , but i was wondering if i can somehow do it from a bash script as well with a json file as parameter.
I have looked into curl so far but it does not work:
bash file
curl -X POST -d req.json http://localhost:9500
Json file (req.json)
{
"id":5,
"name":"Dan",
"age":33,
"cnp":33,
"children":100,
"isMarried":0
}
I just get the error :
HTTP/1.0 503 Service Unavailable
with the trailing HTML
curl should do the job. This will send a normal POST request using the data in req.json as the body:
curl -X POST -H "Content-Type: application/json" -d #req.json http://localhost:9500
The elements you were missing are -H "Content-Type: application/json" and the # in the data flag. Without the -H flag as above curl will send a content type of application/x-www-form-urlencoded, which most applications won't accept if they expect JSON. The # in the -d flag informs curl that you are passing a file name; otherwise it uses the text itself (i.e. "req.json") as the data.
When i give the URL (http://192.168.150.41:8080/filereport/31779/json/) in browser, It automatically downloads the file as 31779_report.json.
Now using i'm trying to download the file using curl but i get the following error.
$ curl -O http://192.168.150.41:8080/filereport/31779/json/
curl: Remote file name has no length!
curl: try 'curl --help' or 'curl --manual' for more information
When using the '-L' switch , I get the JSON content displayed but the file is not saved.
$curl -L http://192.168.150.41:8080/filereport/31779/json/
{
.....
.....
}
How to download the exact file "31779_report.json" using cURL / wget ?
I don't want the contents to be redirected (>) manually to a file (31779_report.json).
Any suggestions please ?
The -O flag of curl tries to use the remote name of the file, but because your URL schema does not end with a filename, it can not do this. The -o flag (lower-case o) can be used to specify a file name manually without redirecting STDOUT like so:
curl <address> -o filename.json
You can manually construct the filename format you want using awk. For example:
URL=http://192.168.150.41:8080/filereport/31779/json/
file_number=$(echo $URL | awk -F/ '{print $(NF-2)}')
file_name="${file_number}_report.json"
curl -L "$URL" -o "$file_name"
Hope this is more helpful.
wget --content-disposition did the trick for me (https://askubuntu.com/a/77713/18665)
$ wget --content-disposition https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
...
Saving to: 'lib32-glibc-2.33-4-x86_64.pkg.tar.zst'
Compare to curl:
$ curl -LO https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
curl: Remote file name has no length!
curl: (23) Failed writing received data to disk/application
And wget without --content-disposition:
$ wget https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
...
Saving to: 'index.html'
I have a curl which sends application/json data .when i type this url directly through unix console,it works fine.But when i take this url and store it in a csv file and through shell script try accessing this file and read each curl and esecute it through backticks i am facing two problems
1) it is not allowing spaces in the json data being posted
2) the content type is not being set
Please find below the same url
curl -i -X PUT -H 'content-type:application/json' -H "Accept:application/json" -d '{"startTime":1426172400000,"endTime":1426173300000,"attributes":{"title":"X X X","link":"https://someurl.com}}' http://10.10.7.90:9084/myapp/rest/app/706128.api`
I found the issue .It was in my shell script .Insead of using the back tick '`' to execute the curl ,i used eval curl .which solved the issue
Curl is refusing to take PUT request data in the url-encoded form, or any form other than JSON.
I am using the curl command on OS X.
Here is the relevant part of the command that i am running:
curl <other options> -H 'Content-Type: application/x-www-form-urlen-coded' --data-urlencode '_id=postcodes-backup-monday'
Anything i try is met with blank refusal and the following error message:
"{"error":"bad_request","reason":"invalid UTF-8 JSON"}"
My response to this is: well of course this is valid JSON - i am sending the data in url-encoded form ...
The command also refuses to parse valid JSON as valid JSON if i try that too.
The data you send is not a JSON.
You should
curl <other options> -H 'Content-Type: application/x-www-form-urlen-coded' \
--data-urlencode '{"_id":"postcodes-backup-monday"}'
I have uploaded a file into my box account and now when I try to create a shared link for it using cURL, I am getting errors.
curl https://api.box.com/2.0/files/FILE_ID -H
"Authorization: Bearer ACCESS_TOKEN" -d '{"shared_link": {"access":
"open"}}' -X PUT
The error is as follows:
{"type":"error","status":400,"code":"bad_request","context_info":{"errors":[{"reason":"invalid_parameter","name":"entity-body","message":"Invalid
value '{shared_link:'. Entity body should be a correctly nested
resource attribute name/value
pair"}]},"help_url":"http://developers.box.com/docs/#errors","message":"Bad
Request","request_id":"144849506852f47c 002e83b"}curl: (3) [globbing]
unmatched brace at pos 9 curl: (3) [globbing] unmatched close
brace/bracket at pos 5
Any thoughts or suggestions on what I am doing wrong?
You are posting the JSON data using curl. So add this header to let the server know you are sending JSON.
-H "Content-Type: application/json"
Also, if you are using from windows, then use double quote(") around the JSON. For example
-d "{test:\"value\"}"
If you have problem with secure connection(https) then use this parameter as well.
-k