Sending json files in curl requests with absolute or relative paths - json

Just wondering how I can send a curl command with the -d option specifying a file with its path and not a file in the current directory.
This is what I'm getting when I try to test my app with the json file in the local dir. Both the app and myself are happy:
curl -XPOST -H 'Content-Type:application/json' -d #all_fields.json http://testcomp.lab.net:8080/stats -v -s
* About to connect() to testcomp.lab.net port 8080
* Trying 10.93.2.197... connected
* Connected to testcomp.lab.net (10.93.2.197) port 8080
> POST /stats HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: testcomp.lab.net:8080
> Accept: */*
> Content-Type:application/json
> Content-Length: 2882
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
HTTP/1.1 200 OK
< Content-Length: 0
* Connection #0 to host testcomp.lab.net left intact
* Closing connection #0
This is what I'm getting when I specify a json file that's in another directory
curl -XPOST -H 'Content-Type:application/json' -d #json/all_fields.json http://testcomp.lab.net:8080/stats -v -s
"Invalid json for Java type interface java.util.List"
Warning: Couldn't read data from file "json/all_fields.json", this makes an
Warning: empty POST.
<snip snip>
<snip snip>
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Transfer-Encoding: chunked
* Connection #0 to host testcomp.lab.net left intact
* Closing connection #0
I didn't see anything in the man page for curl for specifying directories for files passed in as data. Am I unfortunately limited to files in the local directory or is there a special way to specify files in different directories? Thanks in advance for your help.

The -d # command option accepts any resolvable file path, as long as the path actually exists. So you could use:
a path relative to the current directory
a fully qualified path
a path with soft-links in it
and so on
To wit, just the same as hundreds of other *Nix style commands. One quick note, the -d option will attempt to url encode your data, which from what you describe isn't actually what you want. You should use the --data-binary option instead. Something like this:
curl -XPOST
-H 'Content-Type:application/json'
-H 'Accept: application/json'
--data-binary #/full/path/to/test.json
http://localhost:8080/easy/eservices/echo -v -s

Related

Could not resolve host: dev-fiwr-svc-01.tid.es

Trying to upload some measurements to COSMOS. I have an COSMOS account.
Also, I'm using the tutorial here: https://github.com/ging/fiware-cosmos/blob/master/doc/manuals/quick_start_guide_new.md
I'm stuck on Step 3: Upload some data to HDFS.
This is the process flow for me:
curl -v -X PUT -T measurements.txt "http://storage.dit.upm.es:14000/webhdfs/v1/user/MY-USERNAME/MY-FOLDER/measurements.txt?op=CREATE&user.name=MY-USERNAME" -H "Content-Type: application/octet-stream" -H "X-Auth-token: MY-ACCESS-TOKEN"
After this request, I get the expected response according to the tutorial. But then on the next request...
curl -v -X PUT -T measurements.txt "http://dev-fiwr-svc-01.tid.es:14000/webhdfs/v1/user/MY-USERNAME/MY-FOLDER/measurements.txt?op=CREATE&user.name=MY-USERNAME&data=true" -H "Content-Type: application/octet-stream" -H "X-Auth-token: MY-ACCESS-TOKEN"
...I get the following response:
* Hostname was NOT found in DNS cache
* Could not resolve host: dev-fiwr-svc-01.tid.es
* Closing connection 0
curl: (6) Could not resolve host: dev-fiwr-svc-01.tid.es
Anybody else that has had the same problem?
Thanks to #Shihe Zhang for the idea!
Changed dev-fiwr-svc-01.tid.es:14000 to storage.dit.upm.es:14000 and my file was uploaded.

cURL and Error 411 (Content Length)

I'm using cURL to connect to a remote host. Some of my commands execute just fine (like authentication via POST and pulling information via GET). I'm getting a weird error when trying to use a POST to add information with regards to content length. This is my syntax;
curl -k -i -b sessionid -X POST https://hostname/BeyondTrust/api/public/v3/Workgroups/apiTest/Assets?AssetName=string&DnsName=string&DomainName=string -H "Content-Type: application/json" -H "Authorization: PS-Auth key=b8e1...2c27; runas=userName; pwd=[password]" -d ""
This is my output;
[1] 58164
[2] 58165
-bash: -H: command not found
[2]+ Done DnsName=string
HTTP/1.1 411 Length Required
Content-Type: text/html; charset=us-ascii
Server: Microsoft-HTTPAPI/2.0
Date: Fri, 27 Oct 2017 14:29:12 GMT
Connection: close
Content-Length: 344
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Length Required</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Length Required</h2>
<hr><p>HTTP Error 411. The request must be chunked or have a content length.</p>
</BODY></HTML>
I've researched it, and this is a common problem, apparently. -d "" does not fix and have tried --ignore-content-length and Content-Length: 0 in the headers, but I still have the same issue. Can someone point me in the right direction?
You need to quote the URL because it contains some special shell characters.
This output tells you something went wrong:
[1] 58164
[2] 58165
-bash: -H: command not found
[2]+ Done DnsName=string
Your command got split into multiple jobs and resulted in some errors because of the & in the URL's query string.
Putting the URL in double-quotes " will fix the issue.
curl -k -i -b sessionid \
-X POST \
"https://hostname/BeyondTrust/api/public/v3/Workgroups/apiTest/Assets?AssetName=string&DnsName=string&DomainName=string"
This way your URL will be interpreted properly and the shell won't try to start it in the background.

export neo4j data to JSON with cURL

curl -b -j "Content-Type: application/json" -d '["quer
y:"{"MATCH n RETURN n"}]' http://localhost:7474/db/data/
curl: (6) Could not resolve host: Content-Type
Hello,
i want get JSON from. With curl and write it to a local file.
So i can do other things with it.
But when executing this command i get could not resolve Content-Type.
i've tried:
curl -b -j Content-Type: application/json' without the double quotation mark ("")
but then i get this as result
curl: (6) Could not resolve host: Content-Type
curl: (6) Could not resolve host: application
Now i've fixed this syntax error
curl -b -j -d '["query:"{"MATCH n RETURN n"}]' http://
localhost:7474/db/data/ -o .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 28 0 0 100 28 0 13 0:00:02 0:00:02 --:--:-- 13
But no data:(
I had success with:
curl -b -j -H "Content-Type: application/json" -d '{"query":"match n return n limit 10"}}' http://localhost:7474/db/data/cypher

How to get cURL to output only HTTP response body (JSON) and no other headers etc

I am using curl in a bash script to fetch the response of a service as below,
response=$(curl -isb -H "Accept: application/json" "http://host:8080/some/resource")
Service response is of json type and on browser I could perfectly fine response.
However curl response has other unwanted things (such as set-cookie, content-length header in this case) and sometimes the actual response is eaten up.
Here is the output of echo $response >
Set-Cookie: rack.session=BAh7CEkiD3Nlc3Npb25faWQGOgZFVEkiRWJlY2JiOTE2M2Q1ZWI4NThjMDdi%0AYjRiOWRjMGMxMGEwYTBkMjE3NmJhZDVjYzY4YjY4ZTlmMTE2ZGVkYWE3MTMG%0AOwBGS
SIJY3NyZgY7AEZJIiVhZmQ2MmUyZGMxMzFmOGEwMjg3NDlhNWM3YmVm%0AN2FjNwY7AEZJIg10cmFja2luZwY7AEZ7B0kiFEhUVFBfVVNFUl9BR0VOVAY7%0AAFRJIi00MTc0OGM2MWNkMzljZTYxNzY3ZjU0
Y2I5OTdiYWRkN2MyNTBkYmU4%0ABjsARkkiGUhUVFBfQUNDRVBUX0xBTkdVQUdFBjsAVEkiLWRhMzlhM2VlNWU2%0AYjRiMGQzMjU1YmZlZjk1NjAxODkwYWZkODA3MDkGOwBG%0A--ee97a62095e7d42129
tontent-Length: 354c8; path=/; HttpOnly
This is breaking my response parsing logic.
I have seen this happening intermittently which is weird.
Is there a way to get "only" json response from cURL output?
I went through the curl documentation but could not see any thing/ or I could have missed it.
You are specifying the -i option:
-i, --include
(HTTP) Include the HTTP-header in the output. The HTTP-header includes things like server-name, date of the document, HTTP-version and more...
Simply remove that option from your command line:
response=$(curl -sb -H "Accept: application/json" "http://host:8080/some/resource")
I was executing a get request an also want to see just the response and nothing else, seems like magic is done with -silent,-s option.
From the curl man page:
-s, --silent
Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.
Below the examples:
curl -s "http://host:8080/some/resource"
curl --silent "http://host:8080/some/resource"
Using custom headers
curl -s -H "Accept: application/json" "http://host:8080/some/resource"
Using POST method with a header
curl -s -X POST -H "Content-Type: application/json" "http://host:8080/some/resource" -d '{ "myBean": {"property": "value"}}'
You can also customize the output for specific values with -w, below the options I use to get just response codes of the curl:
curl -s -o /dev/null -w "%{http_code}" "http://host:8080/some/resource"

How to escape characters in part of a JSON POST request with CURL?

So, I'm trying to use a service called Postmark to send a formatted HTML email. They have some API documentation here, and they give this example on how to use CURL:
$: curl -X POST "http://api.postmarkapp.com/email" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "X-Postmark-Server-Token: ed742D75-5a45-49b6-a0a1-5b9ec3dc9e5d" \
-v \
-d "{From: 'sender#example.com', To: 'receiver#example.com', Subject: 'Postmark test', HtmlBody: '<html><body><strong>Hello</strong> dear Postmark user.</body></html>'}"
So, that's all good and works just fine when I use my own token. The problem is when I add my own HtmlBody. If I send a simple message, it works just fine. As soon as I add certain special characters, it breaks. For example, if I do something like this:
-d "{From: 'sender#example.com', To: 'receiver#example.com', Subject: 'Postmark test', HtmlBody: '<!DOCTYPE html><html><body><strong>Hello</strong> dear Postmark user.</body></html>'}"
It breaks because of the !. How can I fix this?
UPDATE: As sourcejedi pointed out I am running this from the shell (bash), so the ! issue makes sense to me now. I moved the JSON string to a separate file called email.json and loaded that using -d #email.json. That worked for a simple email with <!DOCTYPE>, but I'm still getting the following error when I try to load the full HTML:
{"ErrorCode":402,"Message":"Received invalid JSON input."}
I believe this has to do with some other special characters. I get the same error when I use --data-urlencode #email.json.
You need to use --data-urlencode, so it should be like curl -X POST --data-urlencode
When I tested on my end..
* About to connect() to api.postmarkapp.com port 80 (#0)
* Trying 50.56.54.211... connected
> POST /email HTTP/1.1
> User-Agent: curl/7.23.1 (x86_64-pc-win32) libcurl/7.23.1 OpenSSL/0.9.8r zlib/1
.2.5
> Host: api.postmarkapp.com
> Accept: application/json
> Content-Type: application/json
> X-Postmark-Server-Token: ed742D75-5a45-49b6-a0a1-5b9ec3dc9e5d
> Content-Length: 164
Still seems to break on me an just returns: -bash: !DOCTYPE: event not found
That's an error from the bash shell. You would need to escape the ! as \! yourself. But you'd be better off reading from a file, I think. Instead of -d data use -d #datafile. (Or -d #- to read from stdin).
I once received the same error when posting JSON data. what I did was enclosed the data with a single quote and use double quote for a string:
so from:
-d "{From: 'sender#example.com', To: 'receiver#example.com', Subject: 'Postmark test', HtmlBody: '<html><body><strong>Hello</strong> dear Postmark user.</body></html>'}"
use:
-d '{From: "sender#example.com", To: "receiver#example.com", Subject: "Postmark test", HtmlBody: "<html><body><strong>Hello</strong> dear Postmark user.</body></html>"}'
hope this helps.