Special Characters ÆØÅ not received correct with curl - json

I am sending from curl to a service with REST API and I don’t understand why the server is not receiving special characters ÆØÅ correct.
When I am doing it from curl in CMD, the server is receiving the Danish characters ÆØÅ incorrect.
When I am using the browser, through StrongLoop API, the characters are correctly inserted.
As far as I have been able to research, I need the charset defined. But this have no effect. Using chcp 65001 and Lucida Console font does not help. I am a bit lost identifying the problem.
This is the command used:
curl -X POST -H "Content-Type: application/json; charset=UTF-8" -H "Accept: application/json" -d "{ \"metadata\": { \"user\": \"aa\", \"version\": \"ED\", \"transaction_date\": \"2016-12-30\", \"context_info\": \"string\" }, \"container\": { \"type\": \"SC\", \"type_source\": \"string\", \"op_status_source\": \"string\", \"identification\": \"øæå\", \"id_source\": \"876332d2321\", \"ver\": 0 }}" "http://xx.com/xx/containers?access_token=xxx
The response:
{"type":"SC","type_source":"string","op_status_source":"string","identification":"���","id_source":"8763wwd32d2321","ver":1}
æøå will either be ∩┐╜∩┐╜∩┐╜", ���, or similar, depending on the chcp.
Also note, the console will display æøå correctly, but not the response.

I managed to get this to work by putting the json with Danish characters in its own file, called test.json. I made sure - and this was they key - that the json file was saved as utf-8: I did this by opening the file with notepad and used "Save As", picking utf-8 encoding as file encoding.
At the top of the bat file containing the curl command, I put chcp 65001.
curl.exe -X PUT --header "Content-Type: application/json; charset=UTF-8" --header "Accept: application/json" -s -d #test.json "<theUrl>"
I added -s to make curl go quiet about an error I was getting (Failed writing body), although the command (which updates a record in a database) appears to run successfully.

Related

Sending POST Request from bash script

I want to execute a bash script after i make a POST request.So far i am using Postman for sending the request , but i was wondering if i can somehow do it from a bash script as well with a json file as parameter.
I have looked into curl so far but it does not work:
bash file
curl -X POST -d req.json http://localhost:9500
Json file (req.json)
{
"id":5,
"name":"Dan",
"age":33,
"cnp":33,
"children":100,
"isMarried":0
}
I just get the error :
HTTP/1.0 503 Service Unavailable
with the trailing HTML
curl should do the job. This will send a normal POST request using the data in req.json as the body:
curl -X POST -H "Content-Type: application/json" -d #req.json http://localhost:9500
The elements you were missing are -H "Content-Type: application/json" and the # in the data flag. Without the -H flag as above curl will send a content type of application/x-www-form-urlencoded, which most applications won't accept if they expect JSON. The # in the -d flag informs curl that you are passing a file name; otherwise it uses the text itself (i.e. "req.json") as the data.

Syntax of the JSON body in CURL call to a REST API

When I try to send a curl req from terminal on OSX as:
curl --anyauth --user usr:pwd -X PUT -d ‘{"events":{"event":"Database Replicate"}}’ -i -H "Content-type: application/json" URL
it gives me below error:
{"errorResponse":{"statusCode":"400", "status":"Bad Request", "messageCode":"XDMP-JSONDOC", "message":"XDMP-JSONDOC: xdmp:unquote(\"‘{events:{event:Database Replicate}}’\", (), \"format-json\") -- Document is not JSON"}}
I tried to escape the quotes, ensured its UTF-8, still doesn't work. When I simply save it in a file and then attach as -d#filename.json to the CURL req, it just works.
Any idea if I am missing something? Thanks in advance.
Typing the request directly on terminal solved the issue. Do not trust on any 3rd party app when it comes to encoding.

parse.com returns "unexpected token %" for curl command that contain accented characters

I am trying to save a object to my parse server through curl commands.
the curl command that i am using is:
curl -X POST \
-H "X-Parse-Application-Id: XXXX" \
-H "X-Parse-Master-Key: XXXXX" \
-H "Content-Type: application/json; charset=utf-8 " \
--data-urlencode " {\"name\": \"Víkingur Reykjavík\"} " \
http://127.0.0.1:1337/parse/classes/Test
But i keep getting this as a response:
{"error":"Unexpected token %"}
I found this weird because i don't have a "%" in my name's.
So i tried to look it up through a proxy and found that the json text that is send is:
%20%7B%22name%22%3A%20%22V%EDkingur%20Reykjav%EDk%22%7D%20
i am using --data-urlencode which decodes it if i don't use it the object is saved but all the characters with accents will be replaced with a "�".
How should i change the curl so that i will be able to save objects with accents correctly?
JSON data payloads can't be URL-encoded since your content type says it's application/json in utf-8 encoding.
The string should look more like this:
{"name":"V\u00edkingur Reykjav\u00edk"}
Depending on what tools and languages you have available, you can encode data to JSON in various ways.

Why do I get a malformed JSON in request body in this cURL call?

I have been trying to call the CloudFlare API v4, using an example provided in their own documentation.
This is the code of the example
curl -X PUT "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records/372e67954025e0ba6aaa6d586b9e0b59" \ -H "X-Auth-Email: user#example.com" \ -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \ -H "Content-Type: application/json" \ --data '{"id":"372e67954025e0ba6aaa6d586b9e0b59","type":"A","name":"example.com","content":"1.2.3.4","proxiable":true,"proxied":false,"ttl":120,"locked":false,"zone_id":"023e105f4ecef8ad9ca31a8372d0c353","zone_name":"example.com","created_on":"2014-01-01T05:20:00.12345Z","modified_on":"2014-01-01T05:20:00.12345Z","data":{}}'
Which can also be found at
Update DNS Records
Using Windows cmd.exe to run this command, I need to make it single line first, so I removed the "" and reformatted it (twice) making sure I altered no part in the process.
This is the same code in one line:
curl -X PUT "https://api.cloudflare.com/client/v4/zones/023e105f4ecef8ad9ca31a8372d0c353/dns_records/372e67954025e0ba6aaa6d586b9e0b59" -H "X-Auth-Email: user#example.com" -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" -H "Content-Type: application/json" --data '{"id":"372e67954025e0ba6aaa6d586b9e0b59","type":"A","name":"example.com","content":"1.2.3.4","proxiable":true,"proxied":false,"ttl":120,"locked":false,"zone_id":"023e105f4ecef8ad9ca31a8372d0c353","zone_name":"example.com","created_on":"2014-01-01T05:20:00.12345Z","modified_on":"2014-01-01T05:20:00.12345Z","data":{}}'
When I run this single-liner in cmd, it works but I get a malformed JSON in request body, however, a visual check, formatting on Notepad++ and a run through the JSON validator are all positive, this JSON (copied from the CloudFlare documentation) is not malformed.
Error Message
{"success":false,"errors":[{"code":6007,"message":"Malformed JSON in request body"}],"messages":[],"result":null}
Googling this error message or the error code gives me nothing and this same command works on a PC running Linux.
Can someone tell me if this is a known bug, if the JSON really is malformed or if something else comes to mind?
I found the answer in the blog post: "Expecting to find valid JSON in request body..." curl for Windows.
For example, for Purge everything --data value will be:
# On Linux
--data '{"purge_everything":true}'
# On Windows
--data "{\"purge_everything\":true}"
For Windows:
Replace the single quotes with double quotes: ' --> "
Escape the double quotes with a backslash: " --> \"
cmd.exe doesn't support single quotes, to run those commands straight from the docs you can use Bash.
Bash can be enabled in Windows 10 : https://www.laptopmag.com/uk/articles/use-bash-shell-windows-10
or Git Bash comes with Git for windows: https://gitforwindows.org/

How to get cURL to output only HTTP response body (JSON) and no other headers etc

I am using curl in a bash script to fetch the response of a service as below,
response=$(curl -isb -H "Accept: application/json" "http://host:8080/some/resource")
Service response is of json type and on browser I could perfectly fine response.
However curl response has other unwanted things (such as set-cookie, content-length header in this case) and sometimes the actual response is eaten up.
Here is the output of echo $response >
Set-Cookie: rack.session=BAh7CEkiD3Nlc3Npb25faWQGOgZFVEkiRWJlY2JiOTE2M2Q1ZWI4NThjMDdi%0AYjRiOWRjMGMxMGEwYTBkMjE3NmJhZDVjYzY4YjY4ZTlmMTE2ZGVkYWE3MTMG%0AOwBGS
SIJY3NyZgY7AEZJIiVhZmQ2MmUyZGMxMzFmOGEwMjg3NDlhNWM3YmVm%0AN2FjNwY7AEZJIg10cmFja2luZwY7AEZ7B0kiFEhUVFBfVVNFUl9BR0VOVAY7%0AAFRJIi00MTc0OGM2MWNkMzljZTYxNzY3ZjU0
Y2I5OTdiYWRkN2MyNTBkYmU4%0ABjsARkkiGUhUVFBfQUNDRVBUX0xBTkdVQUdFBjsAVEkiLWRhMzlhM2VlNWU2%0AYjRiMGQzMjU1YmZlZjk1NjAxODkwYWZkODA3MDkGOwBG%0A--ee97a62095e7d42129
tontent-Length: 354c8; path=/; HttpOnly
This is breaking my response parsing logic.
I have seen this happening intermittently which is weird.
Is there a way to get "only" json response from cURL output?
I went through the curl documentation but could not see any thing/ or I could have missed it.
You are specifying the -i option:
-i, --include
(HTTP) Include the HTTP-header in the output. The HTTP-header includes things like server-name, date of the document, HTTP-version and more...
Simply remove that option from your command line:
response=$(curl -sb -H "Accept: application/json" "http://host:8080/some/resource")
I was executing a get request an also want to see just the response and nothing else, seems like magic is done with -silent,-s option.
From the curl man page:
-s, --silent
Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redirect it.
Below the examples:
curl -s "http://host:8080/some/resource"
curl --silent "http://host:8080/some/resource"
Using custom headers
curl -s -H "Accept: application/json" "http://host:8080/some/resource"
Using POST method with a header
curl -s -X POST -H "Content-Type: application/json" "http://host:8080/some/resource" -d '{ "myBean": {"property": "value"}}'
You can also customize the output for specific values with -w, below the options I use to get just response codes of the curl:
curl -s -o /dev/null -w "%{http_code}" "http://host:8080/some/resource"