nginx logging $request_body as hexadecimal - json

Question

I am trying to log the request body of requests to my api and nginx is turning all quotes (and some other characters like spaces and tabs) into hexadecimal characters.
Here is my log format
log_format postdata '{"ts": "$time_iso8601", "status": $status, "req": "$uri", "meth": "$request_method", "body": "$request_body"}';
Here is what gets logged
{"ts": "2015-05-20T15:31:11-07:00", "status": 400, "req": "/v2/track", "meth": "POST", "body": {\x22id\x22:\x22user id\x22}}
How can I prevent this so that the resulting log line is
{"ts": "2015-05-20T15:31:11-07:00", "status": 400, "req": "/v2/track", "meth": "POST", "body": {"id":"user id"}}

Answer 1

Sinse 1.13 there is an "escape=none" parameter that turns off data escaping.
http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format
log_format api_request_log escape=none '[$time_local] $request \n$request_body';

Answer 2

You can't stop from escaping it and will have to post process it.
Python2 example:
line = '{\x22id\x22:\x22user id\x22}'
line.decode('unicode_escape')
>> u'{"id":"user id"}'
Python3 example:
line = '{\x22id\x22:\x22user id\x22}'
bytes(line, 'utf-8').decode('unicode_escape')
>> '{"id":"user id"}'
Ruby example (from https://stackoverflow.com/a/18752208/2398354):
require 'yaml'
line = '{\x22id\x22:\x22user id\x22}'
YAML.load(%Q(---\n"#{line}"\n))
=> "{\"id\":\"user id\"}"
Note: This last example is useful if post processing a file with logstash

Answer 3

Hope this will be helpful for someone.
In order to log entire json request unescaped.
Do add in http block this configuration
http {
log_format postdata escape=json $request_body;
access_log /var/log/nginx/access.log postdata;
.....
}

Answer 4

Like others said, there is no way to fix this within nginx configuration.
But it is not difficult to post-process it. If the request body is JSON-formatted, you'll probably run into a lot of \x0A (newline) and \x22 (").
Just clear those out with sed before looking into the logfile.
Here is the command for you: LANG='' sed -E 's/(\\x0A|\\x22)//g' access.log