API Logging in Json format - json

Question

I would like to log API request/response as json format.
The expected LogEntry is something like
{
"timestamp" : "...",
"level" : "DEBUG",
"headers" : [
"header1" : "value1",
"header2" : "value2",
"header3" : "value3"
],
"requestPayload" : "<Request Json>" // prefereablly as sub-document, worst case string is fine.
"labels" : [ //key fields which can be used for searching the logentry
"searchField1" : "....",
"searchField2" : "....",
"searchField3" : "...."
]
}
My Question is :
Using Logback, How to log nested fields (e.g. headers, labels, requestPaylod in above example), as json sub-document. I tried MDC, but its limited to
Map of 'String, String' only, and considers all fields after first level as String.
I hate to write my custom logger for this, and would like to use the goodness of proven logging frameworks(logback/log4j) to control the logging level, time stamping log event, etc.

Answer 1

For request/response logging try Logbook and logstash-logback-encoder. You'll be able to use Markers to add structured contents.
For adding 'your own' fields in a JSON structure, I've written a code generator you might find interesting, it adds builder support: json-log-domain

Answer 2

You can use logback-contrib's JsonLayout inside any Logback appender. For example:
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>false</prettyPrint>
</jsonFormatter>
<timestampFormat>yyyy-MM-dd' 'HH:mm:ss.SSS</timestampFormat>
<appendLineSeparator>true</appendLineSeparator>
</layout>
</appender>
Given log statements like this ...
MDC.put("header1", "headerValue1");
logger.info("hello!");
logger.info("good bye!");
... the use of JsonLayout would result in Logback writing this:
{"timestamp":"2017-08-15 09:06:41.813","level":"INFO","thread":"main","mdc":{"header1":"headerValue1"},"logger":"com.stackoverflow.logback.LogbackTest","message":"hello!","context":"default"}
{"timestamp":"2017-08-15 09:06:41.887","level":"INFO","thread":"main","mdc":{"header1":"headerValue1"},"logger":"com.stackoverflow.logback.LogbackTest","message":"good bye!","context":"default"}
I think this ticks the boxes of writing your log events as JSON documents whilst still retaining Logback's behaviour such as "control the logging level, time stamping log event, etc". There is some built-in support for changing the JSON format (e.g. you can include/exclude context, logger name etc) but the JsonLayout class provides an extension point which would allow you to change the attribute names in the resulting JSON by extending JsonLayout and overriding toJsonMap().
Edit 1: addressing your reply ("my question was more on how to achieve json subdocument/nesting as mentioned. MDC is limited to map of only") ... you could serialise your complex MDC values to JSON and add the serialised JSON representations to MDC. For example:
Map<String, Object> complexMdcValue = new HashMap<>();
Map<String, Object> childMdcValue = new HashMap<>();
childMdcValue.put("name", "Joe");
childMdcValue.put("type", "Martian");
complexMdcValue.put("child", childMdcValue);
complexMdcValue.put("category", "etc");
MDC.put("complexNestedValue", objectMapper.writeValueAsString(complexMdcValue));
logger.info("hello!");
Would produce this output (in which the MVC logged key "complexNestedValue" is JSON containing a sub document):
{"timestamp":"2017-08-27 18:03:46.706","level":"INFO","thread":"main","mdc":{"complexNestedValue":"{\"category\":\"etc\",\"child\":{\"name\":\"Joe\",\"type\":\"Martian\"}}"},"logger":"com.stackoverflow.logback.LogbackTest","message":"hello!","context":"default"}