Log in JSON format to cloudfoundry logstash - json

Question

We are using Swisscom application cloud (based on Cloudfoundry) and the provided Kibana/Logstash/Elasticsearch service. Now we d like to log in JSON format from our applications to logstash.
Thats why we integrated Logstash formatter into our wildfly swarm apps and since then they log in JSON format like:
{"#version":1,"#timestamp":"2018-07-24T18:28:51.291+0200","sequence":15299,"loggerClassName":"org.jboss.as.server.logging.ServerLogger_$logger","loggerName":"org.jboss.as.server.deployment","level":"INFO","threadName":"MSC service thread 1-2","message":"WFLYSRV0027: Starting deployment of \"hospush.war\" (runtime-name: \"hospush.war\")","threadId":31,"mdc":{},"ndc":""}
I also added a filter.conf to the logstash app on swisscom appcloud with the following content:
filter {
json {
source => "message"
}
}
When I check the logs of logstash now I can see that it throws an error and no logs are transferred to Kibana.
2018-07-24 20:33:15 [APP/PROC/WEB/0] OUT [2018-07-24T18:33:15,202][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"<14>1 2018-07-24T18:33:15.018187+00:00 HosPush.demo-test.demo-test 3694b57f-bc05-459a-880a-17c174fc6d7c [APP/PROC/WEB/0] - - {\"#version\":1,\"#timestamp\":\"2018-07-24T20:33:15.017+0200\",\"sequence\":3796,\"loggerClassName\":\"org.slf4j.impl.Slf4jLogger\",\"loggerName\":\"com.hospush.business.escalation.EscalationService\",\"level\":\"INFO\",\"threadName\":\"EJB default - 3\",\"message\":\"Found 0 patientNeeds with no open notification.\",\"threadId\":154,\"mdc\":{},\"ndc\":\"\"}\n", :exception=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
2018-07-24 20:33:15 [APP/PROC/WEB/0] OUT at [Source: (byte[])"<14>1 2018-07-24T18:33:15.018187+00:00 HosPush.demo-test.demo-test 3694b57f-bc05-459a-880a-17c174fc6d7c [APP/PROC/WEB/0] - - {"#version":1,"#timestamp":"2018-07-24T20:33:15.017+0200","sequence":3796,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"com.hospush.business.escalation.EscalationService","level":"INFO","threadName":"EJB default - 3","message":"Found 0 patientNeeds with no open notification.","threadId":154,"mdc":{},"ndc":""}
2018-07-24 20:33:30 [APP/PROC/WEB/0] OUT [2018-07-24T18:33:30,117][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"<14>1 2018-07-24T18:33:30.002777+00:00 HosPush.demo-test.demo-test 3694b57f-bc05-459a-880a-17c174fc6d7c [APP/PROC/WEB/0] - - {\"#version\":1,\"#timestamp\":\"2018-07-24T20:33:30.001+0200\",\"sequence\":3797,\"loggerClassName\":\"org.slf4j.impl.Slf4jLogger\",\"loggerName\":\"com.hospush.business.escalation.OrphanEscalationScheduler\",\"level\":\"INFO\",\"threadName\":\"EJB default - 4\",\"message\":\"orphan escalation scheduler called...\",\"threadId\":155,\"mdc\":{},\"ndc\":\"\"}\n", :exception=>#<LogStash::Json::ParserError: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
2018-07-24 20:33:30 [APP/PROC/WEB/0] OUT at [Source: (byte[])"<14>1 2018-07-24T18:33:30.002777+00:00 HosPush.demo-test.demo-test 3694b57f-bc05-459a-880a-17c174fc6d7c [APP/PROC/WEB/0] - - {"#version":1,"#timestamp":"2018-07-24T20:33:30.001+0200","sequence":3797,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"com.hospush.business.escalation.OrphanEscalationScheduler","level":"INFO","threadName":"EJB default - 4","message":"orphan escalation scheduler called...","threadId":155,"mdc":{},"ndc":""}
My guess is, that because of source => "message" logstash parses the message property as JSON which failes. What it should do is parsing the whole "root object" as json instead of only the message property.
Could that be and if yes, how do I need to adjust the filter.conf to make it work?
Thx a lot for your help guys.

Answer 1

Probably not relevant, but I guess the filter here is incorrect.
This says that their is a JSON structure in the message field that should be parsed as JSON. As I see, I do not see any message field, so this will not work.