Using JsonLayout with SOLR logs - json

Question

I'm trying to log all events from SOLR in JSON format by using JSONLayout. I'm using the official docker image from SOLR but I can't get it to work. Here's my setup:
Dockerfile
FROM solr:7
COPY log4j2-json.xml /opt/solr/server/resources/log4j2.xml
USER root
RUN chmod 644 /opt/solr/server/resources/log4j2.xml
RUN chown solr:solr /opt/solr/server/resources/log4j2.xml
USER solr
EXPOSE 8983
CMD ["solr-precreate", "gettingstarted"]
log4j2.xml original file (without JsonLayout) which does create the solr.log file (using PatternLayout):
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout>
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
</Pattern>
</PatternLayout>
</Console>
<RollingFile
name="RollingFile"
fileName="${sys:solr.log.dir}/solr.log"
filePattern="${sys:solr.log.dir}/solr.log.%i" >
<PatternLayout>
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="32 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
<RollingFile
name="SlowFile"
fileName="${sys:solr.log.dir}/solr_slow_requests.log"
filePattern="${sys:solr.log.dir}/solr_slow_requests.log.%i" >
<PatternLayout>
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="32 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="org.apache.hadoop" level="warn"/>
<Logger name="org.apache.solr.update.LoggingInfoStream" level="off"/>
<Logger name="org.apache.zookeeper" level="warn"/>
<Logger name="org.apache.solr.core.SolrCore.SlowRequest" level="info" additivity="false">
<AppenderRef ref="SlowFile"/>
</Logger>
<Root level="info">
<AppenderRef ref="RollingFile"/>
<AppenderRef ref="STDOUT"/>
</Root>
</Loggers>
</Configuration>
log4j2.xml with JsonLayout in RollingFile appender which breaks it. No solr.log file is created:
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout>
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
</Pattern>
</PatternLayout>
</Console>
<RollingFile
name="RollingFile"
fileName="${sys:solr.log.dir}/solr.log"
filePattern="${sys:solr.log.dir}/solr.log.%i" >
<JsonLayout complete="false" compact="true" eventEol="true" />
<Policies>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="32 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
<RollingFile
name="SlowFile"
fileName="${sys:solr.log.dir}/solr_slow_requests.log"
filePattern="${sys:solr.log.dir}/solr_slow_requests.log.%i" >
<PatternLayout>
<Pattern>
%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
</Pattern>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy />
<SizeBasedTriggeringPolicy size="32 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="org.apache.hadoop" level="warn"/>
<Logger name="org.apache.solr.update.LoggingInfoStream" level="off"/>
<Logger name="org.apache.zookeeper" level="warn"/>
<Logger name="org.apache.solr.core.SolrCore.SlowRequest" level="info" additivity="false">
<AppenderRef ref="SlowFile"/>
</Logger>
<Root level="debug">
<AppenderRef ref="RollingFile"/>
<AppenderRef ref="STDOUT"/>
</Root>
</Loggers>
</Configuration>
I would expect to see the solr.log file to be created in the logdir and see the events in json format. But the solr.log file isn't even created. SOLR does however start.
What am I missing?
Thanks for your help.
PS. I found this thread which describes the same problem but no solution: see here

Answer 1

On solr official docker image 8.11.2 the same problem is caused by a missing jackson library in the classpath:
ERROR Could not create plugin of type class org.apache.logging.log4j.core.layout.JsonLayout for element JsonLayout: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ser/FilterProvider java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ser/FilterProvider
at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:159)
This is definitely a bug (https://issues.apache.org/jira/browse/SOLR-16342), but since the jackson library exists elsewhere in the original image, there is a possible workaround to apply while building the docker. Basically you can link the needed jackson jars that exists in /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/ to /opt/solr/server/lib/ext/:
RUN ln -s /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/jackson-core-2.13.3.jar /opt/solr/server/lib/ext/jackson-core-2.13.3.jar && \
ln -s /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/jackson-databind-2.13.3.jar /opt/solr/server/lib/ext/jackson-databind-2.13.3.jar && \
ln -s /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/jackson-annotations-2.13.3.jar /opt/solr/server/lib/ext/jackson-annotations-2.13.3.jar
Moreover:
JsonTemplate is considered deprecated. JsonTemplateLayout provides
more capabilitites and should be used instead.
(see https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout)
If you move to <JsonTemplateLayout /> it works out of the box in solr.