(log4jdbc + slf4j + logback) how to auto line sort sql log? - logback

I installed Log4jdbc-log4j2 + slf4j + logback on my dynamic web server project.
During the test, the log and query statements come out as set, but like other people, the query statement does not wrap automatically.
Only one line.
If you set the log4jdbc.dump.sql.maxlinelength portion of the log4jdbc.log4j2.properties file to 5,
Line breaks occur as much as the set value, but they are sorted out in a mess.
Previously, when using log4sql it was automatically sorted
sql log is
INSERT INTO TB_LOG VALUES ( 'test', 'test', SYSDATE, '0:0:0:0:0:0:0:1', '0')
i want this (Automatically sorted when using log4sql)
INSERT INTO
TB_LOG VALUES (
'test'
, 'test'
, SYSDATE
, '0:0:0:0:0:0:0:1'
, '0')
How do I auto-align this part?
my setting
log4jdbc.log4j2.properties
log4jdbc.spylogdelegator.name=net.sf.log4jdbc.log.slf4j.Slf4jSpyLogDelegator
log4jdbc.dump.sql.maxlinelength=0
logback.xml
<configuration scan="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-35(%d{yyyyMMdd HH:mm:ss.SSS} [%thread]) %-5level %logger{20} - %msg %n%n</pattern>
</encoder>
</appender>
<logger name="jdbc" level="OFF"/>
<logger name="jdbc.sqlonly" level="OFF"/>
<logger name="jdbc.sqltiming" level="DEBUG"/>
<logger name="jdbc.audit" level="OFF"/>
<logger name="jdbc.resultset" level="OFF"/>
<logger name="jdbc.resultsettable" level="OFF"/>
<logger name="jdbc.connection" level="OFF"/>
<root level="INFO">
<appender-ref ref="STDOUT" />
</root>
</configuration>

Related

Logback-spring.xml unable to squash multiline stack trace to single line

I tried multiple solutions on SO, but none of them seem to be working for me. I am fairly new to Logback too, probably why I'm unable to find the right approach for my usecase.
Basically, we push logs to a centrally managed log service where we are required to send stack traces in a single line to avoid increasing the rate of ingestion (each line is a new log in the log service) and to make debugging easier.
Sample ERROR log:
14:50:41.856 ERROR [] 8 --- [trace=394a5287-6719-41b3-91e3-47b68b356856,span=394a5287-6719-41b3-91e3-47b68b356856] [jetty-274] c.e.f.s.c.a.controllers.BaseController : Error while processing request:
com.ExceptionClass : An exception occurred
at com.SomeClass.somefunction(SomeClass:237)
and so on, I want to have these all in a single statement
Here's how my logback-spring.xml file looks like
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- Read https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html -->
<!-- Taken from https://github.com/spring-projects/spring-boot/blob/master/spring-boot/src/main/resources/org/springframework/boot/logging/logback/defaults.xml -->
<conversionRule conversionWord="clr"
converterClass="org.springframework.boot.logging.logback.ColorConverter" />
<conversionRule conversionWord="ex"
converterClass="org.springframework.boot.logging.logback.ThrowableProxyConverter" />
<conversionRule conversionWord="wex"
converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter" />
<conversionRule conversionWord="wEx"
converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter" />
<!-- We use an additional new line %n at the end of every line so that read_mode=1
can be enabled for log service This enables us to add stack traces to the corresponding
error line automatically. See https://sites.google.com/a/flipkart.com/logsvc/home/components/ingestion-frontend/file-based-handoff/nuances -->
<property name="CONSOLE_LOG_PATTERN"
value="%n%clr(%d{HH:mm:ss.SSS}){faint} %highlight(${LOG_LEVEL_PATTERN:-%5p}) [%marker] %clr(${PID:- }){magenta} %clr(---){faint} [trace=%X{X-Trace-Id:-},span=%X{X-Span-Id:-}] %magenta([%t]){faint} %yellow(%-40.40logger{39}){cyan}%clr(:){faint} %m${LOG_EXCEPTION_CONVERSION_WORD:-%throwable}%n" />
<appender name="DEBUG_LEVEL_REMAPPER"
class="org.springframework.boot.logging.logback.LevelRemappingAppender">
<destinationLogger>org.springframework.boot</destinationLogger>
</appender>
<logger name="org.apache.catalina.startup.DigesterFactory" level="ERROR" />
<logger name="org.apache.catalina.util.LifecycleBase" level="ERROR" />
<logger name="org.apache.coyote.http11.Http11NioProtocol" level="WARN" />
<logger name="org.apache.sshd.common.util.SecurityUtils" level="WARN" />
<logger name="org.apache.tomcat.util.net.NioSelectorPool" level="WARN" />
<logger name="org.crsh.plugin" level="WARN" />
<logger name="org.crsh.ssh" level="WARN" />
<logger name="org.eclipse.jetty.util.component.AbstractLifeCycle"
level="ERROR" />
<logger name="org.hibernate.validator.internal.util.Version" level="WARN" />
<logger
name="org.springframework.boot.actuate.autoconfigure.CrshAutoConfiguration"
level="WARN" />
<logger name="org.springframework.boot.actuate.endpoint.jmx"
additivity="false">
<appender-ref ref="DEBUG_LEVEL_REMAPPER" />
</logger>
<!-- http://logback.qos.ch/manual/appenders.html -->
<!-- Taken from https://github.com/spring-projects/spring-boot/blob/master/spring-boot/src/main/resources/org/springframework/boot/logging/logback/console-appender.xml -->
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>${CONSOLE_LOG_PATTERN}</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<!-- Log to the console in addition to the log file -->
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>
I know that I need to update this line
<property name="CONSOLE_LOG_PATTERN"
value="%n%clr(%d{HH:mm:ss.SSS}){faint} %highlight(${LOG_LEVEL_PATTERN:-%5p}) [%marker] %clr(${PID:- }){magenta} %clr(---){faint} [trace=%X{X-Trace-Id:-},span=%X{X-Span-Id:-}] %magenta([%t]){faint} %yellow(%-40.40logger{39}){cyan}%clr(:){faint} %m${LOG_EXCEPTION_CONVERSION_WORD:-%throwable}%n" />
but I can't figure out how.
Here's what I tried
<property name="CONSOLE_LOG_PATTERN"
value="%n%clr(%d{HH:mm:ss.SSS}){faint} %highlight(${LOG_LEVEL_PATTERN:-%5p}) [%marker] %clr(${PID:- }){magenta} %clr(---){faint} [trace=%X{X-Trace-Id:-},span=%X{X-Span-Id:-}] %magenta([%t]){faint} %yellow(%-40.40logger{39}){cyan}%clr(:){faint} %m${LOG_EXCEPTION_CONVERSION_WORD:-%throwable} %replace(%msg){'\n', ''}%n" />
but it did not work as expected
Thanks in advance

SLF4J Logging Exception as JSON or Single-Line String

I need to be able to log an Exception as a single record in my logs, which will make it much easier to investigate issues in Kibana / Elasticsearch. From what I can tell from the documentation for slf4j, the Logger interface requires messages to be Strings. Is my only option to remove newline characters from the Exception message before passing it to the Logger?
For context, I am using the following:
.m2/repository/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.jar
Java 11
Sprint Boot version 2.1.8.RELEASE
This is a trimmed down version of my custom exception handler :
import my.error.Error; // custom Error class
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;
#ControllerAdvice
public class CustomExceptionHandler extends ResponseEntityExceptionHandler {
private final Logger logger = LoggerFactory.getLogger(this.getClass());
private void logError(Error error, Exception ex){
logger.error(String.format("id: %s, message: %s", error.getId(), ex.getMessage()), ex);
}
}
Initially I had attempted to alter the logging behavior with changes to my logback.xml file, within src/main/java/resources. Unfortunately, this appears to do nothing, so my assumption now is that the CustomExceptionHandler that I am creating is overruling the specification set in the logback.xml file. Specifically, the <pattern> of the <encoder> has been changed based on other research. It's attempting to replace all newline characters.
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" debug="true">
<include resource="org/springframework/boot/logging/logback/base.xml" />
<appender name="FILE-ROLLING" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/gateway.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/archived/gateway/gateway.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<!-- each archived file, size max 5MB -->
<maxFileSize>5MB</maxFileSize>
<!-- total size of all archive files, if total size > 10GB, it will delete old archived file -->
<totalSizeCap>10GB</totalSizeCap>
<!-- 30 days to keep -->
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder>
<pattern>%d %p %c{1.} [%t] %m MULTIEXCEPTION %replace(%xException){'\n','\u2028'}%nopex%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="FILE-ROLLING" level="DEBUG" additivity="false"/>
</root>
<springProfile name="local">
<logger name="my.gateway" level="TRACE" additivity="false">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE-ROLLING" />
</logger>
<logger name="com.netflix" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="org.springframework" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="com" level="INFO" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="gov" level="INFO" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="org" level="INFO" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
</springProfile>
</configuration>
Links
Logback Docs
SLF4J Docs
SLF4J Manual
Combine Logback and SLF4J
Logback and SLF4J - StructuredArguments
JSON Logging w/ Logback
Collapse Logs - Multiple Lines into One
Apache Log4j Layouts
SO : Override Logback 1
SO : Override Logback 2
SO : Make Logback Output JSON
Format SLF4J
Collapse multi-line logs into one with Logback or Log4j2
Baeldung Log4j2 JSON Logging
Java Logging Guide
I changed my logError method to this:
import org.slf4j.MDC; // new import to add property
// ... other imports from before
private void logError(Error error, Exception exception){
MDC.put("error id", error.getId().toString()); // add a new property to the thread context
LOGGER.error(exception.getMessage(), exception);
MDC.clear(); // remove all new properties after logging the error
}
My new pom.xml File Dependencies:
<dependencies>
<!-- logging -->
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>5.2</version>
</dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-jackson</artifactId>
<version>0.1.5</version>
</dependency>
<!-- ch.qos.logback.contrib.json.classic.JsonLayout -->
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
<version>0.1.5</version>
</dependency>
<!-- Other dependencies -->
<!-- ... -->
<!-- ... -->
</dependencies>
New logback.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="FILE-ROLLING">
<!-- Old encoder is removed:
<encoder>
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
</encoder>
-->
<file>${LOG_PATH}/data.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/archived/data/data.%d{yyyy-MM-dd}.%i.log.gz
</fileNamePattern>
<!-- each archived file, size max 10MB -->
<maxFileSize>10MB</maxFileSize>
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
<maxHistory>60</maxHistory>
<!-- 60 days to keep -->
<totalSizeCap>20GB</totalSizeCap>
</rollingPolicy>
<!-- TODO : configure a pretty printer -->
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<!-- TODO : configure a pretty printer -->
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>
<!-- The base logback configuration is removed, but note that this will also remove the default Spring start-up messages, so those need to be added back into the <root>
<include resource="org/springframework/boot/logging/logback/base.xml"/>
-->
<root level="INFO">
<appender-ref ref="FILE-ROLLING"/>
<appender-ref ref="CONSOLE"/> <!-- This adds Spring start-up messages back into the logs -->
</root>
<springProfile name="local">
<logger additivity="false" level="TRACE" name="org.springframework.web">
<appender-ref ref="CONSOLE"/>
</logger>
</springProfile>
<springProfile name="dev">
<logger additivity="false" level="DEBUG" name="org.springframework">
<appender-ref ref="FILE-ROLLING"/>
</logger>
</springProfile>
<springProfile name="prod">
<logger additivity="false" level="DEBUG" name="my.app.path">
<appender-ref ref="FILE-ROLLING"/>
</logger>
<logger additivity="false" level="DEBUG" name="org.jooq">
<appender-ref ref="FILE-ROLLING"/>
</logger>
<logger additivity="false" level="DEBUG" name="org.springframework">
<appender-ref ref="FILE-ROLLING"/>
</logger>
</springProfile>
</configuration>
SLF4J Docs - Mapped Diagnostic Context (MDC)
Baeldung Example - MDC + Log4J2

logback smtpAppender mailing on both error and info log level

I am trying to use logback SMTP appender to send email alerts.
I am getting emails only when log level is 'error' (I know this is by default).
How do I get emails when log level is being set to 'info'?
<property resource="application.properties"/>
<appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
<smtpHost>${spring.mail.host}</smtpHost>
<username>${spring.mail.username}</username>
<password>${spring.mail.password}</password>
<smtpPort>${spring.mail.port}</smtpPort>
<STARTTLS>false</STARTTLS>
<SSL>true</SSL>
<subject>Exception: Registraion App %m</subject>
<to>${spring.mail.to}</to>
<from>${spring.mail.from}</from>
<asynchronousSending>true</asynchronousSending>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%date %-5level %logger{35} - %message%n</pattern>
</layout>
</appender>
<logger name="errorLogger" level="error" additivity="false">
<appender-ref ref="EMAIL"/>
</logger>
In the java class I am invoking this as below
public static final Logger emailExceptionLOGGER = LoggerFactory.getLogger("errorLogger");
try{
.....
}catch(Exception e){
emailExceptionLOGGER.error("To send exception email")
}
I would also like to send success emails when log level is being set to 'info'
<appender name="SUCCESS-EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
<smtpHost>${spring.mail.host}</smtpHost>
<username>${spring.mail.username}</username>
<password>${spring.mail.password}</password>
<smtpPort>${spring.mail.port}</smtpPort>
<STARTTLS>false</STARTTLS>
<SSL>true</SSL>
<subject>Registraion App %m</subject>
<to>${spring.mail.to}</to>
<from>${spring.mail.from}</from>
<asynchronousSending>true</asynchronousSending>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%date %-5level %logger{35} - %message%n</pattern>
</layout>
</appender>
<logger name="successLogger" level="info" additivity="false">
<appender-ref ref="SUCCESS-EMAIL"/>
</logger>
In the java class I am invoking this something like below
public static final Logger emailSuccessLOGGER = LoggerFactory.getLogger("successLogger");
if(success){
emailSuccessLOGGER.info("To send success email")
}
I would like to handle both together. Thanks in advance.
include below after layout in your email appender. It should filter out info logs.
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>ERROR</level>
</filter>

How to use JSON appender in Logback?

I am trying to output all my log events in JSON format.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSZ} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="INFO">
<!--<appender-ref ref="kafkaAppender" />-->
<appender-ref ref="STDOUT" />
</root>
Maven Dependency:
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>4.7</version>
</dependency>
I see an error while running this configuration. Did i miss anything?
here is the stacktrace:
Exception in thread "main" java.lang.AbstractMethodError: net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder.headerBytes()[B
at ch.qos.logback.core.OutputStreamAppender.encoderInit(OutputStreamAppender.java:180)
at ch.qos.logback.core.OutputStreamAppender.setOutputStream(OutputStreamAppender.java:171)
at ch.qos.logback.core.ConsoleAppender.start(ConsoleAppender.java:81)
at ch.qos.logback.core.joran.action.AppenderAction.end(AppenderAction.java:90)
at ch.qos.logback.core.joran.spi.Interpreter.callEndAction(Interpreter.java:309)
at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:193)
at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:179)
at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62)
at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:165)
at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:152)
at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:110)
at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:53)
at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:150)
at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:84)
at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273)
I had encountered the same error while using 4.7 version of the logstash-logback-encoder, and upgrading to 4.11 resolved the issue.
Ref- https://github.com/thumbtack/becquerel/issues/2

Logback AyncAppender not printing File and Line number

I have the following configuration file that is very similar to the standard example in the Logback manual. The only difference is the addition of [%F:%L]. while everything works, %F and %L do not work. If I remove the async appender and log directly using the file appender, everything works just great.
can somebody explain what is going on? And how to print the file name and line number as these two parameters are supposed to?
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>myapp.log</file>
<encoder><pattern>%logger{35} - [%F:%L] - %msg%n</pattern></encoder>
</appender>
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
</appender>
<root level="DEBUG"><appender-ref ref="ASYNC" /></root>
</configuration>
You need to set AsyncAppender's includeCallerData property to true. Here is the modified config file:
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>myapp.log</file>
<encoder><pattern>%logger{35} - [%F:%L] - %msg%n</pattern></encoder>
</appender>
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
<!-- add the following line -->
<includeCallerData>true</includeCallerData>
</appender>
<root level="DEBUG"><appender-ref ref="ASYNC" /></root>
</configuration>
I post same answer in groovy format for someone who want groovy style like me.
appender('FILE', ch.qos.logback.core.FileAppender) {
file = 'myapp.log'
encoder(PatternLayoutEncoder) {
pattern = '%logger{35} - [%F:%L] - %msg%n'
}
}
appender('ASYNC', ch.qos.logback.classic.AsyncAppender) {
appenderRef('FILE')
//add the following line
includeCallerData = true
}
root(DEBUG, ['ASYNC'])