Logback AsyncAppender not writing logs to underlying appenders - logback

In logback.xml of my application I have an AsyncAppender defined as below.
<appender name="socketAppender" class="ch.qos.logback.classic.net.SocketAppender">
<param name="RemoteHost" value="127.0.0.1" />
<param name="Port" value="15000" />
<param name="ReconnectionDelay" value="10" />
<param name="Threshold" value="DEBUG" />
</appender>
<appender name="appLog" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>logs/myApp.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>logs/myApp.%d{yyyyMMdd.HH}00.log</fileNamePattern>
</rollingPolicy>
<encoder>
<charset>UTF-8</charset>
<pattern>[%d{yyyy/MM/dd HH:mm:ss.SSS}][%p][%c{0}] %m%n]</pattern>
</encoder>
</appender>
<appender name="AsyncLog" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="appLog" />
<appender-ref ref="socketAppender" />
</appender>
When I deploy the war in Tomcat on Windows 10.
The log server listening on port 15000 wasn't receiving logs.
So I moved up the socketAppender to first position like below.
<appender name="AsyncLog" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="socketAppender" />
<appender-ref ref="appLog" />
</appender>
With this change log server started receiving logs, but the log file stopped writing/appending.
3) I also tried setting the queueSize and discardingThreshold properties on the appender, to no avail.
<discardingThreshold>0</discardingThreshold>
<queueSize>500</queueSize>
Can any logback experts please tell me what I am doing wrong here? Are there any other properties of AsyncAppender that may help fix this behaviour?

A colleague pointed me to this method in the class ch.qos.logback.core.AsyncAppenderBase. AsyncAppender can attach only one appender and ignores any more appenders added beyond that.
public void addAppender(Appender<E> newAppender) {
if (appenderCount == 0) {
appenderCount++;
addInfo("Attaching appender named [" + newAppender.getName() + "] to AsyncAppender.");
aai.addAppender(newAppender);
} else {
addWarn("One and only one appender may be attached to AsyncAppender.");
addWarn("Ignoring additional appender named [" + newAppender.getName() + "]");
}
}

Related

Logback-spring.xml unable to squash multiline stack trace to single line

I tried multiple solutions on SO, but none of them seem to be working for me. I am fairly new to Logback too, probably why I'm unable to find the right approach for my usecase.
Basically, we push logs to a centrally managed log service where we are required to send stack traces in a single line to avoid increasing the rate of ingestion (each line is a new log in the log service) and to make debugging easier.
Sample ERROR log:
14:50:41.856 ERROR [] 8 --- [trace=394a5287-6719-41b3-91e3-47b68b356856,span=394a5287-6719-41b3-91e3-47b68b356856] [jetty-274] c.e.f.s.c.a.controllers.BaseController : Error while processing request:
com.ExceptionClass : An exception occurred
at com.SomeClass.somefunction(SomeClass:237)
and so on, I want to have these all in a single statement
Here's how my logback-spring.xml file looks like
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- Read https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-logging.html -->
<!-- Taken from https://github.com/spring-projects/spring-boot/blob/master/spring-boot/src/main/resources/org/springframework/boot/logging/logback/defaults.xml -->
<conversionRule conversionWord="clr"
converterClass="org.springframework.boot.logging.logback.ColorConverter" />
<conversionRule conversionWord="ex"
converterClass="org.springframework.boot.logging.logback.ThrowableProxyConverter" />
<conversionRule conversionWord="wex"
converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter" />
<conversionRule conversionWord="wEx"
converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter" />
<!-- We use an additional new line %n at the end of every line so that read_mode=1
can be enabled for log service This enables us to add stack traces to the corresponding
error line automatically. See https://sites.google.com/a/flipkart.com/logsvc/home/components/ingestion-frontend/file-based-handoff/nuances -->
<property name="CONSOLE_LOG_PATTERN"
value="%n%clr(%d{HH:mm:ss.SSS}){faint} %highlight(${LOG_LEVEL_PATTERN:-%5p}) [%marker] %clr(${PID:- }){magenta} %clr(---){faint} [trace=%X{X-Trace-Id:-},span=%X{X-Span-Id:-}] %magenta([%t]){faint} %yellow(%-40.40logger{39}){cyan}%clr(:){faint} %m${LOG_EXCEPTION_CONVERSION_WORD:-%throwable}%n" />
<appender name="DEBUG_LEVEL_REMAPPER"
class="org.springframework.boot.logging.logback.LevelRemappingAppender">
<destinationLogger>org.springframework.boot</destinationLogger>
</appender>
<logger name="org.apache.catalina.startup.DigesterFactory" level="ERROR" />
<logger name="org.apache.catalina.util.LifecycleBase" level="ERROR" />
<logger name="org.apache.coyote.http11.Http11NioProtocol" level="WARN" />
<logger name="org.apache.sshd.common.util.SecurityUtils" level="WARN" />
<logger name="org.apache.tomcat.util.net.NioSelectorPool" level="WARN" />
<logger name="org.crsh.plugin" level="WARN" />
<logger name="org.crsh.ssh" level="WARN" />
<logger name="org.eclipse.jetty.util.component.AbstractLifeCycle"
level="ERROR" />
<logger name="org.hibernate.validator.internal.util.Version" level="WARN" />
<logger
name="org.springframework.boot.actuate.autoconfigure.CrshAutoConfiguration"
level="WARN" />
<logger name="org.springframework.boot.actuate.endpoint.jmx"
additivity="false">
<appender-ref ref="DEBUG_LEVEL_REMAPPER" />
</logger>
<!-- http://logback.qos.ch/manual/appenders.html -->
<!-- Taken from https://github.com/spring-projects/spring-boot/blob/master/spring-boot/src/main/resources/org/springframework/boot/logging/logback/console-appender.xml -->
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>${CONSOLE_LOG_PATTERN}</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<!-- Log to the console in addition to the log file -->
<root level="INFO">
<appender-ref ref="CONSOLE" />
</root>
</configuration>
I know that I need to update this line
<property name="CONSOLE_LOG_PATTERN"
value="%n%clr(%d{HH:mm:ss.SSS}){faint} %highlight(${LOG_LEVEL_PATTERN:-%5p}) [%marker] %clr(${PID:- }){magenta} %clr(---){faint} [trace=%X{X-Trace-Id:-},span=%X{X-Span-Id:-}] %magenta([%t]){faint} %yellow(%-40.40logger{39}){cyan}%clr(:){faint} %m${LOG_EXCEPTION_CONVERSION_WORD:-%throwable}%n" />
but I can't figure out how.
Here's what I tried
<property name="CONSOLE_LOG_PATTERN"
value="%n%clr(%d{HH:mm:ss.SSS}){faint} %highlight(${LOG_LEVEL_PATTERN:-%5p}) [%marker] %clr(${PID:- }){magenta} %clr(---){faint} [trace=%X{X-Trace-Id:-},span=%X{X-Span-Id:-}] %magenta([%t]){faint} %yellow(%-40.40logger{39}){cyan}%clr(:){faint} %m${LOG_EXCEPTION_CONVERSION_WORD:-%throwable} %replace(%msg){'\n', ''}%n" />
but it did not work as expected
Thanks in advance

logback smtpAppender mailing on both error and info log level

I am trying to use logback SMTP appender to send email alerts.
I am getting emails only when log level is 'error' (I know this is by default).
How do I get emails when log level is being set to 'info'?
<property resource="application.properties"/>
<appender name="EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
<smtpHost>${spring.mail.host}</smtpHost>
<username>${spring.mail.username}</username>
<password>${spring.mail.password}</password>
<smtpPort>${spring.mail.port}</smtpPort>
<STARTTLS>false</STARTTLS>
<SSL>true</SSL>
<subject>Exception: Registraion App %m</subject>
<to>${spring.mail.to}</to>
<from>${spring.mail.from}</from>
<asynchronousSending>true</asynchronousSending>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%date %-5level %logger{35} - %message%n</pattern>
</layout>
</appender>
<logger name="errorLogger" level="error" additivity="false">
<appender-ref ref="EMAIL"/>
</logger>
In the java class I am invoking this as below
public static final Logger emailExceptionLOGGER = LoggerFactory.getLogger("errorLogger");
try{
.....
}catch(Exception e){
emailExceptionLOGGER.error("To send exception email")
}
I would also like to send success emails when log level is being set to 'info'
<appender name="SUCCESS-EMAIL" class="ch.qos.logback.classic.net.SMTPAppender">
<smtpHost>${spring.mail.host}</smtpHost>
<username>${spring.mail.username}</username>
<password>${spring.mail.password}</password>
<smtpPort>${spring.mail.port}</smtpPort>
<STARTTLS>false</STARTTLS>
<SSL>true</SSL>
<subject>Registraion App %m</subject>
<to>${spring.mail.to}</to>
<from>${spring.mail.from}</from>
<asynchronousSending>true</asynchronousSending>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%date %-5level %logger{35} - %message%n</pattern>
</layout>
</appender>
<logger name="successLogger" level="info" additivity="false">
<appender-ref ref="SUCCESS-EMAIL"/>
</logger>
In the java class I am invoking this something like below
public static final Logger emailSuccessLOGGER = LoggerFactory.getLogger("successLogger");
if(success){
emailSuccessLOGGER.info("To send success email")
}
I would like to handle both together. Thanks in advance.
include below after layout in your email appender. It should filter out info logs.
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>ERROR</level>
</filter>

Logback - how to get each logger logging to a separate log file?

My application has lots of EJBs. The current bespoke Logger implementation creates a logger like this;
private static Logger logger = Logger.getInstance("SERVICE_NAME");
, and the logging will go into a file;
(path)/SERVICE_NAME/SERVICE_NAME.log
I want to replicate this behaviour with logback, but having real trouble grabbing the 'logger' name in the logback.xml configuration. It can be seen in the log encoder.pattern, i.e. "%d %-5level %logger{35} - %msg %n".
Any ideas how I can get this into a property/variable and then use it in the element?
I have a partial solution. If I create my own Discriminator, I can then use the Discriminator in the logback.xml to implement seperate-log-files-per-EJB.
Discriminator;
public class LoggerNameBasedDiscriminator implements Discriminator<ILoggingEvent> {
private static final String KEY = "loggerName";
private boolean started;
#Override
public String getDiscriminatingValue(ILoggingEvent iLoggingEvent) {
return iLoggingEvent.getLoggerName();
}
#Override
public String getKey() {
return KEY;
}
public void start() {
started = true;
}
public void stop() {
started = false;
}
public boolean isStarted() {
return started;
}
}
Then my logback.xml;
<configuration debug="true" scan="true" scanPeriod="30 seconds">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg</pattern>
</encoder>
</appender>
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="package.to.LoggerNameBasedDiscriminator"/>
<sift>
<appender name="FILE-${loggerName}" class="ch.qos.logback.core.FileAppender">
<FILE>path/to/logs/${loggerName}/${loggerName}.log</FILE>
<encoder>
<pattern>%d{HH:mm:ss.SSS} %-50(%level %logger{35}) %msg%n</pattern>
</encoder>
</appender>
</sift>
</appender>
<root level="debug">
<appender-ref ref="STDOUT" />
<appender-ref ref="SIFT" />
</root>
</configuration>
This solution seems to work, but now I have no time or size based log rotation!
Thanks to your example I implemented a solution for a loggername-based discriminator which routes different logger output to different files. Although the documentation of logback is so verbose, I couldn't find this essential information. You've surely found the solution mentioned by yayitswei already.
logback.xml:
[...]
<timestamp key="startTimestamp" datePattern="yyyy-MM-dd"/>
<timestamp key="folderTimestamp" datePattern="MM-yyyy"/>
<property name="LOGDIR" value="/var/log/spock" />
<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="com.enterprise.spock.LoggerNameBasedDiscriminator" />
<sift>
<appender name="FILE-${loggerName}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOGDIR}/${loggerName}-${startTimestamp}.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOGDIR}/${folderTimestamp}/${loggerName}-%d{yyyy-MM-dd}-%i.log.gz</fileNamePattern>
<maxFileSize>500KB</maxFileSize>
<maxHistory>100</maxHistory>
<totalSizeCap>50MB</totalSizeCap>
</rollingPolicy>
<encoder>
<charset>UTF-8</charset>
<pattern>%level %date{HH:mm:ss.SSS}: %msg %n</pattern>
</encoder>
</appender>
</sift>
</appender>
[...]
Edit:
I replaced TimeBasedRollingPolicy with SizeAndTimeBasedRollingPolicy as proposed here. You'll need at least logback 1.1.7 for that.
What you get with this, is a logfile for each Logger created.
Every logfile will look like this: /var/log/loggername-2017-08-03.log.
When about 500KB was written to the file, it will be archived as a gz-zipfile into /var/log/loggername/08-2017/loggername-2017-08-03-0.log.gz.
The 0 at the end of the gz-zipfile-name is the %i from the <fileNamePattern> above. Without the %i it won't work. Remember to use <configuration debug=true> in logback.xml if something won't work.

Logback AyncAppender not printing File and Line number

I have the following configuration file that is very similar to the standard example in the Logback manual. The only difference is the addition of [%F:%L]. while everything works, %F and %L do not work. If I remove the async appender and log directly using the file appender, everything works just great.
can somebody explain what is going on? And how to print the file name and line number as these two parameters are supposed to?
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>myapp.log</file>
<encoder><pattern>%logger{35} - [%F:%L] - %msg%n</pattern></encoder>
</appender>
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
</appender>
<root level="DEBUG"><appender-ref ref="ASYNC" /></root>
</configuration>
You need to set AsyncAppender's includeCallerData property to true. Here is the modified config file:
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>myapp.log</file>
<encoder><pattern>%logger{35} - [%F:%L] - %msg%n</pattern></encoder>
</appender>
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
<!-- add the following line -->
<includeCallerData>true</includeCallerData>
</appender>
<root level="DEBUG"><appender-ref ref="ASYNC" /></root>
</configuration>
I post same answer in groovy format for someone who want groovy style like me.
appender('FILE', ch.qos.logback.core.FileAppender) {
file = 'myapp.log'
encoder(PatternLayoutEncoder) {
pattern = '%logger{35} - [%F:%L] - %msg%n'
}
}
appender('ASYNC', ch.qos.logback.classic.AsyncAppender) {
appenderRef('FILE')
//add the following line
includeCallerData = true
}
root(DEBUG, ['ASYNC'])

SMTPAppender for log4j -- can't get it to work

I'm tying to add the SMTPAppender to an application for error notification (tomcat app). I haven't been able to get it to work up to this point. I have tried it using configuration both in the log4j.properties file as well as through the log4j.xml configuration. I'm not seeing any errors in the log related to a misconfiguration of the appender setup. I'm just not seeing any email attempts at all. Initially I has to bring in javax.mail and javax.activation so I know that it's attempting to set up and use the SMTPAppender, but I'm not seeing anything as far as miconfiguration errors or emails that are sent when errors occur.
Here's the log4j.properties that I tried (changed the private info)
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%-5p %d [%t] %c: %m%n
log4j.appender.email=org.apache.log4j.net.SMTPAppender
log4j.appender.email.Threshold=ERROR
log4j.appender.email.BufferSize=512
log4j.appender.email.To=myemail#mydomain.com
log4j.appender.email.From=from#mydomain.com
log4j.appender.email.SMTPHost=mysmtphost
log4j.appender.email.Subject=MULE -- error
log4j.appender.email.layout=org.apache.log4j.PatternLayout
log4j.appender.email.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n
log4j.appender.email.SMTPUsername=mysmtpuser
log4j.appender.email.SMTPPassword=mysmtppassword
Here's the configuation that I tried in the log4j.xml file
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%5p (%F:%L) - %m%n" />
</layout>
</appender>
<appender name="file" class="org.apache.log4j.RollingFileAppender">
<param name="File" value="/apps/data/pbymuleservices.log"/>
<param name="Append" value="true"/>
<param name="MaxFileSize" value="1000KB"/>
<param name="MaxBackupIndex" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%5p [%d{ISO8601}] (%F:%L) - %m%n"/>
</layout>
</appender>
<appender name="email" class="org.apache.log4j.net.SMTPAppender">
<param name="BufferSize" value="512" />
<param name="SMTPHost" value="mysmtphost" />
<param name="SMTPUsername" value="mysmtpusername" />
<param name="SMTPPassword" value="mysmtppassword" />
<param name="From" value="fromemail#mydomain.com" />
<param name="To" value="toemail#mydomain.com" />
<param name="Threshold" value="error" />
<param name="Subject" value="MULE -- Error" />
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%d{ISO8601}]%n%n%-5p%n%n%c%n%n%m%n%n" />
</layout>
<filter class="org.apache.log4j.varia.LevelRangeFilter">
<param name="LevelMin" value="error" />
<param name="LevelMax" value="fatal" />
</filter>
</appender>
<logger name="com.mytoplevel.package" additivity="false">
<level value="INFO"/>
<appender-ref ref="console"/>
<appender-ref ref="file"/>
</logger>
<logger name="org.mule" additivity="false">
<level value="WARN"/>
<appender-ref ref="console"/>
<appender-ref ref="file"/>
</logger>
<logger name="com.mulesource" additivity="false">
<level value="WARN"/>
<appender-ref ref="console"/>
<appender-ref ref="file"/>
</logger>
<root>
<priority value="error" />
<appender-ref ref="console" />
<appender-ref ref="file"/>
<appender-ref ref="email"/>
</root>
Anyone see anything that should keep this from working? I'm not seeing any configuration related errors in the log, but I'm also not seeing emails when errors occur.
Thanks
Add this param to the CATALINA_OPTS env variable: -Dlog4j.debug It will print where it's picking up the logging configuration from.