How to define multiple rolling policies for a single log file in logback? - logback

I am facing same issue as mentioned in Is there a way to define multiple rolling policies for a single log file in logback?
can any one know how to write multiple rolling policies which creates log like
test.log,
test.1.log,
test.2.log.gz,
test.3.log.gz and so on... Here it first create normal log file and 2nd onward it create archive of same.
Later on this all log will be read by filebeat.
Also tried with two appender like
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>test.log</file>
enter code here
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>archives/tests.%i.log</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>1</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>5MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILETEST" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>test.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>archives/tests.%i.log.gz</fileNamePattern>
<minIndex>2</minIndex>
<maxIndex>7</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>5MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<logger name="test-logger" level="DEBUG" additivity="false">
<appender-ref ref="FILE"/>
<appender-ref ref="FILETEST"/>
</logger>
This leads to generate the test.log, test.1.log, test.2.log.gz which I want but here it causes with it writes in both file test.log and in test.1.log at initial level, this again leads to data loss in rotation.

Related

Logback json configuration return no logs

I'm trying to get json logs from Nexus 3. Since it use logback, I tried ton change the logback.xml file but, for some reason, it result to no logs at all... Does someone know what I did wrong?
The only things I changed is to add those lines to the Nexus's default logback.xml file, to add json logs:
<appender name="json" class="ch.qos.logback.core.ConsoleAppender">
<filter class="org.sonatype.nexus.pax.logging.NexusLogFilter" />
<layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
<jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
<prettyPrint>true</prettyPrint>
</jsonFormatter>
<timestampFormat>yyyy-MM-dd' 'HH:mm:ss.SSS</timestampFormat>
</layout>
</appender>
and change these for logback to actually use it:
<root level="${root.level:-INFO}">
<appender-ref ref="osgi"/>
<appender-ref ref="console"/>
<appender-ref ref="json"/>
<appender-ref ref="logfile"/>
<appender-ref ref="clusterlogfile"/>
<appender-ref ref="tasklogfile"/>
<appender-ref ref="metrics"/>
</root>

Conditionally format logs in logback

I'm currently formatting my logs in logback like below. However, when I run my app locally I don't want all this metadata gunking up my logs, I just wanna see the message. How can I say "If an environment variable = 'local' then format this way, else format as below"?
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<Target>System.out</Target>
<encoder>
<Pattern>
date=%d{yyyy-MM-dd HH:mm:ss} | rte=${RTEID} | runId=%X{RunId} | interface=%X{Interface} | class=%class{0}.%method | level=%-5level | message=%msg %replace(%xException){'\n','\u2028'}%nopex%n
</Pattern>
</encoder>
</appender>
bit late. But if you are using spring boot you can use logback-spring.xml
and inside you can do something like
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<springProfile name="dev-local">
<pattern>FIST PATTERN</pattern>
</springProfile>
<springProfile name="!dev-local">
<pattern>SECOND PATTERN</pattern>
</springProfile>
</encoder>
</appender>

RollingFileAppender not working when used inside SiftingAppender

I have the following logback setup
<appender name="TEST-SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">
<discriminator class="..."/>
<sift>
<appender name="ROLL-${fileName}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>../log/${fileName}.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>../log/${fileName}%i.log</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>10</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>5MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] %logger{36} - %msg%n</pattern>
</encoder>
</appender>
</sift>
</appender>
The discriminator class returns a value by parsing the loggerName. The key is defined as "fileName".
The logs rollover fine when I test just the RollingFileAppender (after replacing the variable references of ${fileName} with a static value) but when I have it nested under SiftingAppender, the logs do not roll over. I tested the sifting appender with "FileAppender" and it is able to create the right file name based on the discriminator.
I also tested the same configuration by using the discriminator as
<discriminator>
<key>fileName</key>
<defaultValue>appname</defaultValue>
</discriminator>
and removing the class tag. This creates appname.log but does not roll over.
Setting debug="true" did not write any additional information to the log file.
Am I missing something here? How do I implement RollingFileAppender inside a SiftingAppender?
I figured out the issue with my setup. My logback.xml has to two RollingFileAppenders (one nested in the sifter and one outside). Appender A was writing to application.log and Appender B in some circumstances was writing to application.log as well (i.e. ${fileName} evaluated to application). So if I remove appender A or rename Appender A's fileName, the logs roll over as configured. This probably means Appender A or B could not close and rename the file because the other appender still has a lock on it?
To test this I used an AsyncAppender as follows:
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
</appender>
where "FILE" is the name of Appender A. Using this configuration works but I see some strange behavior where the files do not rollover at the exact size specified and in some cases, the files are renamed with index 10 and get deleted automatically. Since this behavior is not very reliable, for now I got rid of Appender A.

logback's RollingFileAppender is not rolling file with EvaluatorFilter

Logback framework's ch.qos.logback.core.rolling.RollingFileAppender is not rolling files when EvaluatorFilter is used as following:
<appender name="APPLICATION-INFO"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${log.root}/${CONTEXT_NAME}-INFO.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${log.root}/${CONTEXT_NAME}-INFO-%d{yyyyMMdd}.log
</fileNamePattern>
</rollingPolicy>
<filter class="ch.qos.logback.core.filter.EvaluatorFilter">
<evaluator>
<expression>return (level == INFO || level == WARN);</expression>
</evaluator>
<OnMismatch>DENY</OnMismatch>
<OnMatch>ACCEPT</OnMatch>
</filter>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.xxx.yyyy.zzz.logging.json.JsonLayout">
<timestampFormat>${timestampFormat}</timestampFormat>
<includeMDC>false</includeMDC>
</layout>
<immediateFlush>true</immediateFlush>
</encoder>
</appender>
But the the following configuration rolls the log file.
<appender name="APPLICATION-ERROR"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${log.root}/${CONTEXT_NAME}-ERROR.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${log.root}/${CONTEXT_NAME}-ERROR-%d{yyyyMMdd}.log
</fileNamePattern>
</rollingPolicy>
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>ERROR</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.xxx.yyy.zzz.logging.json.JsonLayout">
<timestampFormat>${timestampFormat}</timestampFormat>
<includeMDC>false</includeMDC>
</layout>
<immediateFlush>true</immediateFlush>
</encoder>
</appender>
I'm a bit lazy to test that locally but I have a clue what might be wrong :-) In the docs of logback it is written:
For various technical reasons, rollovers are not clock-driven but
depend on the arrival of logging events. For example, on 8th of March
2002, assuming the fileNamePattern is set to yyyy-MM-dd (daily
rollover), the arrival of the first event after midnight will trigger
a rollover. If there are no logging events during, say 23 minutes and
47 seconds after midnight, then rollover will actually occur at
00:23'47 AM on March 9th and not at 0:00 AM.
Based on this a possible guess is that there was no proper logging event accepted by the default JaninoEventEvaluator that you configured.
Another speculation is that you do not refer the appender properly but there are not enough details about this in the question.

Logback daily rolling incorrect

We are using quite a basic daily rolling configuration with logback, and it seems not to function properly. Application creates rolled (previous day) file usually in first minute after midnight. The yyyyMMdd fraction of file name contans previous day, that's correct. But messages in log are not from previous but from new day, only that 30-40 seconds !
For example, bim.2013-08-21.log file contains records with timestamps from 2013.08.22 00:00:00 to 2013.08.22 00:00:42. And all messages from previous day gets lost.
Also, there was some "lucky" exceptions when rolling started in the mid of the day , e.g. 16:00 , so we got more records rolled - from current day.
Logback version is 1.0.13
logback-test.xml :
<configuration scan="true" scanPeriod="10 minutes">
<contextName>bim</contextName>
<property name="LOG_DIR" value="/usr/share/apache-tomcat-7.0.41/logs" />
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} %-5level %m \(%logger{36}:%L\)%n</pattern>
</encoder>
</appender>
<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_DIR}/bim.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- daily rollover -->
<fileNamePattern>${LOG_DIR}/bim.%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>5</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %m \(%logger{36}:%L\)%n</pattern>
</encoder>
</appender>
<logger name="com.bim" level="DEBUG" />
<logger name="org.springframework.web.servlet.mvc.support" level="WARN" />
<logger name="org.springframework.web.servlet.mvc.annotation" level="WARN" />
<root level="INFO">
<!--appender-ref ref="console" /-->
<appender-ref ref="file" />
</root>
I have not found any traces of similar problems by search, so your help will be greatly appreciated.
I can't see what part are you missing over here. Even I have done the same and its working fine. Please check below link and try it in the same fashion :
https://github.com/abdulwaheed18/Slf4jTutorial/blob/master/sample7.xml