Logback: SizeAndTimeBasedRollingPolicy deletes all archived files when totalSizeCap reached - logback

I am using SizeAndTimeBasedRollingPolicy in logback. For small values of maxFileSize and totalSizeCap, logback deletes older archived files only, when totalSizeCap limit is reached. But, for large values of totalSizeCap (~ 5GB) , it deletes all archived files.
I would like to delete only the older archived files, when totalSizeCap limit is reached.I am using logback version 1.2.3
Here, is the logback configuration i am using.
<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${tivo.logpath}/${tivo.logfilename}.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!-- Rollover everyday. If file exceeds 1GB within a day, then file is archived with index starting from 0 -->
<fileNamePattern>${tivo.logpath}/${tivo.logfilename}-%d{yyyyMMdd}-%i.log.gz</fileNamePattern>
<!-- Each file should be at most 1GB -->
<maxFileSize>1GB</maxFileSize>
<!-- Keep maximum 30 days worth of archive files, deleting older ones -->
<maxHistory>30</maxHistory>
<!-- Total size of all archived files is at most 5GB -->
<totalSizeCap>5GB</totalSizeCap>
</rollingPolicy>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.tivo.logging.logback.layout.JsonLayout">
<env>${envId}</env>
<datacenter>${dcId}</datacenter>
<serverId>${serverId}</serverId>
<build>${info.properties.buildChange}</build>
<service>${tivo.appname}</service>
</layout>
</encoder>
</appender>

Looks like this is a known issue with the logback version < 1.3.0.
Logback: SizeAndTimeBasedRollingPolicy applies totalSizeCap to each day in maxHistory
https://jira.qos.ch/browse/LOGBACK-1361
So, we might have to update to that version.
One other interesting bug fixed in the logback 1.3.0 will be:
https://jira.qos.ch/browse/LOGBACK-1162

Related

Logback: Rollover is not deleting expired Logs

I have a RollingFileAppender which is archiving the logs to files like this:
/data/work/logs/printPDF/logs/service/printPDF_2020-01/printPDF_2020-01-23_0.log.gz
Unfortunately, it is not deleting old archives (like the one named above).
Does anybody know what I'm doing wrong?
Below is an excerpt from the Startup Logback Console output:
- setting totalSizeCap to 20 GB
- Archive files will be limited to [100 MB] each.
- Will use gz compression
- Will use the pattern /data/work/logs/printPDF/logs/service/printPDF_%d{yyyy-MM, aux}/printPDF_%d{yyyy-MM-dd}_%i.log for the active file
- The date pattern is 'yyyy-MM-dd' from file name pattern '/data/work/logs/printPDF/logs/service/printPDF_%d{yyyy-MM, aux}/printPDF_%d{yyyy-MM-dd}_%i.log.gz'.
- Roll-over at midnight.
- Setting initial period to Tue Nov 10 05:31:17 MET 2020
- Cleaning on start up
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
- first clean up after appender initialization
- Multiple periods, i.e. 32 periods, seem to have elapsed. This is expected at application start.
- Active log file name: /data/work/logs/printPDF/printPDF.log
- File property is set to [/data/work/logs/printPDF/printPDF.log]
As you can see from the Console Output, it has recognised a Daily Date Pattern & expects to rollover at midnight.
It is rolling over at midnight, but expired Logs are not being deleted. Here are a few of the filenames it failed to delete:
/data/work/logs/printPDF/logs/service/printPDF_2020-01/printPDF_2020-01-23_0.log.gz
/data/work/logs/printPDF/logs/service/printPDF_2020-01/printPDF_2020-01-23_1.log.gz
/data/work/logs/printPDF/logs/service/printPDF_2020-02/printPDF_2020-02-28_0.log.gz
And here's the one it just created at midnight:
/data/work/logs/printPDF/logs/service/printPDF_2020-11/printPDF_2020-11-10_0.log.gz
Here's logback.xml, which I expect to retain 90 days:
<configuration debug="true">
<property name="SERVICE" value="printPDF" />
<property name="LOGDIR" value="/data/work/logs/${SERVICE}" />
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<appender name="RollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOGDIR}/${SERVICE}.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<FileNamePattern>${LOGDIR}/logs/service/${SERVICE}_%d{yyyy-MM, aux}/${SERVICE}_%d{yyyy-MM-dd}_%i.log.gz</FileNamePattern>
<maxHistory>90</maxHistory>
<maxFileSize>100MB</maxFileSize>
<totalSizeCap>20GB</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<encoder>
<Pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</Pattern>
</encoder>
</appender>
<logger name="org.apache" level="ERROR"/>
<logger name="ch.qos.logback" level="INFO"/>
<root level="INFO">
<appender-ref ref="STDOUT"/>
<appender-ref ref="RollingFile"/>
</root>
</configuration>
you are using monthly rollover: "%d{yyyy-MM}" so maxhistory=3 will keep three months of logs.

logback how to store log file in folder having name as current date

following logback.xml will create a log file, but i want to create new folder every day which have same name as current date and store the new log file
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<property name="DEV_HOME" value="/home/gaurav/flinklogs" />
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${DEV_HOME}/logFile.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<!-- daily rollover -->
<fileNamePattern>logFile.%d{yyyy-MM-dd}.log</fileNamePattern>
<!-- keep 30 days' worth of history capped at 3GB total size -->
<maxHistory>30</maxHistory>
<totalSizeCap>3GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%-4relative [%thread] %-5level %logger{35} - %msg%n</pattern>
</encoder>
</appender>
<root level="ERROR">
<appender-ref ref="FILE" />
</root>
</configuration>
I also tried following filenamepattern but its not working
<fileNamePattern>${DEV_HOME}/%d{yyyy/MM, aux}/logFile.%d{yyyy-MM-dd}.log</fileNamePattern>
it is creatin log file in home/gaurav/flinklogs/logFile.log
If both <file> and <fileNamePattern> are specified, the the current log file is located as specified in <file> and archive log files are located as specified in the <fileNamePattern> - see documentation.
You need to remove <file>${DEV_HOME}/logFile.log</file> and then change <fileNamePattern>logFile.%d{yyyy-MM-dd}.log</fileNamePattern> to <fileNamePattern>${DEV_HOME}/%d{yyyy/MM, aux}/logFile.%d{yyyy-MM-dd}.log</fileNamePattern> and it should work the way you want it to.
I made a sample code using the answer of "Konrad Botor".
Refer to the below code.
<appender name="dailyAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${payloadLoggingFilePath}-%d{yyyy-MM-dd}.log</fileNamePattern>
</rollingPolicy>
<encoder>
<pattern>%m%n</pattern>
</encoder>

Loggly logback appender is really slow

I'm using the Loggly logback appender as detailed in their setup guide:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="LOGGLY" class="ch.qos.logback.ext.loggly.LogglyAppender">
<endpointUrl>https://logs-01.loggly.com/inputs/MY_TOKEN/tag/logback</endpointUrl>
<pattern>%d{"ISO8601", UTC} %p %t %c %M - %m%n</pattern>
</appender>
<root level="INFO">
<appender-ref ref="LOGGLY" />
</root>
</configuration>
Everything is working as expected (logs appearing in Loggly) but it is incredibely slow, about 1 second per log message. It's bought my application all but to a halt. Is there a performance tweak i'm missing?
I've found the GitHub page for the LogglyAppender and used the LogglyBatchAppender instead of the one recommended by the Loggly doco. This seems to have solved the problem with long blocks writing the log message:
<appender name="LOGGLY" class="ch.qos.logback.ext.loggly.LogglyBatchAppender">
<endpointUrl>https://logs-01.loggly.com/bulk/MY_TOKEN/tag/admin</endpointUrl>
<pattern>%d{"ISO8601", UTC} %p %t %c %M - %m%n</pattern>
<flushIntervalInSeconds>2</flushIntervalInSeconds>
</appender>
The syslog appender is also pretty fast https://www.loggly.com/docs/java-logback-syslog/

making logback filename in TimeBasedRollingPolicy use leading 0s for %i

Is there any way to make logback create TimeBasedRollingPolicy file names that use leading 0s for %i. Here's an excerpt from the logback.xml file
...
<rollingPolicy class="ch.quos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>logs/foo_%d{yyyyMMdd}-%i.log</fileNamePattern>
...
</rollingPolicy>
...
This creates files named like this foo_20130501-0.log, foo_20130501-1.log, ... foo_20130501-9.log, foo_20130501-10.log, ...
Instead, I would like the files named like this: foo_20130501-000.log, foo_20130501-001.log, ... foo_20130501-009.log, foo_20130501-010.log, ...
No, as of 2013-05-01, this is not possible with logback version 1.0.12 or earlier. Please create a jira issue requesting this feature.

How to disable printing the Logback pattern in log files

I noticed that Logback prints the pattern that it uses to format the log entries at the top of every log file.
For instance, I see this at the top of each log file:
#logback.classic pattern: %d [%t] %-5p %c - %m%n
How can I disable this?
Setting outputPatternAsPresentationHeader to false would work. However, the issue is addressed in logback version 1.0.3 where outputPatternAsPresentationHeader is set to false by default.
Okay, I found the problem was answered on the logback-user mailing list.
I added this:
<outputPatternAsPresentationHeader>false</outputPatternAsPresentationHeader>
To the <encoder> element, below the <pattern> element, and my problem was solved!