I am trying to output all my log events in JSON format.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSZ} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="INFO">
<!--<appender-ref ref="kafkaAppender" />-->
<appender-ref ref="STDOUT" />
</root>
Maven Dependency:
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>4.7</version>
</dependency>
I see an error while running this configuration. Did i miss anything?
here is the stacktrace:
Exception in thread "main" java.lang.AbstractMethodError: net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder.headerBytes()[B
at ch.qos.logback.core.OutputStreamAppender.encoderInit(OutputStreamAppender.java:180)
at ch.qos.logback.core.OutputStreamAppender.setOutputStream(OutputStreamAppender.java:171)
at ch.qos.logback.core.ConsoleAppender.start(ConsoleAppender.java:81)
at ch.qos.logback.core.joran.action.AppenderAction.end(AppenderAction.java:90)
at ch.qos.logback.core.joran.spi.Interpreter.callEndAction(Interpreter.java:309)
at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:193)
at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:179)
at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62)
at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:165)
at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:152)
at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:110)
at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:53)
at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:150)
at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:84)
at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273)
I had encountered the same error while using 4.7 version of the logstash-logback-encoder, and upgrading to 4.11 resolved the issue.
Ref- https://github.com/thumbtack/becquerel/issues/2
Related
I need to be able to log an Exception as a single record in my logs, which will make it much easier to investigate issues in Kibana / Elasticsearch. From what I can tell from the documentation for slf4j, the Logger interface requires messages to be Strings. Is my only option to remove newline characters from the Exception message before passing it to the Logger?
For context, I am using the following:
.m2/repository/org/slf4j/slf4j-api/1.7.28/slf4j-api-1.7.28.jar
Java 11
Sprint Boot version 2.1.8.RELEASE
This is a trimmed down version of my custom exception handler :
import my.error.Error; // custom Error class
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.servlet.mvc.method.annotation.ResponseEntityExceptionHandler;
#ControllerAdvice
public class CustomExceptionHandler extends ResponseEntityExceptionHandler {
private final Logger logger = LoggerFactory.getLogger(this.getClass());
private void logError(Error error, Exception ex){
logger.error(String.format("id: %s, message: %s", error.getId(), ex.getMessage()), ex);
}
}
Initially I had attempted to alter the logging behavior with changes to my logback.xml file, within src/main/java/resources. Unfortunately, this appears to do nothing, so my assumption now is that the CustomExceptionHandler that I am creating is overruling the specification set in the logback.xml file. Specifically, the <pattern> of the <encoder> has been changed based on other research. It's attempting to replace all newline characters.
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" debug="true">
<include resource="org/springframework/boot/logging/logback/base.xml" />
<appender name="FILE-ROLLING" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_PATH}/gateway.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/archived/gateway/gateway.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<!-- each archived file, size max 5MB -->
<maxFileSize>5MB</maxFileSize>
<!-- total size of all archive files, if total size > 10GB, it will delete old archived file -->
<totalSizeCap>10GB</totalSizeCap>
<!-- 30 days to keep -->
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder>
<pattern>%d %p %c{1.} [%t] %m MULTIEXCEPTION %replace(%xException){'\n','\u2028'}%nopex%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="FILE-ROLLING" level="DEBUG" additivity="false"/>
</root>
<springProfile name="local">
<logger name="my.gateway" level="TRACE" additivity="false">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE-ROLLING" />
</logger>
<logger name="com.netflix" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="org.springframework" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="com" level="INFO" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="gov" level="INFO" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
<logger name="org" level="INFO" additivity="false">
<appender-ref ref="CONSOLE" />
</logger>
</springProfile>
</configuration>
Links
Logback Docs
SLF4J Docs
SLF4J Manual
Combine Logback and SLF4J
Logback and SLF4J - StructuredArguments
JSON Logging w/ Logback
Collapse Logs - Multiple Lines into One
Apache Log4j Layouts
SO : Override Logback 1
SO : Override Logback 2
SO : Make Logback Output JSON
Format SLF4J
Collapse multi-line logs into one with Logback or Log4j2
Baeldung Log4j2 JSON Logging
Java Logging Guide
I changed my logError method to this:
import org.slf4j.MDC; // new import to add property
// ... other imports from before
private void logError(Error error, Exception exception){
MDC.put("error id", error.getId().toString()); // add a new property to the thread context
LOGGER.error(exception.getMessage(), exception);
MDC.clear(); // remove all new properties after logging the error
}
My new pom.xml File Dependencies:
<dependencies>
<!-- logging -->
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>5.2</version>
</dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-jackson</artifactId>
<version>0.1.5</version>
</dependency>
<!-- ch.qos.logback.contrib.json.classic.JsonLayout -->
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
<version>0.1.5</version>
</dependency>
<!-- Other dependencies -->
<!-- ... -->
<!-- ... -->
</dependencies>
New logback.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="FILE-ROLLING">
<!-- Old encoder is removed:
<encoder>
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
</encoder>
-->
<file>${LOG_PATH}/data.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_PATH}/archived/data/data.%d{yyyy-MM-dd}.%i.log.gz
</fileNamePattern>
<!-- each archived file, size max 10MB -->
<maxFileSize>10MB</maxFileSize>
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
<maxHistory>60</maxHistory>
<!-- 60 days to keep -->
<totalSizeCap>20GB</totalSizeCap>
</rollingPolicy>
<!-- TODO : configure a pretty printer -->
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<!-- TODO : configure a pretty printer -->
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<provider class="net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider"/>
</encoder>
</appender>
<!-- The base logback configuration is removed, but note that this will also remove the default Spring start-up messages, so those need to be added back into the <root>
<include resource="org/springframework/boot/logging/logback/base.xml"/>
-->
<root level="INFO">
<appender-ref ref="FILE-ROLLING"/>
<appender-ref ref="CONSOLE"/> <!-- This adds Spring start-up messages back into the logs -->
</root>
<springProfile name="local">
<logger additivity="false" level="TRACE" name="org.springframework.web">
<appender-ref ref="CONSOLE"/>
</logger>
</springProfile>
<springProfile name="dev">
<logger additivity="false" level="DEBUG" name="org.springframework">
<appender-ref ref="FILE-ROLLING"/>
</logger>
</springProfile>
<springProfile name="prod">
<logger additivity="false" level="DEBUG" name="my.app.path">
<appender-ref ref="FILE-ROLLING"/>
</logger>
<logger additivity="false" level="DEBUG" name="org.jooq">
<appender-ref ref="FILE-ROLLING"/>
</logger>
<logger additivity="false" level="DEBUG" name="org.springframework">
<appender-ref ref="FILE-ROLLING"/>
</logger>
</springProfile>
</configuration>
SLF4J Docs - Mapped Diagnostic Context (MDC)
Baeldung Example - MDC + Log4J2
i just started with Java and i'm trying to configure the logging via logback.
The logging does not work, instead i get an an error in the console:
-ERROR in ch.qos.logback.core.joran.spi.Interpreter#9:29 - no applicable action for [root], current ElementPath is [[configuration][appender][root]]
so no appender is found:
-WARN in Logger[Test01] - No appenders present in context [default] for logger [Test01].
My code is very simple:
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MainClass {
public static void main(String[] args) {
Logger log = LoggerFactory.getLogger(MainClass.class);
log.debug("test"); }
}
Complete console output when running looks like this:
08:41:43,047 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
08:41:43,047 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
08:41:43,047 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/C:/Users/klein/IdeaProjects/Maven01ArtifactId/target/classes/logback.xml]
08:41:43,196 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
08:41:43,199 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
08:41:43,209 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
08:41:43,235 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter#9:29 - no applicable action for [root], current ElementPath is [[configuration][appender][root]]
08:41:43,235 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter#10:41 - no applicable action for [appender-ref], current ElementPath is [[configuration][appender][root][appender-ref]]
08:41:43,235 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
08:41:43,235 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator#78e94dcf - Registering current configuration as safe fallback point
08:41:43,235 |-WARN in Logger[Test01] - No appenders present in context [default] for logger [Test01].
I'm using IntelliJ Community 2018.3 and started a project with Maven. logback.xml and pom.xml are below.
Any ideas what is wrong here?
logback.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<configuration debug="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
</encoder>
<root level="DEBUG">
<appender-ref ref="STDOUT"/>
</root>
</appender>
</configuration>
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>xxx</groupId>
<artifactId>xxx</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<slf4jVersion>1.7.25</slf4jVersion>
<logbackVersion>1.2.3</logbackVersion>
</properties>
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4jVersion}</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<version>${logbackVersion}</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>${logbackVersion}</version>
</dependency>
</dependencies>
</project>
In logback.xml, you put the <root> section inside the <appender> section.
Your logback.xml should be:
<?xml version="1.0" encoding="UTF-8" ?>
<configuration debug="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="STDOUT"/>
</root>
</configuration>
I would like to create a console appender that displays some log info, and also prints out a particular http header, similar to this:
> [INFO] { "time": "2017-08-31 12:14:32,583", "app-id": "my-app", "my-header": "my-header-value" } -- "Hello, World"
I have created a logback-spring.xml file like below, but "my-header" just prints out blank.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<springProperty name="appId" source="spring.app.application_id"/>
<!-- Appender to log to console -->
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<!-- Minimum logging level to be presented in the console logs-->
<level>INFO</level>
</filter>
<encoder>
<pattern>
%clr(%5p) %clr({ "time": "%date{ISO8601}", "app-id": "${appId}", "my-header": "%X{my-header}"}){faint} -- %msg%n
</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="console"/>
</root>
</configuration>
I have read that using logback-access gives you access to HTTP request/response properties, but when I try setting the encoder class I cannot use any of the classic logback conversion words:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<!-- Appender to log to console -->
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<!-- Minimum logging level to be presented in the console logs-->
<level>INFO</level>
</filter>
<encoder class="ch.qos.logback.access.PatternLayoutEncoder">
<pattern class="ch.qos.logback.access.PatternLayoutEncoder">
%clr(%5p) %clr({ "time": "%date{ISO8601}", "app-id": "${appId}", "my-header": "%header{my-header}"}){faint} -- %msg%n
</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="console"/>
</root>
</configuration>
The logback above gives these errors:
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:1.4.0.RELEASE:run (default-cli) on project pd-thundera-server: An exception occurred while running. null: InvocationTargetException: Logback configuration error detected:
[ERROR] ERROR in ch.qos.logback.core.pattern.parser.Compiler#4782f0a4 - There is no conversion class registered for conversion word [p]
[ERROR] ERROR in ch.qos.logback.core.pattern.parser.Compiler#4782f0a4 - [p] is not a valid conversion word
[ERROR] ERROR in ch.qos.logback.core.pattern.parser.Compiler#6071227e - There is no conversion class registered for conversion word [msg]
[ERROR] ERROR in ch.qos.logback.core.pattern.parser.Compiler#6071227e - [msg] is not a valid conversion word
How can I access a request header?
This configuration worked for me:
<encoder>
<charset>utf-8</charset>
<pattern>%t{yyyy-MM-dd HH:mm:ss,SSS} %h X-Forwarded-For: %header{X-Forwarded-For} "%r", Response status:%s, Bytes sent:%b, Response time:%D</pattern>
</encoder>
Name of the header goes in between curly braces, e.g. %header{Content-type} or %header{My-header}, or just %header if you want all headers to be logged.
Source:
https://logback.qos.ch/access.html#configuration
https://logback.qos.ch/manual/layouts.html#AccessPatternLayout
Please find below example with Spring
If you want to log http headers then you need to use MDC feature: http://logback.qos.ch/manual/mdc.html
create Filter:
#Component
class RequestHeaderFilterConfig : Filter {
private val xRequestId = "X-Request-Id"
override fun doFilter(request: ServletRequest?, response: ServletResponse?, chain: FilterChain?) {
val httpRequest = request as HttpServletRequest
MDC.put(xRequestId, httpRequest.getHeader(xRequestId))
chain?.doFilter(request, response)
}
override fun destroy() = MDC.remove(xRequestId)
}
Controller:
#PostMapping
fun operation(#RequestHeader(value = "X-Request-Id", required = false) xRequestId: String? = null): ResponseEntity<Output> {
....
}
logback.xml
<Pattern>%d{dd-MM-yyyy HH:mm:ss.SSS} [%thread] %-5level [X-Request-Id: %X{X-Request-Id}] %logger{36}.%M:%line - %msg%n</Pattern>
I have a defined logback in my project, and here is the configuration:
<?xml version="1.0" encoding="UTF-8"?>
<appender name="DAILY" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/logApp/application.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>/logApp/application.log.%d{yyyy-MM-dd}.%i.gz</fileNamePattern>
</rollingPolicy>
<encoder>
<charset>UTF-8</charset>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36}:%line - %msg%n</pattern>
</encoder>
</appender>
<appender name="HOURLY" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/logApp/application.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>/logApp/application.log.%d{yyyy-MM-dd-HH}.%i.gz</fileNamePattern>
</rollingPolicy>
<encoder>
<charset>UTF-8</charset>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36}:%line - %msg%n</pattern>
</encoder>
</appender>
<root level="info">
<appender-ref ref="DAILY" />
</root>
A you can see I only use daily appender in my root configuration. How can I switch between those defined appenders (HOURLY / DAILY) in runtime?
I have tried with the following code, but my newAppender is always null:
String loggingAppender = "HOURLY";
LoggerContext lc = (LoggerContext) LoggerFactory.getILoggerFactory();
ch.qos.logback.classic.Logger root = lc.getLogger(Logger.ROOT_LOGGER_NAME);
root.detachAndStopAllAppenders();
Appender<ILoggingEvent> newAppender = root.getAppender(loggingAppender);
root.addAppender(newAppender);
I have the following configuration file that is very similar to the standard example in the Logback manual. The only difference is the addition of [%F:%L]. while everything works, %F and %L do not work. If I remove the async appender and log directly using the file appender, everything works just great.
can somebody explain what is going on? And how to print the file name and line number as these two parameters are supposed to?
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>myapp.log</file>
<encoder><pattern>%logger{35} - [%F:%L] - %msg%n</pattern></encoder>
</appender>
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
</appender>
<root level="DEBUG"><appender-ref ref="ASYNC" /></root>
</configuration>
You need to set AsyncAppender's includeCallerData property to true. Here is the modified config file:
<configuration>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>myapp.log</file>
<encoder><pattern>%logger{35} - [%F:%L] - %msg%n</pattern></encoder>
</appender>
<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="FILE" />
<!-- add the following line -->
<includeCallerData>true</includeCallerData>
</appender>
<root level="DEBUG"><appender-ref ref="ASYNC" /></root>
</configuration>
I post same answer in groovy format for someone who want groovy style like me.
appender('FILE', ch.qos.logback.core.FileAppender) {
file = 'myapp.log'
encoder(PatternLayoutEncoder) {
pattern = '%logger{35} - [%F:%L] - %msg%n'
}
}
appender('ASYNC', ch.qos.logback.classic.AsyncAppender) {
appenderRef('FILE')
//add the following line
includeCallerData = true
}
root(DEBUG, ['ASYNC'])