Logback : append log message to outputstream appender - logback

I'm using Logback framework v1.0.1 to do logging. I would like to know how to append the log message to an outputstream in java.
I wanted to format the log message into key=value pairs At the end i want to getformatted log message as an output stream. I retrieved the logger instance and log the message at debug level.
ch.qos.logback.classic.Logger logger = (ch.qos.logback.classic.Logger) LoggerFactory.getLogger("SplunkSearch.SplunkLogger");
logger.info( "wrap = true, setValue = false,");
logger.debug( "wrap = true, setValue = false,");
The logback.xml configuration file is as follows :
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
<timestamp key="bySecond" datePattern="yyyyMMdd'T'HHmmss" timeReference="contextBirth"/>
<contextName>splunksearchcontext</contextName>
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<Target>System.out</Target>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<appender name= "outputstream" class="ch.qos.logback.core.OutputStreamAppender">
<encoder>
<pattern></pattern>
</encoder>
</appender>
<root level="debug">
<appender-ref ref="stdout"/>
<appender-ref ref="outputstream" />
</root>
</configuration>
The output is as follows when i run.
i saw this line in the status message that is sort of error. No outputstream set.
15:07:19,399 |-ERROR in ch.qos.logback.core.OutputStreamAppender[outputstream] - No
output stream set for the appender named "outputstream".
15:07:19.414 [main] INFO SplunkSearch.SplunkLogger - wrap = true, setValue = false,
15:07:19.430 [main] DEBUG SplunkSearch.SplunkLogger - wrap = true, setValue = false,

It said "No output stream set for the appender named "outputstream"." You defined an appender, but not output stream in it. I don't know where to write the message.
logback manual said that you should not use outputstream appender directly. And from the manual, I also didn't find any way to configure the output stream into this appender.
If you really want to use it, I think the only way is that add it in your program.

Related

How to silence the logger from a specific library

I am using a library in my code which has the following logback.xml
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%date - %-5p %t %-25logger{0} %F:%L %m%n</pattern>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="STDOUT" />
</root>
</configuration>
I have no control over this .xml file because the library author owns this file and I just use her library as a jar file.
Now when I use the library as a jar, I see lots of "INFO" statements in my output. I want to shut the output of the logger just from this library.
I don't want to globally switch off (or raise severity of logs) for my application. I just want to silence the logs from this library.
How can I do that?
You can specify the location of your logback.xml as a system property, then you can change it.
As the logback document says:
Specifying the location of the default configuration file as a system property
You may specify the location of the default configuration file with a system property named "logback.configurationFile". The value of this property can be a URL, a resource on the class path or a path to a file external to the application.
java -Dlogback.configurationFile=/path/to/config.xml chapters.configuration.MyApp1
Note that the file extension must be ".xml" or ".groovy". Other extensions are ignored. Explicitly registering a status listener may help debugging issues locating the configuration file.
UPDATE
To suppress logs from a specified package, you can define a logger in the logback.xml.
E.g. To suppress logs from "io.netty" package, add <logger name="io.netty" level="WARN"/> to your logback.xml.
I was able to solve the problem. I am listing my solution here for others.
So the problem is that we have multiple libraries and each has its own logger. we want the output from some and not from others. We can easily simulate this by the example below
package com.abhi
import org.slf4j.LoggerFactory
object Program extends App {
val f = new Foo()
val b = new Bar()
f.sayHello("Test1")
b.sayHello("Test2")
}
class Foo {
val logger = LoggerFactory.getLogger(classOf[Foo])
def sayHello(name: String) : String = {
logger.debug(s"++++++++++++ came inside Foo sayHello(${name}) +++++++++++++++++++++")
"Hello " + name
}
}
class Bar {
val logger = LoggerFactory.getLogger(classOf[Bar])
def sayHello(name: String) : String = {
logger.debug(s"++++++++++++ came inside Bar sayHello(${name}) +++++++++++++++++++++")
"Hello " + name
}
}
Now let's say we want to keep the output from Foo logger but don't want the output from Bar logger.
We will adjust our logback.xml like
<configuration debug="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<logger name="com.abhi.Foo" level="TRACE">
<appender-ref ref="STDOUT" />
</logger>
<root level="off" />
</configuration>
Here we are setting a logger explicitly for foo and setting its level to trace and setting the root to off. All people who don't have specific loggers will go to root and will not be able to log anything.

Log4J2 Programmatic Configuration from XML byte stream

I am at the end of my tether with Log4J2, hopefully somebody can help. I have the following code to initialize Log4J2, pretty soon after startup:
try (InputStream configStream = new ByteArrayInputStream(writer.toString().getBytes("UTF-8"))) {
ConfigurationSource configurationSource = new ConfigurationSource(configStream);
Configurator.initialize(null, configurationSource);
}
Where writer is a StringWriter and toString() produces the following config (which I have validated is correct through other means):
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="C" target="SYSTEM_OUT">
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p %m%n"/>
</Console>
<RollingFile name="R" fileName="C:\temp\logfile.log" filePattern="C:\temp\logfile.log.%d{yyyy-MM-dd}">
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS z} %-5p [%t] %m%n"/>
<Policies>
<TimeBasedTriggeringPolicy interval="1" modulate="true"/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="somename" level="debug">
<appender-ref ref="R"/>
</Logger>
<Root level="debug" additivity="false">
<appender-ref ref="C" level="info"/>
</Root>
</Loggers>
</Configuration>
As you may have guessed, this does not work and I get no error message other than the expected:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
The reason I say this is expected is because I am configuring Log4J2 manually and have not suppressed this message yet.
Unfortunately, I cannot read the config from a file, for legacy reasons.
UPDATE 1 :
After taking Remko's advice, I added the following block before invoking the initialize method:
System.setProperty("log4j2.disable.jmx", "true");
StatusLogger status = StatusLogger.getLogger();
status.clear(); // remove old listeners that may prevent status output
status.setLevel(Level.TRACE);
status.reset(); // I could not see any trace info until I called this
status.trace("Status -- TRACE"); // I added this to prove that trace level logging was working
This gave me the following output:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
TRACE StatusLogger Status -- TRACE
DEBUG StatusLogger Stopping LoggerContext[name=sun.misc.Launcher$AppClassLoader#647e05, org.apache.logging.log4j.core.LoggerContext#a3defe]
DEBUG StatusLogger Stopping LoggerContext[name=sun.misc.Launcher$AppClassLoader#647e05, org.apache.logging.log4j.core.LoggerContext#a3defe]...
DEBUG StatusLogger Unregistering MBean org.apache.logging.log4j2:type=sun.misc.Launcher$AppClassLoader#647e05
DEBUG StatusLogger Unregistering MBean org.apache.logging.log4j2:type=sun.misc.Launcher$AppClassLoader#647e05,component=StatusLogger
DEBUG StatusLogger Unregistering MBean org.apache.logging.log4j2:type=sun.misc.Launcher$AppClassLoader#647e05,component=ContextSelector
DEBUG StatusLogger Unregistering MBean org.apache.logging.log4j2:type=sun.misc.Launcher$AppClassLoader#647e05,component=Appenders,name=Console
TRACE StatusLogger Stopping org.apache.logging.log4j.core.config.DefaultConfiguration#1a1440e...
TRACE StatusLogger AbstractConfiguration stopped 0 AsyncLoggerConfigs.
TRACE StatusLogger AbstractConfiguration stopped 0 AsyncAppenders.
DEBUG StatusLogger Shutting down OutputStreamManager SYSTEM_OUT
TRACE StatusLogger AbstractConfiguration stopped 1 Appenders.
TRACE StatusLogger AbstractConfiguration stopped 0 Loggers.
DEBUG StatusLogger Stopped org.apache.logging.log4j.core.config.DefaultConfiguration#9a6398 OK
DEBUG StatusLogger Stopped LoggerContext[name=sun.misc.Launcher$AppClassLoader#647e05, org.apache.logging.log4j.core.LoggerContext#9a6398]...
UPDATE 2 :
I decided to figure out a way to work with a file rather than a ByteArrayInputStream and got it working. FWIW, I think there is a bug in the Log4J2 code, when attempting to initialize using an InputStream, my theory:
In Log4jContextFactory the following method:
public LoggerContext getContext(final String fqcn, final ClassLoader loader, final Object externalContext,
final boolean currentContext, final ConfigurationSource source)
Has the following if statement, which always evaluates to false, which means the default config is always returned...
if (ctx.getState() == LifeCycle.State.INITIALIZED) {
if (source != null) {
ContextAnchor.THREAD_CONTEXT.set(ctx);
final Configuration config = ConfigurationFactory.getInstance().getConfiguration(source);
LOGGER.debug("Starting LoggerContext[name={}] from configuration {}", ctx.getName(), source);
ctx.start(config);
ContextAnchor.THREAD_CONTEXT.remove();
} else {
ctx.start();
}
}
At first glance I don't see why your configuration does not work. (You could try using forward slashes in the paths to make absolutely sure, but chances are that the slashes are not the problem.)
Can you try the following to generate more log4j2 debug output to see where the configuration goes wrong? Please post the result in your question.
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.status.StatusLogger;
// In an XML configuration you can just use <Configuration status="TRACE"...
// Here we use less elegant code to switch on status logging
// since we're not sure where things break down.
System.setProperty("log4j2.disable.jmx", "true");
StatusLogger status = StatusLogger.getLogger();
status.clear(); // remove old listeners that may prevent status output
status.setLevel(Level.TRACE);
// now configure log4j2...
// This should generate trace-level debug output to the console.
try (InputStream configStream = new ByteArrayInputStream(writer.toString().getBytes())) {
ConfigurationSource configurationSource = new ConfigurationSource(configStream);
Configurator.initialize(null, configurationSource);
}

Why is Mule exception strategy so chatty?

In my Mule app, I've configured some of the flows to use a catch exception strategy in order to do some special processing. For these cases, I want to pop the error and the original payload into an object store. Everywhere else, the default exception strategy is fine.
<flow name="saveLookup">
<vm:inbound-endpoint exchange-pattern="one-way" ref="Lookup_Save_VM" />
<component>
<spring-object bean="insertLookupMDCvalues"/>
</component>
<set-variable variableName="originalPayload" value="#[payload]"/>
<json:json-to-object-transformer returnClass="com.company.LookupData"/>
<set-variable variableName="transactionId" value="#[payload.transactionId]"/>
<transactional action="ALWAYS_BEGIN">
<logger message="${lookup.SQL}" level="INFO"/>
<jdbc:outbound-endpoint exchange-pattern="request-response" queryKey="saveLookup" queryTimeout="-1" connector-ref="JdbcConnector" />
<foreach collection="#[payload.transactional.lookupItems.items]">
<logger message="${lookup.item.SQL}" level="INFO" />
<jdbc:outbound-endpoint exchange-pattern="request-response" queryKey="saveLookupItem" queryTimeout="-1" connector-ref="JdbcConnector"/>
</foreach>
</transactional>
<component>
<spring-object bean="clearLookupMDCvalues"/>
</component>
<catch-exception-strategy>
<message-properties-transformer scope="invocation">
<add-message-property key="errorMap" value="#[['id' : transactionId, 'body' : originalPayload, 'error' : exception.summaryMessage]]"/>
</message-properties-transformer>
<choice>
<when expression="#[message.inboundProperties['resubmit']]">
<logger message="Resubmission of lookup data failed, saving to Dead Letter object store. ID=#[transactionId]" level="INFO"/>
<objectstore:store config-ref="lookupDeadLetterOS" key="#[transactionId]" overwrite="true" value-ref="#[errorMap]"/>
</when>
<otherwise>
<logger message="Saving lookup data failed, saving to Error object store. ID=#[transactionId]" level="INFO"/>
<objectstore:store config-ref="lookupErrorOS" key="#[transactionId]" overwrite="true" value-ref="#[errorMap]"/>
</otherwise>
</choice>
<set-payload value="Error: #[exception.summaryMessage]"/>
<component>
<spring-object bean="clearLookupMDCvalues"/>
</component>
</catch-exception-strategy>
</flow>
My problem is that when an error is encountered, let's say a Null Pointer Exception in the foreach component, I'm seeing four ERROR log statements for each event:
Exception stack is: 1. null (java.lang.NullPointerException) ...and so on. This is logged twice.
CatchMessagingExceptionStrategy - Message : Execution of the expression "payload.transactional.lookupItems.items" failed. (org.mule.api.expression.ExpressionRuntimeException). Message payload is of type: LookupData
DefaultMessagingExceptionStrategy - Message : Execution of the expression "payload.transactional.lookupItems.items" failed. (org.mule.api.expression.ExpressionRuntimeException). Message payload is of type: LookupData
I thought that a flow-specific exception strategy should override the default strategy. Why the duplicate log messages, and is there a way to shush them? I'd like to avoid having to configure the default exception strategy, as it's perfectly acceptable behavior in the majority of the flows.
The issue is that the built in exception strategies inherit from AbstractExceptionListener, and they all use the logException template method. The base implementation always logs at ERROR level, which is sometimes not appropriate for your application.
You can create a simple subclass of CatchMessagingExceptionStrategy that overrides the logException method, and logs however you want. Then, use it in your flow in place of the <catch-exception-strategy> like so:
<custom-exception-strategy class="com.mycompany.mule.QuietCatchExceptionStrategy">
<!-- your message processors here -->
</custom-exception-strategy>

unable to read env variables in logback

Does logback read environment variables during variable substitution? I'm getting REM_HOME_IS_UNDEFINED on the console during app startup. The variable exists though. The manual says that env variables are looked up last in the food chain. http://logback.qos.ch/manual/configuration.html#variableSubstitution
Here's what I'm doing:
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${REM_HOME}/reac/logs/reac.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>${REM_HOME}/reac/logs/archives/reac-%i.log.zip</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>3</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>5MB</maxFileSize>
</triggeringPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %logger{36} %-5level - %msg%n</pattern>
</encoder>
</appender>
Here's what I ended doing:
I'm reading the env variable and calling System.setProperty to set the value as a system variable and resetting the loggercontext.
System.setProperty(REM_HOME, remHome);
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
try {
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(context);
// Call context.reset() to clear any previous configuration, e.g. default
// configuration. For multi-step configuration, omit calling context.reset()
context.reset();
configurator.doConfigure(getClass().getClassLoader().getResourceAsStream(LOG_CONF_FILE));
}
catch (JoranException je) {
// StatusPrinter will handle this
je.printStackTrace();
}
StatusPrinter.printInCaseOfErrorsOrWarnings(context);

Why is log4j not behaving as expected?

I have a co-worker who is trying to get log4j to behave as follows:
Log to Stdout
By default, disable most output
Show only messages from java.sql.PrepareStatement at level debug and up
He's getting caught up in the 'level' vs 'priority'. Here is his config file:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "D:/Java/apache-log4j-1.2.15/src/main/resources/org/apache/log4j/xml/log4j.dtd" >
<log4j:configuration>
<!-- Appenders -->
<appender name="stdout" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%5p %d{ISO8601} [%t][%x] %c - %m%n" />
</layout>
</appender>
<!-- Loggers for ibatus and JDBC database -->
<logger name="java.sql.PreparedStatement">
<level value="debug"/>
</logger>
<!-- The Root Logger -->
<root>
<level value="error"/>
<appender-ref ref="stdout"/>
</root>
</log4j:configuration>
Simpler configuration as shown (Root Log Level = ERROR):
log4j: reset attribute= "false".
log4j: Threshold ="null".
log4j: Retreiving an instance of org.apache.log4j.Logger.
log4j: Setting [java.sql.PreparedStatement] additivity to [true].
log4j: Level value for java.sql.PreparedStatement is [debug].
log4j: java.sql.PreparedStatement level set to DEBUG
log4j: Level value for root is [error].
log4j: root level set to ERROR
log4j: Class name: [org.apache.log4j.ConsoleAppender]
log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"
log4j: Setting property [conversionPattern] to [%5p %d{ISO8601} [%t][%x] %c - %m%n].
log4j: Adding appender named [stdout] to category [root].
Configuration with Root Log Level changed to debug (replace queries with …)
log4j: reset attribute= "false".
log4j: Threshold ="null".
log4j: Retreiving an instance of org.apache.log4j.Logger.
log4j: Setting [java.sql.PreparedStatement] additivity to [true].
log4j: Level value for java.sql.PreparedStatement is [debug].
log4j: java.sql.PreparedStatement level set to DEBUG
log4j: Level value for root is [debug].
log4j: root level set to DEBUG
log4j: Class name: [org.apache.log4j.ConsoleAppender]
log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"
log4j: Setting property [conversionPattern] to [%5p %d{ISO8601} [%t][%x] %c - %m%n].
log4j: Adding appender named [stdout] to category [root].
DEBUG 2010-03-19 12:59:58,256 [main][] com.ibatis.common.jdbc.SimpleDataSource - Created connection 1309601.
DEBUG 2010-03-19 12:59:58,256 [main][] java.sql.Connection - {conn-100000} Connection
DEBUG 2010-03-19 12:59:58,256 [main][] java.sql.Connection - {conn-100000} Preparing Statement: …
DEBUG 2010-03-19 12:59:58,287 [main][] java.sql.PreparedStatement - {pstm-100001} Executing Statement: …
DEBUG 2010-03-19 12:59:58,287 [main][] java.sql.PreparedStatement - {pstm-100001} Parameters: [%ATL]
DEBUG 2010-03-19 12:59:58,287 [main][] java.sql.PreparedStatement - {pstm-100001} Types: [java.lang.String]
DEBUG 2010-03-19 12:59:58,366 [main][] java.sql.ResultSet - {rset-100002} ResultSet
DEBUG 2010-03-19 12:59:58,381 [main][] java.sql.ResultSet - {rset-100002} Header: …
DEBUG 2010-03-19 12:59:58,381 [main][] java.sql.ResultSet - {rset-100002} Result: …
DEBUG 2010-03-19 12:59:58,381 [main][] java.sql.ResultSet - {rset-100002} Result: …
DEBUG 2010-03-19 12:59:58,381 [main][] java.sql.ResultSet - {rset-100002} Result: …
DEBUG 2010-03-19 12:59:58,397 [main][] com.ibatis.common.jdbc.SimpleDataSource - Returned connection 1309601 to pool.
How does he need to change his log4j.xml config file to make it behave as he's expecting?
I looked at the source code for mybatis, shown below. You have to enable DEBUG on java.sql.Connection in order for it execute the logging for java.sql.PreparedStatement. I struggled with this all day!
private Connection wrapConnection(Connection connection) {
if (log.isDebugEnabled()) {
return ConnectionLogger.newInstance(connection);
} else {
return connection;
}
}
Insert the following addition into your log4j.xml:
<!-- *******************************************************
WARNING: iBatus 2.3.3 (only ver. tested) is a little weird.
YOU MUST SET
java.sql.Connection to debug to get any
java.sql.PreparedStatement debug logs.
************************************************************** -->
<logger name="java.sql.Connection" additivity="false">
<level value="debug"/>
</logger>
DEBUG 2010-03-19 14:27:08,425 [main][][java.sql.PreparedStatement] - {pstm-100001} Executing Statement:
...
I think the reason there are no log messages is that code you want to see logs from, doesn't use java.sql.PrepareStatement logger, but different loggers. Loggers are usually (although not necessarilly) named after classes that use them. I.e. com.ibatis.SomeClass would typically not use java.sql.PrepareStatement logger.
Set your root logger to DEBUG, and check out names of loggers that give you messages you want. Then configure these loggers with DEBUG and let root log at ERROR level only.
Btw, it's Prepare_d_Statement, it is interface (i.e. no logging code there), and it definitely doesn't use log4j since it is in JDK.