I have configured my Rollbar Appender in logback-spring.xml:
<appender name="Rollbar" class="com.rollbar.logback.RollbarAppender">
<accessToken>${ROLLBAR_TOKEN}</accessToken>
<environment>${SPRING_PROFILES_ACTIVE}</environment>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>ERROR</level>
</filter>
</appender>
and activated it for required profile:
<!-- debug level for dev -->
<springProfile name="dev">
<root level="debug">
<appender-ref ref="Console"/>
<appender-ref ref="Rollbar"/>
</root>
</springProfile>
everything works fine, and I can receive errors in Rollbar except one issue. There are some errors that I do not want to see in Rollbar but I still may see them in console, e.g.:
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
How I can filter such messages in the most convenient way?
The solution is to create you own implementation of ch.qos.logback.core.filter.Filter:
public class RegexRollbarFilter extends Filter<ILoggingEvent> {
private String regex;
private boolean includeThrowableMessage = true;
#Override
public FilterReply decide(ILoggingEvent event) {
if (!isStarted()) {
return FilterReply.NEUTRAL;
}
IThrowableProxy throwableProxy = event.getThrowableProxy();
if (event.getMessage()
.matches(regex) || includeThrowableMessage && throwableProxy != null && throwableProxy.getMessage()
.matches(regex)) {
return FilterReply.DENY;
} else {
return FilterReply.NEUTRAL;
}
}
public void setRegex(String regex) {
this.regex = regex;
}
public void setIncludeThrowableMessage(boolean includeThrowableMessage) {
this.includeThrowableMessage = includeThrowableMessage;
}
public void start() {
if (this.regex != null) {
super.start();
}
}
}
and use it in the follwoing way:
<appender name="Rollbar" class="com.rollbar.logback.RollbarAppender">
<accessToken>${ROLLBAR_TOKEN}</accessToken>
<environment>${SPRING_PROFILES_ACTIVE}</environment>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>ERROR</level>
</filter>
<filter class="your.package.RegexRollbarFilter">
<regex>^.*An existing connection was forcibly closed by the remote host.*$</regex>
</filter>
<filter class="your.package.RegexRollbarFilter">
<regex>(.|\n)*readAddress\(..\) failed: Connection reset by peer(.|\n)*$</regex>
</filter>
</appender>
I'm attempting to serialize my logs to Json. I'm using log4net with a custom layout, but when an exception is logged, I get the following malformed JSON (Note the additional stacktrace info at the end)
Am I missing a setting for log4net, or is this a serialization issue?
UPDATE: This has to be something with log4net, because json.net serializes an Exception perfectly.
***UPDATE (FIXED): Updated code below.
{
"UserSessionId":"4b146c92-fe99-4f78-bbef-720df2cf7473",
"ProcessSessionId":1,
...
"Logger":"testharness.Program","ThreadName":"1",
"ExceptionObject":{
"ClassName":"System.ApplicationException",
"Message":"Test Exception Logging",
"Data":null,
"InnerException":null,
"HelpURL":null,
"StackTraceString":" at testharness.Program.Main(String[] args) in C:\\temp\\testharness\\Program.cs:line 18",
"RemoteStackTraceString":null,
...
"WatsonBuckets":null
},
...
"log4net:HostName":"ol-4RBNMH2"
}}
System.ApplicationException: Test Exception Logging
at testharness.Program.Main(String[] args) in C:\temp\testharness\Program.cs:line 18
log4net configuratiton
<log4net>
<appender name="RollingFileCompositeAppender" type="log4net.Appender.RollingFileAppender">
<file value="c:\\logs\\testharness.txt"/>
<appendToFile value="true"/>
<rollingStyle value="Composite"/>
<datePattern value="yyyy-MM-dd"/>
<maxSizeRollBackups value="-1"/>
<maximumFileSize value="1MB"/>
<countDirection value="1"/>
<preserveLogFileNameExtension value="false"/>
<staticLogFileName value="false"/>
<layout type="Company.log4net.JsonLayout"></layout>
</appender>
<root>
<level value="ALL"/>
<appender-ref ref="RollingFileCompositeAppender"/>
</root>
</log4net>
The Custom Layout class
public class JsonLayout : LayoutSkeleton
{
public JsonLayout() {
IgnoresException = false;
}
...
/// <inheritdoc />
public override void ActivateOptions()
{
}
/// <inheritdoc />
public override void Format(TextWriter writer, LoggingEvent loggingEvent)
{
_customProperties.PhysicalMemory = Process.GetCurrentProcess().WorkingSet64;
var evt = new CustomLoggingEvent(loggingEvent, _customProperties);
writer.Write(JsonConvert.SerializeObject(evt));
}
}
[JsonObject(MemberSerialization.OptIn)]
public class CustomLoggingEvent
{
...
[JsonProperty]
public Exception ExceptionObject { get; set; }
[JsonProperty]
public long PhysicalMemory { get; set; }
[JsonProperty]
public PropertiesDictionary Properties { get; set; }
}
The Test Harness:
internal class Program
{
private static void Main()
{
Console.WriteLine($"{typeof(JsonLayout)}");
var log = log4net.LogManager.GetLogger(typeof(Program));
try
{
log.Debug("hello again world");
throw new ApplicationException("Test Exception Logging");
}
catch (Exception e)
{
log.Error("Exception Thrown", e);
}
Console.ReadLine();
}
}
I downloaded the log4net source and found the issue. When creating a custom layout implementing LayoutSkeleton, if your layout handles the LoggingEvent.ExceptionObject, then the IgnoresException property should be set to false; the default value is true. I updated the code in the question with a constructor wherein the IgnoresException property is set to true. I probably should have waited a day while I researched, but maybe this will help someone else.
before I upgrade the struts from 2.3.x to 2.5.8, the JSON can return data to me,but now return JSON value is empty in struts2.5.8, whatever what type i have put. here is my code:
TestController.java
public class TestingController implements ModelDriven<Object> {
private ArrayList<Test> testList = null;
public ArrayList<Test> getTestList() { return testList;}
public String tableData() {
jsonMap = new HashMap<String, Object>();
testList = getTestListBySomething();
if(testList!=null && testList.size()>0){
jsonMap.put("test", testList);
}
}
also my struts.xml:
<package name="default" namespace="" extends="rest-default,struts-default" strict-method-invocation="false">
<result-types>
<result-type name="json" class="org.apache.struts2.json.JSONResult"/>
</result-types>
</package>
I have checked that the testList have data and size, but in the response on ajax , it show:
{"test":""}
What wrong of my coding?
In your strust xml add below parameter.
<param name="root">#action</param>
EX :
<result-types>
<result-type name="json" class="org.apache.struts2.json.JSONResult">
<param name="noCache">true</param>
<param name="excludeNullProperties">true</param>
<param name="root">#action</param>
<param name="ignoreHierarchy">false</param>
<param name="excludeProperties">errors</param>
</result-type>
</result-types>
If I put a newline before the exception stack trace in the logging pattern:
<Pattern>%d{HH:mm:ss.SSS} %-5level [%thread] %logger{36} %msg %n %rEx{short} %n</Pattern>
then an extra newline is printed, so normal logging statements end up with blank lines in between them.
If I remove the extra newline:
<Pattern>%d{HH:mm:ss.SSS} %-5level [%thread] %logger{36} %msg %rEx{short} %n</Pattern>
then stack traces do not start on a new line, making them hard to read.
Can anyone think of a way to get logback to only print the extra newline when there is a stack trace, and not otherwise?
I have a working solution, but I dislike it because it's overengineered and should be possible more elegantly.
It's a specific case of a more generic problem: output a pattern only if a certain evaluator (one that checks the presence of an exception) matches. I got the implementation from somewhere here on SO long ago.
Requires janino. If anyone more logback-experienced than me can trim this down to a more reasonable amount of code, please do.
IfPresentConverter.java
public class IfPresentConverter extends CompositeConverter<ILoggingEvent> {
private final List<EventEvaluator<ILoggingEvent>> evaluatorList = new ArrayList<>();
private int errorCount = 0;
#Override
#SuppressWarnings("unchecked")
public void start() {
final List<String> optionList = getOptionList();
final Map<?, ?> evaluatorMap = (Map<?, ?>) getContext().getObject(CoreConstants.EVALUATOR_MAP);
for (final String evaluatorStr : optionList) {
final EventEvaluator<ILoggingEvent> ee = (EventEvaluator<ILoggingEvent>) evaluatorMap.get(evaluatorStr);
if (ee != null) {
evaluatorList.add(ee);
}
}
if (evaluatorList.isEmpty()) {
addError("At least one evaluator is expected, but you have declared none.");
return;
}
super.start();
}
#Override
public String convert(final ILoggingEvent event) {
boolean evalResult = true;
for (final EventEvaluator<ILoggingEvent> ee : evaluatorList) {
try {
if (!ee.evaluate(event)) {
evalResult = false;
break;
}
} catch (final EvaluationException eex) {
evalResult = false;
errorCount++;
if (errorCount < CoreConstants.MAX_ERROR_COUNT) {
addError("Exception thrown for evaluator named [" + ee.getName() + "].", eex);
} else {
final ErrorStatus errorStatus = new ErrorStatus("Exception thrown for evaluator named [" + ee.getName() + "].", this, eex);
errorStatus.add(new ErrorStatus("This was the last warning about this evaluator's errors. " + "We don't want the StatusManager to get flooded.", this));
addStatus(errorStatus);
}
}
}
if (evalResult) {
return super.convert(event);
} else {
return CoreConstants.EMPTY_STRING;
}
}
#Override
protected String transform(final ILoggingEvent event, final String in) {
return in;
}
}
logback.xml
<configuration>
<conversionRule conversionWord="ifPresent" converterClass="org.example.IfPresentConverter"/>
<evaluator name="has_ex">
<expression>return throwableProxy != null;</expression>
</evaluator>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>lorem ipsum %msg{}%ifPresent(\n%ex{full}){has_ex}%n</pattern>
</encoder>
</appender>
<root>
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
I need to be able to search an event for any one of a number of patterns and replace the text in the pattern with a masked value. This is a feature in our application intended to prevent sensitive information falling into the logs. As the information can be from a large variety of sources, it is not practical to apply filters on all the inputs. Besides there are uses for toString() beyond logging and I don't want toString() to uniformly mask for all calls (only logging).
I have tried using the %replace method in logback.xml:
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %replace(%msg){'f k\="pin">(.*?)</f','f k\="pin">**********</f'}%n</pattern>
This was successful (after replacing the angle brackets with character entities), but it can only replace a single pattern. I would also like to perform the equivalent of
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %replace(%msg){'pin=(.*?),','pin=**********,'}%n</pattern>
at the same time, but cannot. There is no way to mask two patterns in the one %replace.
The other way that has been loosely discussed on the interblags is extending something on the appender/encoder/layout hierarchy, but every attempt to intercept the ILoggingEvent has resulted in a collapse of the whole system, usually through instantiation errors or UnsupportedOperationException.
For example, I tried extending PatternLayout:
#Component("maskingPatternLayout")
public class MaskingPatternLayout extends PatternLayout {
#Autowired
private Environment env;
#Override
public String doLayout(ILoggingEvent event) {
String message=super.doLayout(event);
String patternsProperty = env.getProperty("bowdleriser.patterns");
if( patternsProperty != null ) {
String[] patterns = patternsProperty.split("|");
for (int i = 0; i < patterns.length; i++ ) {
Pattern pattern = Pattern.compile(patterns[i]);
Matcher matcher = pattern.matcher(event.getMessage());
matcher.replaceAll("*");
}
} else {
System.out.println("Bowdleriser not cleaning! Naughty strings are getting through!");
}
return message;
}
}
and then adjusting the logback.xml
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<layout class="com.touchcorp.touchpoint.utils.MaskingPatternLayout">
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>logs/touchpoint.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>logs/touchpoint.%i.log.zip</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>3</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10MB</maxFileSize>
</triggeringPolicy>
<encoder>
<layout class="com.touchcorp.touchpoint.utils.MaskingPatternLayout">
<pattern>%date{YYYY-MM-dd HH:mm:ss} %level [%thread] %logger{10} [%file:%line] %msg%n</pattern>
</layout>
</encoder>
</appender>
<logger name="com.touchcorp.touchpoint" level="DEBUG" />
<logger name="org.springframework.web.servlet.mvc" level="TRACE" />
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="STDOUT" />
</root>
</configuration>
I have tried many other insertions, so I was wondering if anyone has actually achieved what I am attempting and if they could provide any clues or a solution.
You need to wrap layout using LayoutWrappingEncoder. And also I believe you cannot use spring here as logback is not managed by spring.
Here is the updated class.
public class MaskingPatternLayout extends PatternLayout {
private String patternsProperty;
public String getPatternsProperty() {
return patternsProperty;
}
public void setPatternsProperty(String patternsProperty) {
this.patternsProperty = patternsProperty;
}
#Override
public String doLayout(ILoggingEvent event) {
String message = super.doLayout(event);
if (patternsProperty != null) {
String[] patterns = patternsProperty.split("\\|");
for (int i = 0; i < patterns.length; i++) {
Pattern pattern = Pattern.compile(patterns[i]);
Matcher matcher = pattern.matcher(event.getMessage());
if (matcher.find()) {
message = matcher.replaceAll("*");
}
}
} else {
}
return message;
}
}
And sample logback.xml
<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
<file>c:/logs/kp-ws.log</file>
<append>true</append>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.kp.MaskingPatternLayout">
<patternsProperty>.*password.*|.*karthik.*</patternsProperty>
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="fileAppender1" />
</root>
UPDATE
Here its better approach, set Pattern during init itself. such that we can avoid recreating Pattern again and again and this implementation is close to realistic usecase.
public class MaskingPatternLayout extends PatternLayout {
private String patternsProperty;
private Optional<Pattern> pattern;
public String getPatternsProperty() {
return patternsProperty;
}
public void setPatternsProperty(String patternsProperty) {
this.patternsProperty = patternsProperty;
if (this.patternsProperty != null) {
this.pattern = Optional.of(Pattern.compile(patternsProperty, Pattern.MULTILINE));
} else {
this.pattern = Optional.empty();
}
}
#Override
public String doLayout(ILoggingEvent event) {
final StringBuilder message = new StringBuilder(super.doLayout(event));
if (pattern.isPresent()) {
Matcher matcher = pattern.get().matcher(message);
while (matcher.find()) {
int group = 1;
while (group <= matcher.groupCount()) {
if (matcher.group(group) != null) {
for (int i = matcher.start(group); i < matcher.end(group); i++) {
message.setCharAt(i, '*');
}
}
group++;
}
}
}
return message.toString();
}
}
And the updated Configuration file.
<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
<file>c:/logs/kp-ws.log</file>
<append>true</append>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.kp.MaskingPatternLayout">
<patternsProperty>(password)|(karthik)</patternsProperty>
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="fileAppender1" />
</root>
Output
My username=test and password=*******
From the documentation:
replace(p){r, t}
The pattern p can be arbitrarily complex and in particular can contain multiple conversion keywords.
Facing same problem having to replace 2 patterns in a message, I just tried to chain so p is just an invocation of replace, in my case:
%replace( %replace(%msg){'regex1', 'replacement1'} ){'regex2', 'replacement2'}
Worked great, though I wonder if I'm pushing it a bit and p can be indeed that arbitrarily complex.
A very similar but slightly different approach evolves around customizing CompositeConverter and defining a <conversionRule ...> within the logback that references the custom converter.
In one of my tech-demo projects I defined a MaskingConverter class that defines a series of patterns the logging event is analyzed with and on a match updated which is used inside my logback configuration.
As link-only answers are not that beloved here at SO I'll post the important parts of the code here and explain what it does and why it is set up like that. Starting with the Java-based custom converter class:
public class MaskingConverter<E extends ILoggingEvent> extends CompositeConverter<E> {
public static final String CONFIDENTIAL = "CONFIDENTIAL";
public static final Marker CONFIDENTIAL_MARKER = MarkerFactory.getMarker(CONFIDENTIAL);
private Pattern keyValPattern;
private Pattern basicAuthPattern;
private Pattern urlAuthorizationPattern;
#Override
public void start() {
keyValPattern = Pattern.compile("(pw|pwd|password)=.*?(&|$)");
basicAuthPattern = Pattern.compile("(B|b)asic ([a-zA-Z0-9+/=]{3})[a-zA-Z0-9+/=]*([a-zA-Z0-9+/=]{3})");
urlAuthorizationPattern = Pattern.compile("//(.*?):.*?#");
super.start();
}
#Override
protected String transform(E event, String in) {
if (!started) {
return in;
}
Marker marker = event.getMarker();
if (null != marker && CONFIDENTIAL.equals(marker.getName())) {
// key=value[&...] matching
Matcher keyValMatcher = keyValPattern.matcher(in);
// Authorization: Basic dXNlcjpwYXNzd29yZA==
Matcher basicAuthMatcher = basicAuthPattern.matcher(in);
// sftp://user:password#host:port/path/to/resource
Matcher urlAuthMatcher = urlAuthorizationPattern.matcher(in);
if (keyValMatcher.find()) {
String replacement = "$1=XXX$2";
return keyValMatcher.replaceAll(replacement);
} else if (basicAuthMatcher.find()) {
return basicAuthMatcher.replaceAll("$1asic $2XXX$3");
} else if (urlAuthMatcher.find()) {
return urlAuthMatcher.replaceAll("//$1:XXX#");
}
}
return in;
}
}
This class defines a number of RegEx patterns the respective log-line should be compared against and on a match lead to an update of the event by masking the passwords.
Note that this code sample assumes that a log line only contains one kind of password. You are of course free to adapt the bahvior to your needs in case you want to probe each line for multiple pattern matches.
To apply this converter one simply has to add the following line to the logback configuration:
<conversionRule conversionWord="mask" converterClass="at.rovo.awsxray.utils.MaskingConverter"/>
which defines a new function mask which can be used in a pattern in order to mask any log events matching any of the patterns defined in the custom converter. This function can now be used inside a pattern to tell Logback to perform the logic on each log event. The respective pattern might be something along the lines below:
<property name="patternValue"
value="%date{yyyy-MM-dd HH:mm:ss} [%-5level] - %X{FILE_ID} - %mask(%msg) [%thread] [%logger{5}] %n"/>
<!-- Appender definitions-->
<appender class="ch.qos.logback.core.ConsoleAppender" name="console">
<encoder>
<pattern>${patternValue}</pattern>
</encoder>
</appender>
where %mask(%msg) will take the original log-line as input and perform the password masking on each of the lines passed to that function.
As probing each line for one or multiple pattern matches might be costly, the Java code above includes Markers that can be used in log statements to send certain meta information on the log statement itself to Logback/SLF4J. Based on such markers different behaviors might be achievable. In the scenario presented a marker interface can be used to tell Logback that the respective log line contains confidential information and thus requires masking if it matches. Any log line that isn't marked as confidential will be ignored by this converter which helps in pumping out the lines faster as no pattern matching needs to be performed on those lines.
In Java such a marker can be added to a log statement like this:
LOG.debug(MaskingConverter.CONFIDENTIAL_MARKER, "Received basic auth header: {}",
connection.getBasicAuthentication());
which might produce a log line similar to Received basic auth header: Basic QlRXXXlQ= for the above mentioned custom converter, which leaves the first and last couple of characters in tact but obfuscates the middle bits with XXX.
Here is my approach, maybe it can help somebody
Try this one.
1. First of all, we should create a class for handling our logs (each row)
public class PatternMaskingLayout extends PatternLayout {
private Pattern multilinePattern;
private List<String> maskPatterns = new ArrayList<>();
public void addMaskPattern(String maskPattern) { // invoked for every single entry in the xml
maskPatterns.add(maskPattern);
multilinePattern = Pattern.compile(
String.join("|", maskPatterns), // build pattern using logical OR
Pattern.MULTILINE
);
}
#Override
public String doLayout(ILoggingEvent event) {
return maskMessage(super.doLayout(event)); // calling superclass method is required
}
private String maskMessage(String message) {
if (multilinePattern == null) {
return message;
}
StringBuilder sb = new StringBuilder(message);
Matcher matcher = multilinePattern.matcher(sb);
while (matcher.find()) {
if (matcher.group().contains("creditCard")) {
maskCreditCard(sb, matcher);
} else if (matcher.group().contains("email")) {
// your logic for this case
}
}
return sb.toString();
}
private void maskCreditCard(StringBuilder sb, Matcher matcher) {
//here is our main logic for masking sensitive data
String targetExpression = matcher.group();
String[] split = targetExpression.split("=");
String pan = split[1];
String maskedPan = Utils.getMaskedPan(pan);
int start = matcher.start() + split[0].length() + 1;
int end = matcher.end();
sb.replace(start, end, maskedPan);
}
}
The second step is we should create appender for logback into logback.xml
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.bpcbt.micro.utils.PatternMaskingLayout">
<maskPattern>creditCard=\d+</maskPattern> <!-- SourcePan pattern -->
<pattern>%d{dd/MM/yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n%ex</pattern>-->
</layout>
</encoder>
Now we can use logger into our code
log.info("card context set for creditCard={}", creditCard);
As a result, we will see
one row from logs
card context set for creditCard=11111******111
without these options, our logs would be like this row
card context set for creditCard=1111111111111
I've used censor based on RegexCensor from library https://github.com/tersesystems/terse-logback.
In logback.xml
<!--censoring information-->
<newRule pattern="*/censor" actionClass="com.tersesystems.logback.censor.CensorAction"/>
<conversionRule conversionWord="censor" converterClass="com.tersesystems.logback.censor.CensorConverter" />
<!--impl inspired by com.tersesystems.logback.censor.RegexCensor -->
<censor name="censor-sensitive" class="com.mycompaqny.config.logging.SensitiveDataCensor"></censor>
where i put list regex replacements.
#Getter#Setter
public class SensitiveDataCensor extends ContextAwareBase implements Censor, LifeCycle {
protected volatile boolean started = false;
protected String name;
private List<Pair<Pattern, String>> replacementPhrases = new ArrayList<>();
public void start() {
String ssnJsonPattern = "\"(ssn|socialSecurityNumber)(\"\\W*:\\W*\".*?)-(.*?)\"";
replacementPhrases.add(Pair.of(Pattern.compile(ssnJsonPattern), "\"$1$2-****\""));
String ssnXmlPattern = "<(ssn|socialSecurityNumber)>(\\W*.*?)-(.*?)</";
replacementPhrases.add(Pair.of(Pattern.compile(ssnXmlPattern), "<$1>$2-****</"));
started = true;
}
public void stop() {
replacementPhrases.clear();
started = false;
}
public CharSequence censorText(CharSequence original) {
CharSequence outcome = original;
for (Pair<Pattern, String> replacementPhrase : replacementPhrases) {
outcome = replacementPhrase.getLeft().matcher(outcome).replaceAll(replacementPhrase.getRight());
}
return outcome;
}
}
and used it in logback.xml like this
<message>[ignore]</message> <---- IMPORTANT to disable original message field so you get only censored message
...
<pattern>
{"message": "%censor(%msg){censor-sensitive}"}
</pattern>
I was trying to mask some sensitive data in my demo project logs. I tried with but it didn't worked for me because of Java Reflections as I took variable name as pattern. I am adding the solution which worked for me incase if it helps anyone else also.
I added below code in logback.xml(inside encoder tag) file for masking field1 and field2 information in the logs.
<encoder class="com.demo.config.CustomJsonMaskLogEncoder">
<patterns>
<pattern>\"field1\"\s*:\s*\"(.*?)\"</pattern>
<pattern>\"field2\"\s*:\s*\"(.*?)\"</pattern>
<pattern>%-5p [%d{ISO8601,UTC}] [%thread] %c: %m%n%rootException</pattern>
</patterns>
</encoder>
I have written a CustomJsonMaskLogEncoder which does the job of masking the field data as per regex.
package com.demo.config;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.pattern.ExtendedThrowableProxyConverter;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.classic.spi.LoggingEvent;
import java.util.ArrayList;
import net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder;
import org.slf4j.LoggerFactory;
public class CustomJsonMaskLogEncoder extends LoggingEventCompositeJsonEncoder {
private final CustomPatternMaskingLayout customPatternMaskingLayout;
private boolean maskEnabled;
public JsonMaskLogEncoder() {
super();
customPatternMaskingLayout = new CustomPatternMaskingLayout();
maskEnabled = true;
}
#Override
public byte[] encode(ILoggingEvent event) {
return maskEnabled ? getMaskedJson(event) : super.encode(event);
}
private byte[] getMaskedJson(ILoggingEvent event) {
final Logger logger =
(ch.qos.logback.classic.Logger) LoggerFactory.getLogger(event.getLoggerName());
final String message = customPatternMaskingLayout.maskMessage(event.getFormattedMessage());
final LoggingEvent loggingEvent =
new LoggingEvent(
"", logger, event.getLevel(), message, getThrowable(event), event.getArgumentArray());
return super.encode(loggingEvent);
}
private Throwable getThrowable(ILoggingEvent event) {
return event.getThrowableProxy() == null ? null : new Throwable(getStackTrace(event));
}
private String getStackTrace(ILoggingEvent event) {
final ExtendedThrowableProxyConverter throwableConverter =
new ExtendedThrowableProxyConverter();
throwableConverter.start();
final String errorMessageWithStackTrace = throwableConverter.convert(event);
throwableConverter.stop();
return errorMessageWithStackTrace;
}
#SuppressWarnings("unused")
public void setEnableMasking(boolean enabled) {
this.maskEnabled = enabled;
}
#SuppressWarnings("unused")
public void setPatterns(Patterns patterns) {
customPatternMaskingLayout.addMaskPatterns(patterns);
}
public static class Patterns extends ArrayList<String> {
#SuppressWarnings("unused")
public void addPattern(String pattern) {
add(pattern);
}
}
}
And below is the code for actual CustomPatternMaskingLayout:
package com.demo.config;
import static java.lang.String.format;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import java.util.stream.Stream;
public class CustomPatternMaskingLayout {
private Pattern multilinePattern;
private final List<String> maskPatterns = new ArrayList<>();
public CustomPatternMaskingLayout() {
compilePattern();
}
void addMaskPatterns(CustomJsonMaskLogEncoder.Patterns patterns) {
maskPatterns.addAll(patterns);
compilePattern();
}
private void compilePattern() {
multilinePattern = Pattern.compile(String.join("|", maskPatterns),Pattern.MULTILINE);
}
String maskMessage(String message) {
if (multilinePattern == null) {
return message;
}
StringBuilder sb = new StringBuilder(message);
Matcher matcher = multilinePattern.matcher(sb);
while (matcher.find()) {
IntStream.rangeClosed(1, matcher.groupCount()).forEach(group -> {
if (matcher.group(group) != null) {
IntStream.range(matcher.start(group), matcher.end(group)).forEach(i -> sb.setCharAt(i, '*'));
}
});
}
return sb.toString();
}
}
Hope this helps!!!