Additional Stacktrace info when serializing log4net properties to Json - json

I'm attempting to serialize my logs to Json. I'm using log4net with a custom layout, but when an exception is logged, I get the following malformed JSON (Note the additional stacktrace info at the end)
Am I missing a setting for log4net, or is this a serialization issue?
UPDATE: This has to be something with log4net, because json.net serializes an Exception perfectly.
***UPDATE (FIXED): Updated code below.
{
"UserSessionId":"4b146c92-fe99-4f78-bbef-720df2cf7473",
"ProcessSessionId":1,
...
"Logger":"testharness.Program","ThreadName":"1",
"ExceptionObject":{
"ClassName":"System.ApplicationException",
"Message":"Test Exception Logging",
"Data":null,
"InnerException":null,
"HelpURL":null,
"StackTraceString":" at testharness.Program.Main(String[] args) in C:\\temp\\testharness\\Program.cs:line 18",
"RemoteStackTraceString":null,
...
"WatsonBuckets":null
},
...
"log4net:HostName":"ol-4RBNMH2"
}}
System.ApplicationException: Test Exception Logging
at testharness.Program.Main(String[] args) in C:\temp\testharness\Program.cs:line 18
log4net configuratiton
<log4net>
<appender name="RollingFileCompositeAppender" type="log4net.Appender.RollingFileAppender">
<file value="c:\\logs\\testharness.txt"/>
<appendToFile value="true"/>
<rollingStyle value="Composite"/>
<datePattern value="yyyy-MM-dd"/>
<maxSizeRollBackups value="-1"/>
<maximumFileSize value="1MB"/>
<countDirection value="1"/>
<preserveLogFileNameExtension value="false"/>
<staticLogFileName value="false"/>
<layout type="Company.log4net.JsonLayout"></layout>
</appender>
<root>
<level value="ALL"/>
<appender-ref ref="RollingFileCompositeAppender"/>
</root>
</log4net>
The Custom Layout class
public class JsonLayout : LayoutSkeleton
{
public JsonLayout() {
IgnoresException = false;
}
...
/// <inheritdoc />
public override void ActivateOptions()
{
}
/// <inheritdoc />
public override void Format(TextWriter writer, LoggingEvent loggingEvent)
{
_customProperties.PhysicalMemory = Process.GetCurrentProcess().WorkingSet64;
var evt = new CustomLoggingEvent(loggingEvent, _customProperties);
writer.Write(JsonConvert.SerializeObject(evt));
}
}
[JsonObject(MemberSerialization.OptIn)]
public class CustomLoggingEvent
{
...
[JsonProperty]
public Exception ExceptionObject { get; set; }
[JsonProperty]
public long PhysicalMemory { get; set; }
[JsonProperty]
public PropertiesDictionary Properties { get; set; }
}
The Test Harness:
internal class Program
{
private static void Main()
{
Console.WriteLine($"{typeof(JsonLayout)}");
var log = log4net.LogManager.GetLogger(typeof(Program));
try
{
log.Debug("hello again world");
throw new ApplicationException("Test Exception Logging");
}
catch (Exception e)
{
log.Error("Exception Thrown", e);
}
Console.ReadLine();
}
}

I downloaded the log4net source and found the issue. When creating a custom layout implementing LayoutSkeleton, if your layout handles the LoggingEvent.ExceptionObject, then the IgnoresException property should be set to false; the default value is true. I updated the code in the question with a constructor wherein the IgnoresException property is set to true. I probably should have waited a day while I researched, but maybe this will help someone else.

Related

Mask sensitive data in logs with logback

I need to be able to search an event for any one of a number of patterns and replace the text in the pattern with a masked value. This is a feature in our application intended to prevent sensitive information falling into the logs. As the information can be from a large variety of sources, it is not practical to apply filters on all the inputs. Besides there are uses for toString() beyond logging and I don't want toString() to uniformly mask for all calls (only logging).
I have tried using the %replace method in logback.xml:
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %replace(%msg){'f k\="pin">(.*?)</f','f k\="pin">**********</f'}%n</pattern>
This was successful (after replacing the angle brackets with character entities), but it can only replace a single pattern. I would also like to perform the equivalent of
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %replace(%msg){'pin=(.*?),','pin=**********,'}%n</pattern>
at the same time, but cannot. There is no way to mask two patterns in the one %replace.
The other way that has been loosely discussed on the interblags is extending something on the appender/encoder/layout hierarchy, but every attempt to intercept the ILoggingEvent has resulted in a collapse of the whole system, usually through instantiation errors or UnsupportedOperationException.
For example, I tried extending PatternLayout:
#Component("maskingPatternLayout")
public class MaskingPatternLayout extends PatternLayout {
#Autowired
private Environment env;
#Override
public String doLayout(ILoggingEvent event) {
String message=super.doLayout(event);
String patternsProperty = env.getProperty("bowdleriser.patterns");
if( patternsProperty != null ) {
String[] patterns = patternsProperty.split("|");
for (int i = 0; i < patterns.length; i++ ) {
Pattern pattern = Pattern.compile(patterns[i]);
Matcher matcher = pattern.matcher(event.getMessage());
matcher.replaceAll("*");
}
} else {
System.out.println("Bowdleriser not cleaning! Naughty strings are getting through!");
}
return message;
}
}
and then adjusting the logback.xml
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<layout class="com.touchcorp.touchpoint.utils.MaskingPatternLayout">
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>logs/touchpoint.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>logs/touchpoint.%i.log.zip</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>3</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10MB</maxFileSize>
</triggeringPolicy>
<encoder>
<layout class="com.touchcorp.touchpoint.utils.MaskingPatternLayout">
<pattern>%date{YYYY-MM-dd HH:mm:ss} %level [%thread] %logger{10} [%file:%line] %msg%n</pattern>
</layout>
</encoder>
</appender>
<logger name="com.touchcorp.touchpoint" level="DEBUG" />
<logger name="org.springframework.web.servlet.mvc" level="TRACE" />
<root level="INFO">
<appender-ref ref="FILE" />
<appender-ref ref="STDOUT" />
</root>
</configuration>
I have tried many other insertions, so I was wondering if anyone has actually achieved what I am attempting and if they could provide any clues or a solution.
You need to wrap layout using LayoutWrappingEncoder. And also I believe you cannot use spring here as logback is not managed by spring.
Here is the updated class.
public class MaskingPatternLayout extends PatternLayout {
private String patternsProperty;
public String getPatternsProperty() {
return patternsProperty;
}
public void setPatternsProperty(String patternsProperty) {
this.patternsProperty = patternsProperty;
}
#Override
public String doLayout(ILoggingEvent event) {
String message = super.doLayout(event);
if (patternsProperty != null) {
String[] patterns = patternsProperty.split("\\|");
for (int i = 0; i < patterns.length; i++) {
Pattern pattern = Pattern.compile(patterns[i]);
Matcher matcher = pattern.matcher(event.getMessage());
if (matcher.find()) {
message = matcher.replaceAll("*");
}
}
} else {
}
return message;
}
}
And sample logback.xml
<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
<file>c:/logs/kp-ws.log</file>
<append>true</append>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.kp.MaskingPatternLayout">
<patternsProperty>.*password.*|.*karthik.*</patternsProperty>
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="fileAppender1" />
</root>
UPDATE
Here its better approach, set Pattern during init itself. such that we can avoid recreating Pattern again and again and this implementation is close to realistic usecase.
public class MaskingPatternLayout extends PatternLayout {
private String patternsProperty;
private Optional<Pattern> pattern;
public String getPatternsProperty() {
return patternsProperty;
}
public void setPatternsProperty(String patternsProperty) {
this.patternsProperty = patternsProperty;
if (this.patternsProperty != null) {
this.pattern = Optional.of(Pattern.compile(patternsProperty, Pattern.MULTILINE));
} else {
this.pattern = Optional.empty();
}
}
#Override
public String doLayout(ILoggingEvent event) {
final StringBuilder message = new StringBuilder(super.doLayout(event));
if (pattern.isPresent()) {
Matcher matcher = pattern.get().matcher(message);
while (matcher.find()) {
int group = 1;
while (group <= matcher.groupCount()) {
if (matcher.group(group) != null) {
for (int i = matcher.start(group); i < matcher.end(group); i++) {
message.setCharAt(i, '*');
}
}
group++;
}
}
}
return message.toString();
}
}
And the updated Configuration file.
<appender name="fileAppender1" class="ch.qos.logback.core.FileAppender">
<file>c:/logs/kp-ws.log</file>
<append>true</append>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.kp.MaskingPatternLayout">
<patternsProperty>(password)|(karthik)</patternsProperty>
<pattern>%d [%thread] %-5level %logger{35} - %msg%n</pattern>
</layout>
</encoder>
</appender>
<root level="DEBUG">
<appender-ref ref="fileAppender1" />
</root>
Output
My username=test and password=*******
From the documentation:
replace(p){r, t}
The pattern p can be arbitrarily complex and in particular can contain multiple conversion keywords.
Facing same problem having to replace 2 patterns in a message, I just tried to chain so p is just an invocation of replace, in my case:
%replace( %replace(%msg){'regex1', 'replacement1'} ){'regex2', 'replacement2'}
Worked great, though I wonder if I'm pushing it a bit and p can be indeed that arbitrarily complex.
A very similar but slightly different approach evolves around customizing CompositeConverter and defining a <conversionRule ...> within the logback that references the custom converter.
In one of my tech-demo projects I defined a MaskingConverter class that defines a series of patterns the logging event is analyzed with and on a match updated which is used inside my logback configuration.
As link-only answers are not that beloved here at SO I'll post the important parts of the code here and explain what it does and why it is set up like that. Starting with the Java-based custom converter class:
public class MaskingConverter<E extends ILoggingEvent> extends CompositeConverter<E> {
public static final String CONFIDENTIAL = "CONFIDENTIAL";
public static final Marker CONFIDENTIAL_MARKER = MarkerFactory.getMarker(CONFIDENTIAL);
private Pattern keyValPattern;
private Pattern basicAuthPattern;
private Pattern urlAuthorizationPattern;
#Override
public void start() {
keyValPattern = Pattern.compile("(pw|pwd|password)=.*?(&|$)");
basicAuthPattern = Pattern.compile("(B|b)asic ([a-zA-Z0-9+/=]{3})[a-zA-Z0-9+/=]*([a-zA-Z0-9+/=]{3})");
urlAuthorizationPattern = Pattern.compile("//(.*?):.*?#");
super.start();
}
#Override
protected String transform(E event, String in) {
if (!started) {
return in;
}
Marker marker = event.getMarker();
if (null != marker && CONFIDENTIAL.equals(marker.getName())) {
// key=value[&...] matching
Matcher keyValMatcher = keyValPattern.matcher(in);
// Authorization: Basic dXNlcjpwYXNzd29yZA==
Matcher basicAuthMatcher = basicAuthPattern.matcher(in);
// sftp://user:password#host:port/path/to/resource
Matcher urlAuthMatcher = urlAuthorizationPattern.matcher(in);
if (keyValMatcher.find()) {
String replacement = "$1=XXX$2";
return keyValMatcher.replaceAll(replacement);
} else if (basicAuthMatcher.find()) {
return basicAuthMatcher.replaceAll("$1asic $2XXX$3");
} else if (urlAuthMatcher.find()) {
return urlAuthMatcher.replaceAll("//$1:XXX#");
}
}
return in;
}
}
This class defines a number of RegEx patterns the respective log-line should be compared against and on a match lead to an update of the event by masking the passwords.
Note that this code sample assumes that a log line only contains one kind of password. You are of course free to adapt the bahvior to your needs in case you want to probe each line for multiple pattern matches.
To apply this converter one simply has to add the following line to the logback configuration:
<conversionRule conversionWord="mask" converterClass="at.rovo.awsxray.utils.MaskingConverter"/>
which defines a new function mask which can be used in a pattern in order to mask any log events matching any of the patterns defined in the custom converter. This function can now be used inside a pattern to tell Logback to perform the logic on each log event. The respective pattern might be something along the lines below:
<property name="patternValue"
value="%date{yyyy-MM-dd HH:mm:ss} [%-5level] - %X{FILE_ID} - %mask(%msg) [%thread] [%logger{5}] %n"/>
<!-- Appender definitions-->
<appender class="ch.qos.logback.core.ConsoleAppender" name="console">
<encoder>
<pattern>${patternValue}</pattern>
</encoder>
</appender>
where %mask(%msg) will take the original log-line as input and perform the password masking on each of the lines passed to that function.
As probing each line for one or multiple pattern matches might be costly, the Java code above includes Markers that can be used in log statements to send certain meta information on the log statement itself to Logback/SLF4J. Based on such markers different behaviors might be achievable. In the scenario presented a marker interface can be used to tell Logback that the respective log line contains confidential information and thus requires masking if it matches. Any log line that isn't marked as confidential will be ignored by this converter which helps in pumping out the lines faster as no pattern matching needs to be performed on those lines.
In Java such a marker can be added to a log statement like this:
LOG.debug(MaskingConverter.CONFIDENTIAL_MARKER, "Received basic auth header: {}",
connection.getBasicAuthentication());
which might produce a log line similar to Received basic auth header: Basic QlRXXXlQ= for the above mentioned custom converter, which leaves the first and last couple of characters in tact but obfuscates the middle bits with XXX.
Here is my approach, maybe it can help somebody
Try this one.
1. First of all, we should create a class for handling our logs (each row)
public class PatternMaskingLayout extends PatternLayout {
private Pattern multilinePattern;
private List<String> maskPatterns = new ArrayList<>();
public void addMaskPattern(String maskPattern) { // invoked for every single entry in the xml
maskPatterns.add(maskPattern);
multilinePattern = Pattern.compile(
String.join("|", maskPatterns), // build pattern using logical OR
Pattern.MULTILINE
);
}
#Override
public String doLayout(ILoggingEvent event) {
return maskMessage(super.doLayout(event)); // calling superclass method is required
}
private String maskMessage(String message) {
if (multilinePattern == null) {
return message;
}
StringBuilder sb = new StringBuilder(message);
Matcher matcher = multilinePattern.matcher(sb);
while (matcher.find()) {
if (matcher.group().contains("creditCard")) {
maskCreditCard(sb, matcher);
} else if (matcher.group().contains("email")) {
// your logic for this case
}
}
return sb.toString();
}
private void maskCreditCard(StringBuilder sb, Matcher matcher) {
//here is our main logic for masking sensitive data
String targetExpression = matcher.group();
String[] split = targetExpression.split("=");
String pan = split[1];
String maskedPan = Utils.getMaskedPan(pan);
int start = matcher.start() + split[0].length() + 1;
int end = matcher.end();
sb.replace(start, end, maskedPan);
}
}
The second step is we should create appender for logback into logback.xml
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="com.bpcbt.micro.utils.PatternMaskingLayout">
<maskPattern>creditCard=\d+</maskPattern> <!-- SourcePan pattern -->
<pattern>%d{dd/MM/yyyy HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n%ex</pattern>-->
</layout>
</encoder>
Now we can use logger into our code
log.info("card context set for creditCard={}", creditCard);
As a result, we will see
one row from logs
card context set for creditCard=11111******111
without these options, our logs would be like this row
card context set for creditCard=1111111111111
I've used censor based on RegexCensor from library https://github.com/tersesystems/terse-logback.
In logback.xml
<!--censoring information-->
<newRule pattern="*/censor" actionClass="com.tersesystems.logback.censor.CensorAction"/>
<conversionRule conversionWord="censor" converterClass="com.tersesystems.logback.censor.CensorConverter" />
<!--impl inspired by com.tersesystems.logback.censor.RegexCensor -->
<censor name="censor-sensitive" class="com.mycompaqny.config.logging.SensitiveDataCensor"></censor>
where i put list regex replacements.
#Getter#Setter
public class SensitiveDataCensor extends ContextAwareBase implements Censor, LifeCycle {
protected volatile boolean started = false;
protected String name;
private List<Pair<Pattern, String>> replacementPhrases = new ArrayList<>();
public void start() {
String ssnJsonPattern = "\"(ssn|socialSecurityNumber)(\"\\W*:\\W*\".*?)-(.*?)\"";
replacementPhrases.add(Pair.of(Pattern.compile(ssnJsonPattern), "\"$1$2-****\""));
String ssnXmlPattern = "<(ssn|socialSecurityNumber)>(\\W*.*?)-(.*?)</";
replacementPhrases.add(Pair.of(Pattern.compile(ssnXmlPattern), "<$1>$2-****</"));
started = true;
}
public void stop() {
replacementPhrases.clear();
started = false;
}
public CharSequence censorText(CharSequence original) {
CharSequence outcome = original;
for (Pair<Pattern, String> replacementPhrase : replacementPhrases) {
outcome = replacementPhrase.getLeft().matcher(outcome).replaceAll(replacementPhrase.getRight());
}
return outcome;
}
}
and used it in logback.xml like this
<message>[ignore]</message> <---- IMPORTANT to disable original message field so you get only censored message
...
<pattern>
{"message": "%censor(%msg){censor-sensitive}"}
</pattern>
I was trying to mask some sensitive data in my demo project logs. I tried with but it didn't worked for me because of Java Reflections as I took variable name as pattern. I am adding the solution which worked for me incase if it helps anyone else also.
I added below code in logback.xml(inside encoder tag) file for masking field1 and field2 information in the logs.
<encoder class="com.demo.config.CustomJsonMaskLogEncoder">
<patterns>
<pattern>\"field1\"\s*:\s*\"(.*?)\"</pattern>
<pattern>\"field2\"\s*:\s*\"(.*?)\"</pattern>
<pattern>%-5p [%d{ISO8601,UTC}] [%thread] %c: %m%n%rootException</pattern>
</patterns>
</encoder>
I have written a CustomJsonMaskLogEncoder which does the job of masking the field data as per regex.
package com.demo.config;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.pattern.ExtendedThrowableProxyConverter;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.classic.spi.LoggingEvent;
import java.util.ArrayList;
import net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder;
import org.slf4j.LoggerFactory;
public class CustomJsonMaskLogEncoder extends LoggingEventCompositeJsonEncoder {
private final CustomPatternMaskingLayout customPatternMaskingLayout;
private boolean maskEnabled;
public JsonMaskLogEncoder() {
super();
customPatternMaskingLayout = new CustomPatternMaskingLayout();
maskEnabled = true;
}
#Override
public byte[] encode(ILoggingEvent event) {
return maskEnabled ? getMaskedJson(event) : super.encode(event);
}
private byte[] getMaskedJson(ILoggingEvent event) {
final Logger logger =
(ch.qos.logback.classic.Logger) LoggerFactory.getLogger(event.getLoggerName());
final String message = customPatternMaskingLayout.maskMessage(event.getFormattedMessage());
final LoggingEvent loggingEvent =
new LoggingEvent(
"", logger, event.getLevel(), message, getThrowable(event), event.getArgumentArray());
return super.encode(loggingEvent);
}
private Throwable getThrowable(ILoggingEvent event) {
return event.getThrowableProxy() == null ? null : new Throwable(getStackTrace(event));
}
private String getStackTrace(ILoggingEvent event) {
final ExtendedThrowableProxyConverter throwableConverter =
new ExtendedThrowableProxyConverter();
throwableConverter.start();
final String errorMessageWithStackTrace = throwableConverter.convert(event);
throwableConverter.stop();
return errorMessageWithStackTrace;
}
#SuppressWarnings("unused")
public void setEnableMasking(boolean enabled) {
this.maskEnabled = enabled;
}
#SuppressWarnings("unused")
public void setPatterns(Patterns patterns) {
customPatternMaskingLayout.addMaskPatterns(patterns);
}
public static class Patterns extends ArrayList<String> {
#SuppressWarnings("unused")
public void addPattern(String pattern) {
add(pattern);
}
}
}
And below is the code for actual CustomPatternMaskingLayout:
package com.demo.config;
import static java.lang.String.format;
import java.util.ArrayList;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import java.util.stream.Stream;
public class CustomPatternMaskingLayout {
private Pattern multilinePattern;
private final List<String> maskPatterns = new ArrayList<>();
public CustomPatternMaskingLayout() {
compilePattern();
}
void addMaskPatterns(CustomJsonMaskLogEncoder.Patterns patterns) {
maskPatterns.addAll(patterns);
compilePattern();
}
private void compilePattern() {
multilinePattern = Pattern.compile(String.join("|", maskPatterns),Pattern.MULTILINE);
}
String maskMessage(String message) {
if (multilinePattern == null) {
return message;
}
StringBuilder sb = new StringBuilder(message);
Matcher matcher = multilinePattern.matcher(sb);
while (matcher.find()) {
IntStream.rangeClosed(1, matcher.groupCount()).forEach(group -> {
if (matcher.group(group) != null) {
IntStream.range(matcher.start(group), matcher.end(group)).forEach(i -> sb.setCharAt(i, '*'));
}
});
}
return sb.toString();
}
}
Hope this helps!!!

Logging Exception.Data using Log4Net

We're just getting started with Log4Net (and wishing we'd done it earlier). Whilst we can see inner exceptions, etc. the one thing that seems to be missing from the output when logging an exception is any key/value information held inside the "Exception.Data". Is there anyway we can do this "out of the box"? If not, as we really are only just starting out where should be looking to find a way to implement this functionality?
As an example please see the very basic pseudo code below. We don't want to pollute the exception message with context information just what the problem was (We'd probably have lost more information in the data which would help in investigating the actual problem). But right now all we see in our logs is the type of exception, the message, any stack trace - but no exception "data". This means in our logs we lose the customer id, etc. How can we easily get this information into our logs (without having to code it by hand in each exception catch).
try
{
var ex = new ApplicationException("Unable to update customer");
ex.Data.Add("id", customer.Id);
throw ex;
}
catch(ApplicationException ex)
{
logger.Error("An error occurred whilst doing something", ex);
throw;
}
Following Stefan's lead:
namespace YourNamespace {
public sealed class ExceptionDataPatternConverter : PatternLayoutConverter {
protected override void Convert(TextWriter writer, LoggingEvent loggingEvent) {
var data = loggingEvent.ExceptionObject.Data;
if (data != null) {
foreach(var key in data.Keys) {
writer.Write("Data[{0}]={1}" + Environment.NewLine, key, data[key]);
}
}
}
}
}
And in your configuration add %ex_data and the converter:
<appender ...>
...
<layout type="log4net.Layout.PatternLayout,log4net">
<conversionPattern value="%date %d{HH:mm:ss.fff} [%t] %-5p %c %l - %m%n %ex_data"/>
<converter>
<name value="ex_data" />
<type value="YourNamespace.ExceptionDataPatternConverter" />
</converter>
</layout>
If you have multiple appenders defined you can use a custom renderer rather than defining the converter for every layout.
web/app.config
<log4net>
...
<renderer renderingClass="YourNamespace.ExceptionObjectLogger, YourAssembly" renderedClass="System.Exception" />
...
</log4net>
ExceptionObjectLogger
public class ExceptionObjectLogger : IObjectRenderer
{
public void RenderObject(RendererMap rendererMap, object obj, TextWriter writer)
{
var ex = obj as Exception;
if (ex == null)
{
// Shouldn't happen if only configured for the System.Exception type.
rendererMap.DefaultRenderer.RenderObject(rendererMap, obj, writer);
}
else
{
rendererMap.DefaultRenderer.RenderObject(rendererMap, obj, writer);
const int MAX_DEPTH = 10;
int currentDepth = 0;
while (ex != null && currentDepth <= MAX_DEPTH)
{
this.RenderExceptionData(rendererMap, ex, writer, currentDepth);
ex = ex.InnerException;
currentDepth++;
}
}
}
private void RenderExceptionData(RendererMap rendererMap, Exception ex, TextWriter writer, int depthLevel)
{
var dataCount = ex.Data.Count;
if (dataCount == 0)
{
return;
}
writer.WriteLine();
writer.WriteLine($"Exception data on level {depthLevel} ({dataCount} items):");
var currentElement = 0;
foreach (DictionaryEntry entry in ex.Data)
{
currentElement++;
writer.Write("[");
ExceptionObjectLogger.RenderValue(rendererMap, writer, entry.Key);
writer.Write("]: ");
ExceptionObjectLogger.RenderValue(rendererMap, writer, entry.Value);
if (currentElement < dataCount)
{
writer.WriteLine();
}
}
}
private static void RenderValue(RendererMap rendererMap, TextWriter writer, object value)
{
if (value is string)
{
writer.Write(value);
}
else
{
IObjectRenderer keyRenderer = rendererMap.Get(value.GetType());
keyRenderer.RenderObject(rendererMap, value, writer);
}
}
}
I think a more log4net way of approaching this problem would be to write a PatternLayoutConverter. An example can be found here.
In the convert method you can access your data like this (and write it the way you like):
override protected void Convert(TextWriter writer, LoggingEvent loggingEvent)
{
var data = loggingEvent.ExceptionObject.Data;
}
I think Massimiliano has the right idea but I would modify his solution slightly.
If you plan on sticking all of of your additional data in the dictionary Data within an exception I would change his extension method to the following:
public static class ExLog4Net
{
public static void Error(this ILog log, Exception ex)
{
StringBuilder formattedError = new StringBuilder();
formattedError.AppendFormat("Exception: {0}\r\n", ex.ToString());
foreach (DictionaryEntry de in ex.Data)
formattedError.AppendFormat("{0}: {1}\r\n", de.Key, de.Value);
log.Error(formattedError.ToString());
}
}
You would then stick this method extension in a library you would use in all of your applications. If you don't have one you would have to add this to every project.
you could create an Extension method for your logger to log the customer Id : you should not add important information to the exception
You can abstract the concept of "Additional Information to log" and create an interface with a method that return the additional information you want to log
public interface IDataLogger
{
string GetAdditionalInfo();
}
public class UserDataLogger: IDataLogger
{
public string GetAdditionalInfo()
{
return "UserName";
}
}
public class MoreDataLogger : IDataLogger
{
public string GetAdditionalInfo()
{
return "something";
}
}
you can create different "data Logger" and maybe combine them together
then you could create an generic extension method that get the type of the logger
public static class ExLog4Net
{
public static void Error<T>(this ILog log, Exception ex) where T:IDataLogger,new()
{
var dataLogger=new T();
log.Error(ex.ToString() + " " + dataLogger.GetAdditionalInfo());
}
}
you will be able to do the below:
try
{
}
catch (Exception ex)
{
logger.Error<UserDataLogger>(ex);
logger.Error<MoreDataLogger>(ex);
throw;
}

How to provide JSON data to Dojo tree using Struts2 framework

I'm currently developing a web application using Struts2 framework. This application requires to dynamically update the objects on the screen based on data received from another application.
At the moment, I would like to implement a dynamic tree view which nodes are updated periodically with data provided by an Action class. I’m trying to do so using the dojo.dijit.tree object from the dojo toolkit. I’m aware that I can do so using the dojo tags which are part of the struts framework, however, it lacks much of the functionality that I need (persistence, open and close branches dynamically, etc) therefore, I have opted for using the dojo toolkit instead.
My problem with the dojo.dijit.tree is that I don’t know how to provide its data using a JSON result type. I have already created a class which returns a JSON result type with the same structure needed by the dojo tree component. I have tested the generation of a dojo tree using a file “test.txt” which was generated by the class and it works as expected. However, I would like to pass the JSON data directly to the dojo.dijit.tree component without saving a file on disk. When I execute the application I get a “save as” window to save the returned JSON result.
This is my struts.xml file:
<struts>
<constant name="struts.devMode" value="true" />
<package name="default" namespace="/" extends="struts-default">
<action name="devResult" class="gui.JsonAction">
<result name="success">/start1.jsp</result>
</action>
</package>
<package name="example" namespace="/" extends="json-default">
<result-types>
<result-type name="json" class="org.apache.struts2.json.JSONResult"></result-type>
</result-types>
<action name="getJSONResult" class="gui.JsonAction">
<result type="json"/>
</action>
</package>
This is the jsp file which displays the tree:
<head>
<title>Testing Tree</title>
<style type="text/css">
#import "js/dojo/dojo/resources/dojo.css";
#import "js/dojo/dijit/themes/nihilo/nihilo.css";
</style>
<script src="http://ajax.googleapis.com/ajax/libs/dojo/1.6/dojo/dojo.xd.js"
djConfig="isDebug: true,parseOnLoad: true">
</script>
<script type="text/javascript">
dojo.require("dojo.data.ItemFileReadStore");
dojo.require("dijit.Tree");
dojo.require("dojo.parser");
</script>
<body class="nihilo">
The Tree:<br><br>
<s:url id="devResult" action="jsonAction.action"></s:url>
<div dojoType="dojo.data.ItemFileReadStore" href="%{devResult}" jsid="popStore" />
<div dojoType="dijit.Tree" store="popStore" labelAttr="sname" label="Tree" />
</body>
This is the Action class which produces the JSON result:
public class JsonAction extends ActionSupport {
private static final long serialVersionUID = 7392602552908646926L;
private String label = "name";
private String identifier = "name";
private List<ChildrenClass> items = new ArrayList<ChildrenClass>();
public JsonAction() {
ChildrenClass item1 = new ChildrenClass("name1", "cat");
ChildrenClass item2 = new ChildrenClass("name2", "cat");
ChildrenClass item3 = new ChildrenClass("name3", "cat");
ChildrenClass item4 = new ChildrenClass("name4", "cat");
items.add(item1);
items.add(item2);
items.add(item3);
items.add(item4);
}
public String execute() {
return SUCCESS;
}
public void setLabel(String label) {
this.label = label;
}
public String getLabel() {
return label;
}
public void setIdentifier(String identifier) {
this.identifier = identifier;
}
public String getIdentifier() {
return identifier;
}
public void setItems(List<ChildrenClass> lists) {
this.items = lists;
}
public List<ChildrenClass> getItems() {
return items;
}
}
This is the ChildrenClass which is used in the class above:
public class ChildrenClass {
private String name;
private String type;
private ChildrenClass[] children;
public ChildrenClass() {
name = "DefaultName";
type = "DefaultType";
}
public ChildrenClass(String aName, String aType) {
name = aName;
type = aType;
}
public void setName(String name) {
this.name = name;
}
public String getName() {
return name;
}
public void setType(String type) {
this.type = type;
}
public String getType() {
return type;
}
public void setChildren(ChildrenClass[] children) {
this.children = children;
}
public ChildrenClass[] getChildren() {
return children;
}
}
I would like to ask to the stackoverflow reader to please indicate me how to do to read the JSON data in the jsp file in order to populate the dojo tree. In addition, I would like to ask how can I refresh the content of it periodically.
PS: If somebody has a better method to implement something similar to this, I would be glad to receive your comments.
Thanks in advance.
I have found out a way to pass data directly from a JSON result to a dojo.dijit.tree component. Setting the "url" parameter to the action name which returns the json result type.
This is my new body of the .jsp file:
Simple Tree:<br><br>
<div dojoType="dojo.data.ItemFileReadStore" url=getJSONResult handleAs="json" jsid="popStore" />
<div dojoType="dijit.Tree" store="popStore" labelAttr="sname" label="PID 512" />

Can't Seem to get my data out of MySQL DB with NHibernate

So I have setup a MySQL database with table with one record. My Solution is made up of three projects (1 domain model library, test library and my Web project). In my MVC project I have implemented NHibernate with all necessary Dll's, and
In Web project root:
nhibernate-configuration.xsd
nhibernate-mapping.xsd
nhibernate.config and
<classname>.hbm.xml file - with the class it is mapping
In my Global.asax.cs file I have my event handlers to bind the current session:
public class MvcApplication : System.Web.HttpApplication
{
public MvcApplication()
{
BeginRequest += (MvcApplication_BeginRequest);
EndRequest += (MvcApplication_EndRequest);
}
void MvcApplication_BeginRequest(object sender, EventArgs e)
{
CurrentSessionContext.Bind(BootStrapper.SessionFactory.OpenSession());
}
void MvcApplication_EndRequest(object sender, EventArgs e)
{
CurrentSessionContext.Unbind(BootStrapper.SessionFactory).Dispose();
}
Then I have my BootStrapper class which returns the current session:
public static readonly ISessionFactory SessionFactory = CreateSessionFactory();
private static ISessionFactory CreateSessionFactory()
{
var cfg = new Configuration().Configure(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "nhibernate.config"));
cfg.SetProperty(NHibernate.Cfg.Environment.ConnectionStringName, System.Environment.MachineName);
return cfg.BuildSessionFactory();
}
public static ISession GetSession()
{
return SessionFactory.GetCurrentSession();
}
My Controller is being handed an object by my Ninject IoC
ProductController.cs
public class ProductsController : Controller
{
private readonly IProductsRepository productsRepository;
public ProductsController(IProductsRepository productsRepository)
{
this.productsRepository = productsRepository;
}
public ViewResult List()
{
return View(productsRepository.Products.ToList());
}
}
NinjectControllerFactory.cs
public class NinjectControllerFactory : DefaultControllerFactory
{
//Supplies Object instances
private IKernel kernel = new StandardKernel(new DaisyblossomsServices());
//MVC calls this to get the controller for each requests
protected override IController GetControllerInstance(System.Web.Routing.RequestContext requestContext, Type controllerType)
{
if (controllerType == null)
return null;
return (Controller)kernel.Get(controllerType);
}
}
Which you will sell calls my services class DaisyblossomsServices:
public class DaisyblossomsServices : NinjectModule
{
public override void Load()
{
Bind<IProductsRepository>().To<ProductsRepository>();
}
}
Where you can see IProductsRepository is bound to my ProductsRepository class:
public class ProductsRepository : IProductsRepository
{
public IQueryable Products
{
get { var session = BootStrapper.GetSession();
return session.CreateCriteria(typeof(Product)).List<Product>().AsQueryable();
}
}
}
And my ProductsController is handed an IProductsRepository object
public interface IProductsRepository
{
IQueryable Products { get; }
}
As additional information My Product.hbm.xml file which maps my Product.cs class
<?xml version="1.0" encoding="utf-8"?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2"
assembly="Daisyblossoms.Domain"
namespace="Daisyblossoms">
<class name="Product"
table="product">
<id name="ProductID">
<generator class="assigned" />
</id>
<property name="Name" column="Name" />
<property name="Price" column="Price" />
</class>
</hibernate-mapping>
And my nhibernate.config:
<?xml version="1.0"?>
<hibernate-configuration xmlns="urn:nhibernate-configuration-2.2" >
<session-factory name="Daisyblossoms.Domain">
<property name="connection.provider">NHibernate.Connection.DriverConnectionProvider</property>
<property name="connection.driver_class">NHibernate.Driver.MySqlDataDriver</property>
<property name="generate_statistics">true</property>
<property name="current_session_context_class">web</property>
<property name="proxyfactory.factory_class">NHibernate.ByteCode.Castle.ProxyFactoryFactory, NHibernate.ByteCode.Castle</property>
<property name="dialect">NHibernate.Dialect.MySQL5Dialect</property>
<mapping assembly="Daisyblossoms.WebUI"/>
</session-factory>
</hibernate-configuration>
And my connectionsStrings part of Web.config:
<connectionStrings>
<add name="daisyblossoms" connectionString="Server=localhost;Port=3306;Database=dbName;Uid=user;Pwd=somePSWD;pooling=false;"
providerName="MySql.Data.MySqlClient"/>
Any thoughts what might be my issue?
Verify that hibernate.cfg.xml has output set to "Update if newer" and that your *.hbm.xml files are marked as Embedded Resources. Those are the two most common mistakes. It also sounds like you're trying to get a lot of moving parts working at the same time. You might want to simplify things to just get a console app to connect to MySQL using NHibernate. Something like this:
internal class Program {
private static void Main() {
var cfg = new Configuration();
cfg.Configure(); // Uses hibernate.cfg.xml by default.
// cfg.Configure("nhibernate.config"); // Or use this overload if you prefer your own name.
var sessionFactory = cfg.BuildSessionFactory();
using(var session = sessionFactory.OpenSession())
using(var tx = session.BeginTransaction()) {
var query = session.CreateCriteria<Product>().List();
query.ForEach(x => Console.WriteLine(x.Name));
tx.Commit();
}
Console.WriteLine("Press <ENTER> to exit...");
Console.ReadLine();
}
}
This would allow you to verify that your mappings and configuration files are correct without worrying about MVC, Ninject, etc. at the same time.

Turning HtmlUnit Warnings off

Do you know how can I turn Warnings, Notes, Errors in HtmlUnit off?
Put this somewhere around the start of your code; it will shut its dirty mouth:
LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(Level.OFF);
java.util.logging.Logger.getLogger("org.apache.commons.httpclient").setLevel(Level.OFF);
webClient = new WebClient(bv);
webClient.setCssEnabled(false);
webClient.setIncorrectnessListener(new IncorrectnessListener() {
#Override
public void notify(String arg0, Object arg1) {
// TODO Auto-generated method stub
}
});
webClient.setCssErrorHandler(new ErrorHandler() {
#Override
public void warning(CSSParseException exception) throws CSSException {
// TODO Auto-generated method stub
}
#Override
public void fatalError(CSSParseException exception) throws CSSException {
// TODO Auto-generated method stub
}
#Override
public void error(CSSParseException exception) throws CSSException {
// TODO Auto-generated method stub
}
});
webClient.setJavaScriptErrorListener(new JavaScriptErrorListener() {
#Override
public void timeoutError(HtmlPage arg0, long arg1, long arg2) {
// TODO Auto-generated method stub
}
#Override
public void scriptException(HtmlPage arg0, ScriptException arg1) {
// TODO Auto-generated method stub
}
#Override
public void malformedScriptURL(HtmlPage arg0, String arg1, MalformedURLException arg2) {
// TODO Auto-generated method stub
}
#Override
public void loadScriptError(HtmlPage arg0, URL arg1, Exception arg2) {
// TODO Auto-generated method stub
}
});
webClient.setHTMLParserListener(new HTMLParserListener() {
#Override
public void warning(String arg0, URL arg1, int arg2, int arg3, String arg4) {
// TODO Auto-generated method stub
}
#Override
public void error(String arg0, URL arg1, int arg2, int arg3, String arg4) {
// TODO Auto-generated method stub
}
});
webClient.setThrowExceptionOnFailingStatusCode(false);
webClient.setThrowExceptionOnScriptError(false);
The code in Arsen Zahray's answer helped in removing almost all the logs generated by HtmlUnit.
But one edit helps to remove them all. Use:
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);
instead of:
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(Level.OFF);
To remove all output from the latest version of HtmlUnit you just have to add these lines in a static block or in your main class:
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);
System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
It is NOT needed to override any method as some other answers state.
Try the following code to turn the logging level down to off:
java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(Level.OFF);
Here you can get info on how to manipulate logging of HtmlUnit.
This is what I added to my log4j.properties in order to disable verbose debugging messages from HtmlUnit components:
# Set specific logger levels.
log4j.logger.org.mortbay.log=fatal
log4j.logger.org.apache.http=fatal
log4j.logger.org.apache.http.headers=fatal
log4j.logger.org.apache.http.wire=fatal
# For HttpClient 3, which is used by FirefoxDriver
log4j.logger.httpclient.wire=fatal
log4j.logger.org.apache.commons=fatal
log4j.logger.com.gargoylesoftware.htmlunit=fatal
log4j.logger.com.gargoylesoftware.htmlunit.WebTestCase=fatal
# Change this to TRACE when enabling the debugger.
log4j.logger.com.gargoylesoftware.htmlunit.javascript.DebugFrameImpl=fatal
I am using the code below and it works perfectly:
LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(Level.OFF);
java.util.logging.Logger.getLogger("org.apache.commons.httpclient").setLevel(Level.OFF);
Just add this string to your log4.properties:
log4j.logger.com.gargoylesoftware.htmlunit=fatal
Turn the loggers off. But that is not a good solution, since you might want to have some uncommon issues in the logs.
I know HtmlUnit produces a lot of unimportant exceptions, warnings, etc. You can suppress of few of those using:
client.getOptions().setThrowExceptionOnFailingStatusCode(false);
client.getOptions().setThrowExceptionOnScriptError(false);
client.getOptions().setPrintContentOnFailingStatusCode(false);
Now in HtmlUnit 2.9, WebClient.setCssErrorHandler(new SilentCssErrorHandler()) can conveniently ignore the warnings and errors. For example:
#Override
protected WebClient modifyWebClient(WebClient client) {
// currently does nothing, but may be changed in future versions
WebClient modifiedClient = super.modifyWebClient(client);
modifiedClient.getOptions().setThrowExceptionOnScriptError(false);
modifiedClient.setCssErrorHandler(new SilentCssErrorHandler());
return modifiedClient;
}
Have a look at the docs.
There is a sample log4 file used by the test suite, you can find it here, you can disable everything if you wish.
This worked for me:
#Test
public void homePage() throws Exception {
final WebClient webClient = new WebClient();
webClient.setThrowExceptionOnScriptError(false);
final HtmlPage page = webClient.getPage("http://localhost:8080/web/guest/home");
Try adding this to your code:
LogFactory.getFactory().setAttribute("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
Basically, this makes the logger log to NoOpLog, which doesn't write the log information anywhere.
I must be doing something different to everyone above. I have htmlunit set up as a Spring project currently and removing the logs required adding a logback.xml to my resources dir. Add the following as logback.xml to your main/java/resources dir - this will only output INFO level log statements and nothing below (When to use the different log levels)
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!--<include resource="org/springframework/boot/logging/logback/base.xml"/>-->
<!--<logger name="org.springframework.web" level="INFO"/>-->
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] [%logger{20}] %-5level - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
<file>gdaxDesktop.log</file>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %logger{20} %-5level - %msg%n</pattern>
</encoder>
</appender>
<root level="info">
<!--<appender-ref ref="FILE"/>-->
<appender-ref ref="CONSOLE"/>
</root>
</configuration>
One option which worked well for me is to change the HtmlUnit logger to log to a different file just so that I have those errors in case I need to refer it some time and it also doesn't clutter up my main logs.
Below is the log4j change I made to log4j.properties:
log4j.logger.com.gargoylesoftware.htmlunit=ERROR, HTMLUNIT
log4j.additivity.com.gargoylesoftware.htmlunit=false
log4j.appender.HTMLUNIT = org.apache.log4j.RollingFileAppender
log4j.appender.HTMLUNIT.layout=org.apache.log4j.PatternLayout
log4j.appender.HTMLUNIT.layout.ConversionPattern=%m%n
log4j.appender.HTMLUNIT.File=logs/HtmlUnitLog4j.log
log4j.appender.HTMLUNIT.MaxFileSize=5MB
log4j.appender.HTMLUNIT.MaxBackupIndex=5
If you don't need JavaScript support, it is the easiest way to disable it:
WebClient client = new WebClient(BrowserVersion.BEST_SUPPORTED);
client.getOptions().setThrowExceptionOnFailingStatusCode(false);
client.getOptions().setPrintContentOnFailingStatusCode(false);
client.getOptions().setThrowExceptionOnScriptError(false);
client.getOptions().setJavaScriptEnabled(false);
client.setCssErrorHandler(new SilentCssErrorHandler());
client.setHTMLParserListener(new HTMLParserListener() {
#Override
public void error(String message, URL url, String html, int line, int column, String key) {
}
#Override
public void warning(String message, URL url, String html, int line, int column, String key) {
}
});
You also disable exceptions and log on failing status codes, JavaScript, CSS errors and HTML parse errors.
If you need JavaScript support you can use a custom implementation for JavaScript errors:
client.setJavaScriptErrorListener(new JavaScriptErrorListener() {
#Override
public void timeoutError(HtmlPage arg0, long arg1, long arg2) {
}
#Override
public void scriptException(HtmlPage arg0, ScriptException arg1) {
}
#Override
public void malformedScriptURL(HtmlPage arg0, String arg1, MalformedURLException arg2) {
}
#Override
public void loadScriptError(HtmlPage arg0, URL arg1, Exception arg2) {
}
});
If you don't need you also can just disable it:
client.getOptions().setCssEnabled(false);
So there is no need to configure any other Logger.
I am using spring-boot, only solved with this:
import com.gargoylesoftware.htmlunit.SilentCssErrorHandler;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.javascript.SilentJavaScriptErrorListener;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
#Configuration
public class HtmlUnitConfiguration {
#Bean
public WebClient webClient() {
WebClient webClient = new WebClient();
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener());
webClient.setCssErrorHandler(new SilentCssErrorHandler());
return webClient;
}
}
And then calling the bean with #Autowired or in class constructor. And without this line:
webClient.getOptions().setThrowExceptionOnScriptError(false);
The two lines under it will throw a bizarre error. This line has the magic.
I'm adding this answer for logback users:
<!-- LOG "com.gargoylesoftware.htmlunit.javascript*" at OFF (closed) level -->
<logger name="com.gargoylesoftware.htmlunit.javascript"
level="OFF" additivity="false">
<appender-ref ref="RollingFile" />
<appender-ref ref="Console" />
</logger>
Setting the logger level off as previous answers from apache logging and java util logging did not work for me (at least when im running tests on my local computer using htmlunit).
And passing the dummy listener as javascriptErrorListener for WebClient instance also was not working for me.
So it was the only solution for closing the annoying javascript exception logs.
Try this, it worked for me.
WebClient webClient = new WebClient();
webClient.getOptions().setJavaScriptEnabled(false);
webClient.getOptions().setCssEnabled(false);