Any idea how to approach this issue:
I have classes with a lot of unit tests
Those classes use Log4j with following levels: DEBUG, INFO, ERROR
When I simply run unit test and calculate coverage depending on the configuration I will not get through all the logging statements. I use if (logger.isDebugEnabled()) ...
I have already had problems in the past when application started to fail on production, and we changed the log level do DEBUG to get more insight, but this caused NullPointerExceptions (bad coding of course)
In theory we can run all the tests with different levels of logging.
Any smart way to do it or any tools that help to solve this issue?
As suggested in comments:
use a test-specific log4j configuration that sets the level to DEBUG.
You also mention:
we changed the log level do DEBUG to get more insight, but this caused NullPointerExceptions
That sounds like a bug. Identifying bugs like this is the whole point of tests: now you can fix them.
This may be a stupid question, as most of my programming consists of one-man scientific computing research prototypes and developing relatively low-level libraries. I've never programmed in the large in an enterprise environment before. I've always wondered, what are the main things that logging libraries make substantially easier than just using good old fashioned print statements or file output, simple programming logic and a few global variables to determine how verbosely things get logged? How do you know when a few print statements or some basic file output ain't gonna cut it and you need a real logging library?
Logging helps debug problems especially when you move to production and problems occur on people's machines you can't control. Best laid plans never survive contact with the enemy, and logging helps you track how that battle went when faced with real world data.
Off the shel logging libraries are easy to plug in and play in less than 5 minutes.
Log libraries allow for various levels of logging per statement (FATAL, ERROR, WARN, INFO, DEBUG, etc).
And you can turn up or down logging to get more of less information at runtime.
Highly threaded systems help sort out what thread was doing what. Log libraries can log information about threads, timestamps, that ordinary print statements can't.
Most allow you to turn on only portions of the logging to get more detail. So one system can log debug information, and another can log only fatal errors.
Logging libraries allow you to configure logging through an external file so it's easy to turn on or off in production without having to recompile, deploy, etc.
3rd party libraries usually log so you can control them just like the other portions of your system.
Most libraries allow you to log portions or all of your statements to one or many files based on criteria. So you can log to both the console AND a log file.
Log libraries allow you to rotate logs so it will keep several log files based on many different criteria. Say after the log gets 20MB rotate to another file, and keep 10 log files around so that log data is always 100MB.
Some log statements can be compiled in or out (language dependent).
Log libraries can be extended to add new features.
You'll want to start using a logging libraries when you start wanting some of these features. If you find yourself changing your program to get some of these features you might want to look into a good log library. They are easy to learn, setup, and use and ubiquitous.
There are used in environments where the requirements for logging may change, but the cost of changing or deploying a new executable are high. (Even when you have the source code, adding a one line logging change to a program can be infeasible because of internal bureaucracy.)
The logging libraries provide a framework that the program will use to emit a wide variety of messages. These can be described by source (e.g. the logger object it is first sent to, often corresponding to the class the event has occurred in), severity, etc.
During runtime the actual delivery of the messaages is controlled using an "easily" edited config file. For normal situations most messages may be obscured altogether. But if the situation changes, it is a simpler fix to enable more messages, without needing to deploy a new program.
The above describes the ideal logging framework as I understand the intention; in practice I have used them in Java and Python and in neither case have I found them worth the added complexity. :-(
They're for logging things.
Or more seriously, for saving you having to write it yourself, giving you flexible options on where logs are store (database, event log, text file, CSV, sent to a remote web service, delivered by pixies on a velvet cushion) and on what is logged at runtime, rather than having to redefine a global variable and then recompile.
If you're only writing for yourself then it's unlikely you need one, and it may introduce an external dependency you don't want, but once your libraries start to be used by others then having a logging framework in place may well help your users, and you, track down problems.
I know that a logging library is useful when I have more than one subsystem with "verbose logging," but where I only want to see that verbose data from one of them.
Certainly this can be achieved by having a global log level per subsystem, but for me it's easier to use a "system" of some sort for that.
I generally have a 2D logging environment too; "Info/Warning/Error" (etc) on one axis and "AI/UI/Simulation/Networking" (etc) on the other. With this I can specify the logging level that I care about seeing for each subsystem easily. It's not actually that complicated once it's in place, indeed it's a lot cleaner than having if my_logging_level == DEBUG then print("An error occurred"); Plus, the logging system can stuff file/line info into the messages, and then getting totally fancy you can redirect them to multiple targets pretty easily (file, TTY, debugger, network socket...).
I'm trying to implement the advice found in this great blog post for batch processing in grails with MySQL. The problem that I'm having is that inclusion of periodic calls to session.clear() in my loop causes org.hibernate.LazyInitializationException's to be thrown. There's a quote down in the comments section of the page:
You’re second point about potentially
causing LIEs is absolutely true. If
you’re doing other things outside of
importing with the current thread,
you’ll want to make sure to reattach
any objects to the session after
you’re doing your clearing.
But how do I do that? Can anyone help me specifically understand how to "reattach any objects to the session after I'm done clearing?
I'm also interested in parallelizing the database insertion process so that I can take advantage of having a multi core processor. Can anyone provide advice on how to do that in Grails?
Grails has a few methods to help with this (they leverage hibernate under the covers).
If you know an object is detached, you can use the attach method to reconnect it.
If you've made changes to the object while it was detatched, you can use merge.
If for whatever reason, you're not sure if an object is attached to the session, you can use the link text method to find out if it is or isn't.
It might also be worth reviewing the Hibernate documentation on Session.
I'm looking for a way to collect a set of data that will only be used for debugging. I.e., the data should only be logged if I log an exception. When I get an exception argument with ILog.Error, Fatal or Debug I want to log the extra information. When logging other data with an exception, the extra information should not be logged.
I plan to use the GlobalContext or ThreadContext for building the dataset.
My idea was to hook into Log4Net and attach to an event I would imagine, to alter the message pattern to include contexts, but I can't find any event that would help me. Perhaps there is an easier way?
What do you think about the overall design of this? Am I on the right track or am I missing something?
If this way is good, how can I implement it?
Hooking in and changing the pattern doesn't feel quite right. I would suggest looking at the filter stuff. So you would set up your "ForMessagesWithExceptionsOnly" appender which uses the MessagesWithExceptionOnly filter. This appender would ofcourse only process messages that contains an exception.
To implement your MessagesWithExceptionOnly filter, have a look at FilterSkeleton.
I've recently started to play with Ruby on Rails which favours convention over configuration and relies on sensible defaults to tie various aspects of the application together.
I was thinking that it might be useful if this concept of sensible default configuration was used in general configation for various frameworks then it might save some development headache.
For example, in a .net app I usually want to log an exception in the windows event log using enterprise library exception handling block but if I don't explicity state the behaviour I want in a config file then EL will complain. I think that instead, if it can't find custom configuration then it should revert to a sensible default configuration, like logging my exception in the event log.
Would this be a good or bad concept for frameworks to adopt for their configuration?
I work a lot with a framework that does this exact thing. My trouble with this way of working is that:
the framework grew to having an excessive amount of configuration keys that are actually never used/set in a configuration file.
behavior of the software becomes implicit sometimes, I want to explicitly set the system to behave a certain way instead of having it fallback on some other code path due to a 'default'.
a missed typo in configuration key may result in a very long diagnostic session before figuring out what is going on.
When forgetting to set a configuration value I rather have the software tell me, instead of assuming some form of behavior that I might not at all be after.
I'd prefer a 'template' configuration file in which I change what I want and have the unchanged settings serve as the default.
Figuring which out which convention the software picked when debugging can be a lot of time wasted also.