Which system information are useful - especially when tracing an exception or other problems down - in a java application?
I am thinking about details about exceptions, java/os information, memory/object consumptions, io information, environment/enchodings etc.
Besides the obvious - the exception stack trace - the more info you can get is better. So you should get all the system properties as well as environment variables. Also if your application have some settings, get all their values. Of course you should put all this info into your log file, I used System.out her for simplicity:
System.out.println("----Java System Properties----");
System.getProperties().list(System.out);
System.out.println("----System Environment Variables----");
Map<String, String> env = System.getenv();
Set<String> keys = env.keySet();
for (String key : keys) {
System.out.println(key + "=" + env.get(key));
}
For most cases this will be "too much" information, but for most cases the stack trace will be enough. Once you will get a tough issue you will be happy that you have all that "extra" information
Check out the Javadoc for System.getProperties() which documents the properties that are guaranteed to exist in every JVM.
For pure java applications:
System.getProperty("org.xml.sax.driver")
System.getProperty("java.version")
System.getProperty("java.vm.version")
System.getProperty("os.name")
System.getProperty("os.version")
System.getProperty("os.arch")
In addition, for java servlet applications:
response.getCharacterEncoding()
request.getSession().getId()
request.getRemoteHost()
request.getHeader("User-Agent")
pageContext.getServletConfig().getServletContext().getServerInfo()
One thing that really helps me- to see where my classes are getting loaded from.
obj.getClass().getProtectionDomain().getCodeSource().getLocation();
note: protectiondomain can be null as can code source so do the needed null checks
Related
Chisel throws an exception with an elaboration error message. The following is a result of my code as an example.
chisel3.core.Binding$ExpectedHardwareException: data to be connected 'chisel3.core.Bool#81' must be hardware, not a bare Chisel type. Perhaps you forgot to wrap it in Wire(_) or IO(_)?
This exception message is interesting because 81 behind chisel3.core.Bool# looks like ID, not hashcode.
Indeed, Data type extends HasId trait which has _id field, and
_id field seems to generate a unique ID for each components.
I've thought Data type overrides toString to make string that has type#ID, but it does not override. That is why $node in below code should not be able to use ID.
throw Binding.ExpectedHardwareException(s"$prefix'$node' must be hardware, " +
"not a bare Chisel type. Perhaps you forgot to wrap it in Wire(_) or IO(_)?")
Instead of toString, toNamed method exists in Data. However, this method seems to be called to generate a firrtl code, not to convert component into string.
Why can Data type show its ID?
If it is not ID, but exactly hashcode, this question is from my misunderstanding.
I think you should take a look at Chisel PR #985. It changes the way that Data's toString method is implemented. I'm not sure if it answers your question directly but it's possible this will make the meaning and location of the error clearer. If not you should comment on it.
Scala classes come with a default toString method that is of the form className#hashCode.
As you noted, the chisel3.core.Bool#81 sure looks like it's using the _id rather than the hashCode. That's because in the most recently published version of Chisel (3.1.6), the hashcode was the id! You can see this if you inspect the source files at the tag for that version: https://github.com/freechipsproject/chisel3/blob/dc4200f8b622e637ec170dc0728c7887a7dbc566/chiselFrontend/src/main/scala/chisel3/internal/Builder.scala#L81
This is no longer the case on master which probably the source of any confusion! As Chick noted, we have just changed the .toString method to be more informative than the default; expect more informative representations in 3.2.0!
(Possibly a duplicate of Can't send a keyedMessage to brokers with partitioner.class=kafka.producer.DefaultPartitioner, although the OP of that question didn't mention kafka-python. And anyway, it never got an answer.)
I have a Python program that has been successfully (for many months) sending messages to the Kafka broker, using essentially the following logic:
producer = kafka.KafkaProducer(bootstrap_servers=[some_addr],
retries=3)
...
msg = json.dumps(some_message)
res = producer.send(some_topic, value=msg)
Recently, I tried to upgrade it to send messages to different partitions based on a definite key value extracted from the message:
producer = kafka.KafkaProducer(bootstrap_servers=[some_addr],
key_serializer=str.encode,
retries=3)
...
try:
key = some_message[0]
except:
key = None
msg = json.dumps(some_message)
res = producer.send(some_topic, value=msg, key=key)
However, with this code, no messages ever make it out of the program to the broker. I've verified that the key value extracted from some_message is always a valid string. Presumably I don't need to define my own partitioner, since, according to the documentation:
The default partitioner implementation hashes each non-None key using the same murmur2 algorithm as the java client so that messages with the same key are assigned to the same partition.
Furthermore, with the new code, when I try to determine what happened to my send by calling res.get (to obtain a kafka.FutureRecordMetadata), that call throws a TypeError exception with the message descriptor 'encode' requires a 'str' object but received a 'unicode'.
(As a side question, I'm not exactly sure what I'd do with the FutureRecordMetadata if I were actually able to get it. Based on the kafka-python source code, I assume I'd want to call either its succeeded or its failed method, but the documentation is silent on the point. The documentation does say that the return value of send "resolves to" RecordMetadata, but I haven't been able to figure out, from either the documentation or the code, what "resolves to" means in this context.)
Anyway: I can't be the only person using kafka-python 1.3.3 who's ever tried to send messages with a partitioning key, and I have not seen anything on teh Intertubes describing a similar problem (except for the SO question I referenced at the top of this post).
I'm certainly willing to believe that I'm doing something wrong, but I have no idea what that might be. Is there some additional parameter I need to supply to the KafkaProducer constructor?
The fundamental problem turned out to be that my key value was a unicode, even though I was quite convinced that it was a str. Hence the selection of str.encode for my key_serializer was inappropriate, and was what led to the exception from res.get. Omitting the key_serializer and calling key.encode('utf-8') was enough to get my messages published, and partitioned as expected.
A large contributor to the obscurity of this problem (for me) was that the kafka-python 1.3.3 documentation does not go into any detail on what a FutureRecordMetadata really is, nor what one should expect in the way of exceptions its get method can raise. The sole usage example in the documentation:
# Asynchronous by default
future = producer.send('my-topic', b'raw_bytes')
# Block for 'synchronous' sends
try:
record_metadata = future.get(timeout=10)
except KafkaError:
# Decide what to do if produce request failed...
log.exception()
pass
suggests that the only kind of exception it will raise is KafkaError, which is not true. In fact, get can and will (re-)raise any exception that the asynchronous publishing mechanism encountered in trying to get the message out the door.
I also faced the same error. Once I added json.dumps while sending the key, it worked.
producer.send(topic="first_topic", key=json.dumps(key)
.encode('utf-8'), value=json.dumps(msg)
.encode('utf-8'))
.add_callback(on_send_success).add_errback(on_send_error)
I have test cases defined in an Excel sheet. I am reading a string from this sheet (my expected result) and comparing it to a result I read from a database (my actual result). I then use AssertEquals(expectedResult, actualResult) which prints any errors to a log file (i'm using log4j), e.g. I get java.lang.AssertionError: Different output expected:<10> but was:<7> as a result.
I now need to write that result into the Excel sheet (the one that defines the test cases). If only AssertEquals returned String, with the AssertionError text that would be great, as I could just write that immediately to my Excel sheet. Since it returns void though I got stuck.
Is there a way I could read the AssertionError without parsing the log file?
Thanks.
I think you're using junit incorrectly here. THis is why
assertEquals not AssertEquals ( ;) )
you shouldnt need to log. You should just let the assertions do their job. If it's all green then you're good and you dont need to check a log. If you get blue or red (eclipse colours :)) then you have problems to look at. Blue is failure which means that your assertions are wrong. For example you get 7 but expect 10. Red means error. You have a null pointer or some other exception that is throwing while you are running
You should need to read from an excel file or databse for the unit tests. If you really need to coordinate with other systems then you should try and stub or mock them. With the unit test you should work on trying to testing the method in code
if you are bootstrapping on JUnit to try and compare an excel sheet and database then I would ust export the table in excel as well and then just do a comparison in excel between columns
Reading from/writing to files is not really what tests should be doing. The input for the tests should be defined in the test, not in the external file which can change - this can either introduce false negatives or even worse false positives (making your tests effectively useless while also giving false confidence that everything is ok because tests are green).
Given your comment (a loop with 10k different parameters coming from file), I would recommend converting this excel file into JUnit Parameterized test. You may want to put the array definition in another class, because 10k lines is quite a lot.
If it is some corporate bureaucracy, and you need to have this excel file, then it makes sense to not write a classic "test". I would recommend just a main method that does the job - reads the file, runs the code, checks the output using simple if (output.equals(expected)) and then writes back to file.
Wrap your AssertEquals(expectedResult, actualResult) with try catch
in catch
catch(AssertionError e){
//deal with e.getMessage or etc.
}
But it not good idea for some reasons, I guess.
And try google something like soft assert
Documentation on assertEquals is pretty clear on what the method does:
Asserts that two objects are equal. If they are not, an AssertionError
without a message is thrown.
You need to wrap the assertion with try-catch block and in the exception handling do Your logging. You will need to make Your own message using the information from the specific test case, but this what You asked for.
Note:
If expected and actual are null, they are considered equal.
Under the debugger I have a case where DbContext.ChangeTracker.Entry(e) returns an entry with a State of Detached. When I enumerate the results of DbContext.ChangeTracker.Entries() and the entries of the underlying ObjectContext when looking for e, I find an entry with a State of Unchanged (expected).
What is going on?
Here are some additional details:
using POCO entities.
change Tracking is on
proxy creation is off
lazy loading is off
problem does not occur when saving an entity for the first time (e.g. adding to context); occurs when getting old entity into context then trying to make changes to it. This is an aggregate root with many "reference" entities that aren't supposed to change
Equals is overridden on the entities and IEquatable<T> is implemented. That code is generated by T4.
I am using a generic repository implementation that is declaratively configured to generate rules for saving (e.g. whether entities should be added, attached/modified, attached/unchanged. It seems to be doing this in the right order. For example the aggregate root is added/attached last because attaching it first brings in other entities in a modified state (adding those first as unchanged prevents this).
(Answered in a question edit. Converted to a community wiki answer. See Question with no answers, but issue solved in the comments (or extended in chat) )
The OP wrote:
I have "solved" the problem, but I still want to know what's going on, because my solution doesn't do anything to address the root cause. My "solution" looks for an entity in the change tracker (I have also looked via the context.Entry() and context.Set().Local -- when I do it with this code (I did it as a loop instead of LINQ so I could set breakpoints), it works:
private DbEntityEntry GetChangeTrackedEntry(IEntity mine, Type type)
{
foreach (var en in context.ChangeTracker.Entries())
{
if (en.Entity.GetType() != type)
continue;
if (((IEntity)en.Entity).Id != mine.Id)
continue;
return en;
}
return null;
}
When I attempt to lookup an entity (via change tracker, the set, etc.) via using mine directly, that's when I end up with a detached case.
I thought perhaps there were cases of EF using ReferenceEquals but #Ladislav's comment may indicate something wrong with Equals implementation.
If anyone has a further explanation they can edit that into this community wiki answer.
Imagine an instance of some lookup of configuration settings called "configuration", used like this:
if(! string.IsNullOrEmpty(configuration["MySetting"])
{
DoSomethingWithTheValue(configuration["MySetting"]);
}
The meaning of the setting is overloaded. It means both "turn this feature on or off" and "here is a specific value to do something with". These can be decomposed into two settings:
if(configuration["UseMySetting"])
{
DoSomethingWithTheValue(configuration["MySetting"]);
}
The second approach seems to make configuration more complicated, but slightly easier to parse, and it separate out the two sorts of behaviour. The first seems much simpler at first but it's not clear what we choose as the default "turn this off" setting. "" might actually a valid value for MySetting.
Is there a general best practice rule for this?
I find the question to be slightly confusing, because it talks about (1) parsing, and (2) using configuration settings, but the code samples are for only the latter. That confusion means that my answer might be irrelevant to what you intended to ask. Anyway...
I suggest an approach that is illustrated by the following pseudo-code API (comments follow afterwards):
class Configuration
{
void parse(String fileName);
boolean exists(String name);
String lookupString(String name);
String lookupString(String name, String defaultValue);
int lookupInt(String name);
int lookupInt(String name, int defaultValue);
float lookupFloat(String name);
float lookupFloat(String name, float defaultValue);
boolean lookupBoolean(String name);
boolean lookupBoolean(String name, boolean defaultValue);
... // more pairs of lookup<Type>() operations for other types
}
The parse() operation parses a configuration file and stores the parsed data in a convenient format, for example, in a map or hash-table. (If you want, parse() can delegate the parsing to a third-party library, for example, a parser for XML, Java Properties, JSON, .ini files or whatever.)
After parsing is complete, your application can invoke the other operations to retrieve/use the configuration settings.
A lookup<Type>() operation retrieves the value of the specified name and parses it into the specified type (and throws an exception if the parsing fails). There are two overloadings for each lookup<Type>() operation. The version with one parameter throws an exception if the specified variable does not exist. The version with an extra parameter (denoting a default value) returns that default value if the specified variable does not exist.
The exists() operation can be used to test whether a specified name exists in the configuration file.
The above pseudo-code API offers two benefits. First, it provides type-safe access to configuration data (which wasn't a stated requirement in your question, but I think it is important anyway). Second, it enables you to distinguish between "variable is not defined in configuration" and "variable is defined but its value happens to be an empty string".
If you have already committed yourself to a particular configuration syntax, then just implement the above Configuration class as a thin wrapper around a parser for the existing configuration syntax. If you haven't already chosen a configuration syntax and if your project is in C++ or Java, then you might want to look at my Config4* library, which provides a ready-to-use implementation of the above pseudo-code class (with a few extra bells and whistles).