Unexpected behaviour from Gson - json

I developed a small application that stores data coming from a device: I chose to store data in JSON format, and the serialization/deserialization of the data works just fine, even if it involves some custom types created by me...but only I work in the IDE (Eclipse, for that matter).
When I export a runnable JAR file though, the deserialization of the data encounters some kind of problem, because the software always throws this exception:
Caused by: java.lang.UnsupportedOperationException: Cannot allocate class LocalDateTime
at com.google.gson.internal.UnsafeAllocator$4.newInstance(UnsafeAllocator.java:104)
at com.google.gson.internal.ConstructorConstructor$14.construct(ConstructorConstructor.java:225)
... 88 common frames omitted
I thought I'd encounter problems with custom types, not a built-in one. At this point, I discovered two things:
if I use a full JRE 9 to run the JAR file, the exception is not thrown: I double checked the modules included in the custom JRE I created with Jlink.exe, and everything is included correctly. I still want to use a smaller JRE, so I did not investigate further yet (I guess this explains why in the IDE it works perfectly)
I added a custom deserializer to the Gson object (see below), with which I simply manually converted the JSON string into a valid data, and that avoided the exception on the LocalDateTime class...but the exception reappeared simply on another class, this time a custom-made one.
At this point, I guess I can simply add a deserializer for each data type that causes problem, but I'm wondering why the issue won't happen with a full JRE, and why a smaller JRE causes this, even if all the modules required are included. Maybe it's worth mentioning also that I added no custom serializer to the Gson object that saves the data, it is all serialized as per Gson default.
LocalDateTime deserializer:
#Override
public LocalDateTime deserialize(JsonElement json, java.lang.reflect.Type type,
JsonDeserializationContext jsonDeserializationContext) throws JsonParseException {
JsonObject joDate = json.getAsJsonObject().get("date").getAsJsonObject();
JsonObject joTime = json.getAsJsonObject().get("time").getAsJsonObject();
//JSON example: {"date":{"year":2019,"month":1,"day":9},"time":{"hour":6,"minute":14,"second":1,"nano":0}
return LocalDateTime.of(joDate.get("year").getAsInt(),
joDate.get("month").getAsInt(),
joDate.get("day").getAsInt(),
joTime.get("hour").getAsInt(),
joTime.get("minute").getAsInt(),
joTime.get("second").getAsInt(),
joTime.get("nano").getAsInt());
}
}
Jdeps.deps modules list:
com.google.gson
java.base
javafx.base
javafx.controls
javafx.fxml
javafx.graphics
org.slf4j
After the answer I received, I opened an issue here.

TL;DR
You need a runtime image (e.g. full JDK or something built with jlink) that includes the module jdk.unsupported.
Full Answer
GSON wants to create instances of classes it deserializes without calling any constructors (so nothing gets initialized without GSON saying so). This can't normally be done, but sun.misc.Unsafe offers a way to do this with the method allocateInstance. To that end, GSON needs an instance of sun.misc.Unsafe. The topmost frame in the call stack is from UnsafeAllocator, which uses common trickery to get Unsafe.
The problem is, sun.misc.Unsafe is in module jdk.unsupported, which is present in a full JDK but you won't usually find in runtime images.
When creating your runtime image with jlink, make sure to include the option --add-modules jdk.unsupported and you should be good to go.
Arguably, GSON should declare an optional dependency on jdk.unsupported with requires static.

I have faced the same issue when packing compose a desktop application.
update build.gradle file, add an unsupported module.
compose.desktop {
application {
mainClass = "MainKt"
nativeDistributions {
targetFormats(TargetFormat.Dmg, TargetFormat.Msi, TargetFormat.Deb)
packageName = "admin"
packageVersion = "1.0.0"
modules("java.sql")
modules("jdk.unsupported")
}
}
}

Related

Spring Boot JMS - Generic JSON messages without _type property

I'm implementing JMS in a Spring Boot application. Everything is going well. However I'm very surprised at the tight coupling between JSON messages and Java objects. I am looking for some direction on a more flexible solution.
Going through examples and using the MappingJackson2MessageConverter, everything is great as long as you are sending and receiving in the same application. Under the covers it's extremely tightly coupled to the java object. If I have a simple java object called person:
package acme.receivingapp.dto;
public class Person {
private String firstName;
private String lastName;
...
}
When the JmsTemplate turns that into a message the JSON looks generic enough:
{"firstName":"John", "lastName":"Doe"}
However it includes this property:
"_type" : "acme.superapp.dto.Person"
If the JmsListener isn't using that exact Java class, it throws an exception. That's true even if the class is functionally the same as in this example where it's effectively the same class but just in a different package:
package wonderco.sendingapp.dto;
public class Person {
private String firstName;
private String lastName;
...
}
We will be receiving messages from many external entities from mainframes, python apps, .Net, etc. I cannot require them to include our object types in a _type property.
I could create my own MessageConverter specifically for a Person object, but if we have hundreds of more messages / java classes it would be unwieldy to have so many message converters. I would need to design something more generic that can work for any type of JSON message / java class.
Before I go down the path of designing my own generic solution is there anything more generic that works like Spring RestControllers and Spring RestTemplates in the sense that the JSON messages aren't so tightly coupled to the very specific Java classes? I feel like I can't possibly be the first person trying to crack this nut.
I think I've got a handle on this. I'll try to explain it to hopefully help the next person who is new to Spring / JMS.
As M.Deinum points out, unlike a REST endpoint, a queue could potentially contain many different types of messages. Even if your implementation will only have one type of message per queue. Because queues allow any number of different messages that was the design for the provided MappingJackson2MessageConverter. Because the assumption was made there will always be multiple types of messages, there must be a mechanism to determine how to unmarshal the JSON for different types of messages into the correct type of Java Object.
All the examples you'll find of using a MappingJackson2MessageConverter will have this setup in them:
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTypeIdPropertyName("_type");
That's telling the message converter to set the object type in a property called _type when creating a message or to read the object type from that property when reading a message. There's no magic in that _type property. It's not a standard. It's just what the Spring folks used in their examples and then a bazillion people cut and pasted it. So for your own messages, you can change that to a more appropriate property name if you like. So in my example, I might call the property acme_receivingapp_message_type if I wanted. I would then tell the external entities sending me messages to include that property with the message type.
By default, the MappingJackson2MessageConverter will write the object type into whatever property name you chose (_type or whatever) as the fully qualified class name. In my example, it's acme.receivingapp.dto.Person. When a message is received, it looks at the type property to determine what type of Java object to create from the JSON.
Pretty straightforward so far, but still not very convenient if the people sending me messages are not using Java. Even if I can convince everyone to send me acme.receivingapp.dto.Person, what happens if I refactor that class from Person to Human? Or even just restructure the packages? Now I've got to go back and tell the 1,000 external entities to stop sending the property as acme.receivingapp.dto.Person and now send it as acme.receivingapp.dto.Human?
Like I stated in my original question, the message and Java class are being very tightly coupled together which doesn't work when you are dealing with external systems/entities.
The answer to my problem is right in the name of the **Mapping**Jackson2MessageConverter message converter. The key there is the "mapping". Mapping refers to mapping message types to Java classes which is what we want. It's just that, by default, because no mapping information is provided, the MappingJackson2MessageConverter simply uses the fully qualified java class names for creating and receiving messages. All we need to do is provide the mapping information to the message converter so it can map from friendly message-types (e.g.. "Person") to specific classes within our application (e.g. acme.receivingapp.dto.Person).
If you wanted your external systems/entities that will be sending you messages to simply include the property acme_receivingapp_message_type : Person and you wanted that unmarshalled to an acme.receivingapp.dto.Person object when it's received on your end, you'd setup your message converter like this:
#Bean
public MessageConverter jacksonJmsMessageConverter() {
MappingJackson2MessageConverter converter = new MappingJackson2MessageConverter();
converter.setTargetType(MessageType.TEXT);
converter.setTypeIdPropertyName("acme_receivingapp_message_type");
// Set up a map to convert our friendly message types to Java classes.
Map<String, Class<?>> typeIdMap = new HashMap<>();
typeIdMap.put("Person", acme.receivingapp.dto.Person.class);
converter.setTypeIdMappings(typeIdMap);
return converter;
}
That solves the problem of tight coupling between the message type property and Java class names. But what if you'll only be dealing with a single message type in your queue and don't want the people sending messages to have to include any property to indicate the message type? Well MappingJackson2MessageConverter simply doesn't support that. I tried using a "null" key in the map and then leaving the property off the message and unfortunately it doesn't work. I wish it did support that "null" mapping to use when the property wasn't present.
If you have the scenario where your queue will only deal with one type of message and you don't want the sender to have to include a special property to indicate the message type, you'll likely want to write your own message converter. That convertor will blindly unmarshal the JSON to the one java class you'll always be dealing with. Or maybe you opt to just receive it as a TextMessage and unmarshal it in your listener.
Hopefully this helps someone because I found it quite confusing initially.
I'm reacting to this thread because I have exactly the same feeling!
Why spring isn't able to deserialise the event based on the prototype function that implement the #JmsListener ?
If you have a function like
#JmsListener(destination = "#{beanQueue.queueName}")
public void onEvent(MyEvent event) {
// Do what you want
}
Why do we need to explicitly define the _type property that allow to know the java type output? We can extract it from the parameter function.
I don't perform deep search under the hood, but it look reasonable to me
[EDIT] After some debugging and some quick research, I found a solution. I'm not convinced that it's the more elegant solution, but at least it allow to have a kind of generic converter.
Convert all to String
Convert automatically all event to the java.lang.String type, in order to have a generic _type for all consumer
It can be done by encapsulate the actual MappingJackson2MessageConverter
#Bean
public MessageConverter jsonMessageConverter() {
final ObjectMapper objectMapper = new ObjectMapper();
// Setup the object mapper
final MappingJackson2MessageConverter messageConverter = new MappingJackson2MessageConverter();
messageConverter.setObjectMapper(objectMapper);
messageConverter.setTargetType(MessageType.TEXT);
messageConverter.setTypeIdPropertyName("_type");
return new MessageConverter() {
#Override
public Message toMessage(Object object, Session session) throws JMSException, MessageConversionException {
try {
final String stringValue = objectMapper.writeValueAsString(object);
return messageConverter.toMessage(stringValue, session);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
}
#Override
public Object fromMessage(Message message) throws JMSException, MessageConversionException {
return messageConverter.fromMessage(message);
}
};
}
Read all from String
In order to read all from String without need to rewrite all the actual spring implementation, we can take benefit of the ConversionService
#Override
public void addFormatters(FormatterRegistry registry) {
registry.addConverter((Converter<String, MyEvent>) source -> {
try {
return new ObjectMapper().readValue(source, MyEvent.class);
} catch (JsonProcessingException e) {
throw new RuntimeException(e);
}
});
}
Limitation
There is some limitation, since the event is now transfered as a String, which is not elegant at all..
I don't actually investigate with the org.springframework.jms.support.converter.MessageType that allow to define other type of message.
In addition, it force the client to always define a converter for all event that are listening inside the application.

Grails JSON marhsaling using introspection causes severe bottleneck on Classloader.loadClass()

I am using Grails 2.2.4 and have a controller endpoint which converts a domain object list to JSON. Under load (as little as 5 concurrent requests) the marshaling performance is very poor. Taking thread dumps the threads are blocked on:
java.lang.ClassLoader.loadClass(ClassLoader.java:291)
There is a single marhsaler registered to marshal all domain objects using reflection and introspection. Realizing that reflection and introspection is slower than direct method calls, I am still seeing unexpected behavior in that the class loader is caller every time and in turn blocking occurs. An example stacktrace is as follows:
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:291)
- waiting to lock <785e31830> (a org.grails.plugins.tomcat.ParentDelegatingClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.beans.Introspector.instantiate(Introspector.java:1470)
at java.beans.Introspector.findExplicitBeanInfo(Introspector.java:431)
at java.beans.Introspector.<init>(Introspector.java:380)
at java.beans.Introspector.getBeanInfo(Introspector.java:167)
at java.beans.Introspector.getBeanInfo(Introspector.java:230)
at java.beans.Introspector.<init>(Introspector.java:389)
at java.beans.Introspector.getBeanInfo(Introspector.java:167)
at java.beans.Introspector.getBeanInfo(Introspector.java:230)
at java.beans.Introspector.<init>(Introspector.java:389)
at java.beans.Introspector.getBeanInfo(Introspector.java:167)
at java.beans.Introspector.getBeanInfo(Introspector.java:230)
at java.beans.Introspector.<init>(Introspector.java:389)
at java.beans.Introspector.getBeanInfo(Introspector.java:167)
at org.springframework.beans.CachedIntrospectionResults.<init>(CachedIntrospectionResults.java:217)
at org.springframework.beans.CachedIntrospectionResults.forClass(CachedIntrospectionResults.java:149)
at org.springframework.beans.BeanWrapperImpl.getCachedIntrospectionResults(BeanWrapperImpl.java:324)
at org.springframework.beans.BeanWrapperImpl.getPropertyValue(BeanWrapperImpl.java:727)
at org.springframework.beans.BeanWrapperImpl.getPropertyValue(BeanWrapperImpl.java:721)
at org.springframework.beans.PropertyAccessor$getPropertyValue.call(Unknown Source)
at com.ngs.id.RestDomainClassMarshaller.extractValue(RestDomainClassMarshaller.groovy:203)
...
...
A simple benchmark loading the same endpoint with the same parameters results in the loadClass call.
I was under the impression the classes would be at least cached by the class loader and not loaded on every method call to get the property to be marshaled.
The code to retrieve the property value is as follows:
BeanWrapper beanWrapper = PropertyAccessorFactory.forBeanPropertyAccess(domainObject);
return beanWrapper.getPropertyValue(property.getName());
Is there a configuration setting that is needed to ensure the classes are only loaded once? or perhaps a different way to get the property that doesn't result in class loading every time? Or perhaps a more performant way to achieve this?
Writing a custom marshaler per domain class would avoid the reflection and introspection but is going to be a lot of repeat code.
Appreciate any input.
So after much digging this is what I found out.
Using the BeanUtils.getPropertyDescriptors and getValue will always try and find a BeanInfo class describing the bean using the class loader. In this case we don't provide BeanInfo classes for our grails domain classes so this call is redundant. I found some information where you can provide a custom BeanInfoFactory to bypass this and exclude your packages but I couldn't find how to configure it with Grails.
Also searching the springframework documentation there is a configuration option you can pass Introspector.IGNORE_ALL_BEANINFO that will tell CachedIntorspectionResults to never look up the bean classes. However this was not available in version 3.1.4 of springframework which was current for grails 2.2.4. The newer versions do appear to have this option.
So, if using BeanUtils you can't by pass this initial lookup on the class loader. However subsequent loaders should be cached by CachedIntrospectionResults. Unfortunately this doesn't happen in our scenario. There looks to be a bug in the test to see if the lookup is cacheable. See more info on this below.
The fix was ultimately to fall back to use pure reflection. Rather than use:
beanWrapper.getPropertyValue(property.getName());
To use:
PropertyDescription pd = BeanUtils.getPropertyDescriptor(domainObject.getClass(), property.getName())
pd.readMethod.invoke(domainObject)
Where the pd is cached.
After fixing this the profiler still showed a lack of caching on CachedIntorspectionResults for the out of the box grails marshaller. This was due to the bad caching implementation in CachedIntrospectionResults. The work around for this was to add the correct class loader to the acceptedClassLoaders in the CachedIntrospectionResults.
public class EnhanceCachedIntrospectionResultsAcceptedClassLoadersListener implements ServletContextListener {
public void contextInitialized(ServletContextEvent event) {
CachedIntrospectionResults.acceptClassLoader(Thread.currentThread().getContextClassLoader().getParent());
}
public void contextDestroyed(ServletContextEvent event) {
CachedIntrospectionResults.clearClassLoader(Thread.currentThread().getContextClassLoader().getParent());
Introspector.flushCaches();
}
}
Note that it was required to add the parent to the accepted class loader list rather than the current class loader. Not sure if this is specific to grails or not but this fixed the issue. I'm not sure if there may be a side effect to this fix.
In summary we went from 10 requests/sec in the original setup to 120 requests/sec after using direct reflection and fixing the CachedIntrospectionResults cache.
However the real eye opened was that if we use a 1-1 marshaller per domain class we were seeing another x2 improvement in performance over the generic marshaller where we test objects for whether they're instances of class etc. We're saving a lot of code with the generic marshaller but there's a lot more work to do to get comparable performance to writing a 1-1 marshaller.
Hopefully this will be useful to someone else who runs into this ...

Grails JSON Marshalling Works after first running compile

I'm experiencing a discrepancy between the first compilation of a Grails app and the compilation that happens when a file changes while the app is running.
Background:
My app creates some spring beans from Spring LDAP (docs) using conf/spring/resources.groovy.
I have an LdapUser.groovy class in src/groovy (I'm using it similarly to a domain class, except it isn't in grails-app/domain as it doesn't map to a database table).
In BootStrap.groovy I register a JSON marshaller for LdapUser (using JSON.registerObjectMarshaller).
I have a controller with an index method that responds a list of LdapUser objects. This renders correctly in JSON (according to the marshaller).
With that background, here are the pieces of the problem:
When the show method, which responds a single LdapUser, gets called, I get an exception that LdapUser cannot be converted to grails.converters.JSON. (fair enough)
But, if I save the LdapUser.groovy file, thus invoking a recompile on the file while the app is running, the JSON marshaller suddenly works fine.
Before saving the LdapUser.groovy, my controller has a to an LdapUserRepo (a class instantiated via an #EnableLdapRepositories annotation on the controller), but this reference becomes null after I save LdapUser.groovy. I'm not sure how this relates to the problem, as I was also able to reproduce the problem in a controller lacking an injected LdapUserRepo (but with the annotated controllers still in the app).
I also at one point was setting an asType method on the LdapUser class, which was called as expected before the save-invoked recompile. After the recompile, however, my asType method was no longer called and the JSON marshaller was taking over. ( I was doing exception-worthy things in the asType that were throwing before recompile and not throwing after... )
My understanding of the problem is therefore:
Somehow the asType method of the LdapUser.groovy class is not being automatically generated on first compile when running the app, but is being generated on subsequent compiles.
The LdapUser class is tied to the LdapUserRepo in more ways than merely being a type the Repo uses, and the recompile is not reflecting that connection correctly.
Methods rendering lists of objects are somehow unaffected by the asType method. This leads me to believe that the JSON marshaller gets called directly on list elements (instead of via asType) when the list asType has been called (whether or not the "as" operation is implicit...).
My question then is:
what is the Grails compiler doing differently on run-app vs on compile while app is running that could be causing this behavior?
how can I restructure things to ensure it works properly out of the box?
If I need to RTFM, what would be the FM section? (My google-fu is sadly quite weak).
Note: this question is vaguely similar, but doesn't have any meaningfulness to the answer:
Grails: Defining a JSON custom marshaller as static method in domain

WINRT XAML System.Void Compile Error in

after a little code change inside a Store App I ran into a compilation error:
-> System.Void cannot be used from C# -- use typeof(void) to get the void type object.
Problem is: this comes from a generated file: XamlTypeInfo.g.cs.
case "System.Void"
userType= new ... ,typeof(global::System.Void), ...
...
Rolling back the changes did not help, as deleting bin & obj, restarting, etc.
Is the actual System.Void case entry maybe an indicator that something within a XAML file could not be recognized by the code generator? Is there an System.Void entry in a working XamlTypeInfo.g.cs?
--- ADDITION ---
I can now produce the compile error when changing specific lines. I have a custom control deriving from ItemsControl. I define a regular DP which works fine. I am also providing AttachedProperties for Template, TemplateSelector and Style. Think of a Textbox that gets an Label via an AttachedProperty and its not just a string but like HeaderedControls you can define a Template etc. for the Lable.
The Problem is related to the Get/Set Methods for the AttachedProp. When I either change the Getter return type to DataTemplate or I comment out the Setter fully then the compile error comes:
public static DataTemplate GetLabelTEmplate(UIElement element)
{
return (DataTemplate)element.GetValue(LabelTemplateProperty;
}
public static void SetLabelTemplate(UIElement element, object value)
{
element.SetValue(LabelTemplateProperty, value);
}
Any ideas would be highly appreciated.
Best regards
Gope
After filing a bug complaint with microsoft they pointed me to the problem: The setter's value cannot be of type object. This information is needed for the XamlTypeInfo generation so when I changed object to DataTemplate it compiled.
Although I haven't tried it yet, I believe object is fine for plain WPF, but for Win 8 Store apps this does result in an compilation Error. Funny stuff... :)

How did I solve the Json serializing circular reference error?

There is post here that asks how to solve the circular reference error when returning a serialized object via EF4 CTP5. I ran into this same problem with a WCF web forms project a while back.
I was able to "solve" this problem in my WCF/web forms project and in my MVC3 project. I don't think it matters what type of project as this appears to be a EF serialization "thing".
I solved the problem by disabling ProxyCreation in my ObjectContext constructor like this:
public class MyObjectContext : DbContext, IDbContext
{
public MyObjectContext(string connectionStringName) : base(connectionStringName)
{
((IObjectContextAdapter)this).ObjectContext.ContextOptions.ProxyCreationEnabled = false;
}
public DbSet<Product> Products {get;set;}
//etc.
}
My question is: Could someone explain why this would seemingly solve the problem?
I think the problem has to do with navigation properties in my POCO's but after that I am stumped. Thanks.
If you turn off proxy creation you also turn off lazy loading. When serialization of entity occures it visits all navigation properties. If lazy loading is enabled it loads all related objects and tries to serialize them as well. Again it visits all their properties including navigation properties pointing back to parent object. At this point you have to say serialization that this property is circular reference or it will serialize the object again and continue in infinite loop.
The trick here could be to annotate your circular navigation property in child entity with the ScriptIgnore attribute.
The circular reference happens because you use eager loading on the object.
You have a couple of methods:
Turn off eager loading when your loading your Query (linq or lambda)
DbContext.Configuration.ProxyCreationEnabled = false;
Remove the virtual keyword from the Domainmodel
Include them while loading the objects
Detach the objects (= no eager loading functionality & no proxy)
Repository.Detach(entityObject)
DbContext.Entry(entityObject).EntityState = EntityState.Detached
Clone the properties
You could use something like AutoMapper to clone the object, don't use the ICloneable interface, because it also clones the ProxyProperties in the object, so that won't work.
In case you are building an API, try using a separte project with a different configuration (that doesn't return proxies)
PS. Proxies is the object that's created by EF when you load it from the Entity Framework. In short: It means that it holds the original values and updated values so they can be updated later. It handles other things to ;-)
Quick note: if you still face the exception remember getting rid of
.Include("NestedObject")
This way parent-child relationship will be gone as well as the exception