I'm using xStream to some JSON. I've used xstream quite extensively over the years. However this issue has me stumped.
I'm getting the following ConversionException...
com.thoughtworks.xstream.converters.ConversionException: For input string: ".232017E.232017E44"
---- Debugging information ----
message : For input string: ".232017E.232017E44"
cause-exception : java.lang.NumberFormatException
cause-message : For input string: ".232017E.232017E44"
class : java.sql.Timestamp
required-type : java.sql.Timestamp
converter-type : com.etepstudios.xstream.XStreamTimestampConverter
line number : -1
class[1] : com.pbp.bookacall.dataobjects.AppleReceipt
converter-type[1] : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
class[2] : com.pbp.bookacall.dataobjects.AppleReceiptCollection
version : 1.4.10
-------------------------------
at com.etepstudios.xstream.XStreamTimestampConverter.unmarshal(XStreamTimestampConverter.java:87)
In my XStreamTimestampConverter class I print out the value that is attempting to be converted.. Which turns out to be the following...
XStreamTimestampConverter value = 2017-08-05 23:44:23.GMT
Here is the unmarshal function in my converter...
public Object unmarshal(HierarchicalStreamReader reader, UnmarshallingContext context)
{
Timestamp theTimestamp;
Date theDate;
String value = reader.getValue ();
try
{
SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.Z");
theDate = formatter.parse(value);
theTimestamp = new Timestamp (theDate.getTime());
}
catch (Exception e)
{
System.out.println ("XStreamTimestampConverter value = " + value);
throw new ConversionException(e.getMessage(), e);
}
return theTimestamp;
}
Any idea where this odd string is coming from? It does not exist anywhere in my JSON. Does xstream have some odd .[num]E.[num]E[num] notation for something? These numbers can change as I run this each time. Also I get an For input string: "" on occasion too. Yet the value is similar to the what is above. It's like it's randomly getting odd values for somewhere.
The data source is from Apple's In-App Purchase /VerifyReceipt web call. The system works just fine some times but then others it does not. It's also important to note that in this very case it parsed 100s of other Date/Timestamp strings using this converter. It just get's confused. Perhaps due to the size of the data?
So I figured out what was going on here. The unmarshal function above is not exactly as I have it in code...
The SimpleDateFormat formatter is actually set in the class rather than in the unmarshal method. Therefore if Xstream holds on to an instance of my converter and the unmarshal is called across multiple threads then it is possible that the formatter can get confused since it is the same object.
That's my only guess at this point as moving the formatter initialization into the method solved the issue. I would say SimpleDateFormatter is not thread safe?
It was the just the sheer about of data and the number of times it was concurrently being called that exposed this issue. Just a tip for anyone else in case this happens to them.
Related
Is it possible to get the whole object from debugger as Json?
There is an option View text but can I somehow View JSON?
EDIT: as noted in the comments, this is not perfect, as for some variables you will get a "stackoverflow" response
As suggested by #Mr Han's answer, here's how you can do this:
Add a new way to view objects in Intellij debugger as json by
Going to File | Settings | Build, Execution, Deployment | Debugger | Data Views | Java Type Renderers
Click + to add new renderer
Call it JSON renderer
Supply java.lang.Object for Apply renderer to objects of type
Choose Use following expression: and supply an expression like so:
if (null == this || this instanceof String)
return this;
new com.google.gson.GsonBuilder().setPrettyPrinting().create().toJson(this);
Click OK
Now when you choose Copy Value on a variable, it will copy as json.
Alternatively, as seen here, you can use the following piece of code in your debug watcher:
new ObjectMapper()
.setSerializationInclusion(JsonInclude.Include.NON_NULL)
.writerWithDefaultPrettyPrinter()
.writeValueAsString( myObject )
You can try this code fragment into the Evaluate Expression(Alt + F8) on IntelliJ :
new com.fasterxml.jackson.databind.ObjectMapper() .registerModule(new com.fasterxml.jackson.datatype.jsr310.JavaTimeModule()) .disable(com.fasterxml.jackson.databind.SerializationFeature.WRITE_DATES_AS_TIMESTAMPS) .writerWithDefaultPrettyPrinter() .writeValueAsString( myObject );
image IntelliJ
You could use the Show as ... plugin for IntelliJ.
A small plugin to display formatted data out of the debugger and console.
Uses IntelliJ's build-in formatting capabilities.
No more need to copy values from debugger or console to a file to format them there. Following formats are supported: JSON, SQL, XML, Base64 encoded JSON, Base64 encoded text
If you have gson dependency in your project you can create a watch variable
new GsonBuilder().setPrettyPrinting().create().gson.toJson(myObject)
where myObject is your object.
Just follow it : File | Settings | Build, Execution, Deployment | Debugger | Data Views | Java Type Renderers, click + to add new render ,
copy is OK :) u can choose another jar to format it
And now , Apply, join it ~
Follow the instructions of #BradParks, and use the following expression.
For me it did not work without fully-qualified class names. I also added some modifications to the ObjectMapper. For some reason which I don't understand, even if I have Apply renderers to object of type set to java.lang.Object, I needed to typecast this as (Object)this when used as a parameter of the writeValueAsString() method.
if (this == null
|| this instanceof CharSequence
|| this instanceof Number
|| this instanceof Character
|| this instanceof Boolean
|| this instanceof Enum) {
// Here you may add more sophisticated test which types you want to exclude from the JSON conversion.
return this;
}
new com.fasterxml.jackson.databind.ObjectMapper()
.registerModule(new com.fasterxml.jackson.datatype.jsr310.JavaTimeModule())
.disable(com.fasterxml.jackson.databind.SerializationFeature.WRITE_DATES_AS_TIMESTAMPS)
.setVisibility(
com.fasterxml.jackson.annotation.PropertyAccessor.FIELD,
JsonAutoDetect.Visibility.ANY)
.setSerializationInclusion(com.fasterxml.jackson.annotation.JsonInclude.Include.NON_NULL)
.writerWithDefaultPrettyPrinter()
.writeValueAsString((Object)this);
In case someone is having hard time to make the renderers work for more complex objects - below you can find a combined JSON renderer from:
https://stackoverflow.com/a/60189764/3837638
https://stackoverflow.com/a/69802068/3837638
https://stackoverflow.com/a/63060407/3837638
The following renderer helped me to identify multiple fields with the same name in the class hierarchy, so I could change that.
Initially I was having IllegalArgumentException for serialization of a nested object that I wasn't able to analyse.
If there is an issue during serialization, with this renderer you can find the stack trace from the exception that you need to fix in the console.
Good luck!
if (null == this)
return "null";
if (this instanceof CharSequence
|| this instanceof Number
|| this instanceof Character
|| this instanceof Boolean
|| this instanceof Enum) {
// Here you may add more sophisticated test which types you want to exclude from the JSON conversion.
return this;
}
try {
String json = new GsonBuilder().setPrettyPrinting().create().toJson(this);
return json;
} catch (Exception e) {
e.printStackTrace();
}
Use Intellij plugin Debug Variable Extractor
More information -
https://plugins.jetbrains.com/plugin/16362-debug-variable-extractor
Firstly, please forgive any rookie mistakes here - I'm not a regular poster I'm afraid.
Now on to the nitty gritty...
I am trying to use ServiceStack.Text to serialize objects to CSV. If I keep it simple, everything works as expected when serializing objects of a known type.
However I want to serialize many objects and I don't know the type at runtime so I am writing a reusable component where all data is treated as a System.Object. We already do this same routine for Json serialization without problems. But CsvSerializer appears to handle objects differently during serialization.
Sample code
public void TestIEnumerableObjectSerialization()
{
var data = GenerateSampleData();
JsConfig<DateTime>.SerializeFn =
time => new DateTime(time.Ticks, DateTimeKind.Utc).ToString("yyyy-MM-dd HH:mm:ss");
var csv = CsvSerializer.SerializeToCsv(data);
Console.WriteLine(csv);
Assert.Equal("DateTime\r\n"
+ "2017-06-14 00:00:00\r\n"
+ "2017-01-31 01:23:45\r\n",
csv);
}
object[] GenerateSampleData()
{
return new object[] {
new POCO
{
DateTime = new DateTime(2017,6,14)
},
new POCO
{
DateTime = new DateTime(2017,1,31, 01, 23, 45)
}
};
}
public class POCO
{
public DateTime DateTime { get; set; }
}
The result of this code is that the custom serialization function is not invoked, and the DateTime is written out using the standard ToString() method.
The cause?
The CsvWriter.Write method is inspecting the type of the records and if the type is Object it is treated as a Dictionary<string, object> and CsvDictionaryWriter generates the output.
In turn, CsvDictionaryWriter uses the ToCsvField() extension method to write each property a record.
The problem is that ToCsvField() converts the value of each property to a string using ToString() meaning no custom serialization is performed.
JsonSerializer uses TypeSerializer.SerializeToString(text) to serialize the properties of an Object using any configured custom serialization functions; but this doesn't happen with CsvSerializer.
A possible solution?
Without complicating CsvSerializer, the ToCsvField() extension method could be updated to use TypeSerializer to handle the serialization to a string. Here is what I've been testing with so far:
public static object ToCsvField(this object text)
{
var textSerialized = TypeSerializer.SerializeToString(text).StripQuotes();
return textSerialized == null || !CsvWriter.HasAnyEscapeChars(textSerialized)
? textSerialized
: string.Concat
(
CsvConfig.ItemDelimiterString,
textSerialized.Replace(CsvConfig.ItemDelimiterString, CsvConfig.EscapedItemDelimiterString),
CsvConfig.ItemDelimiterString
);
}
So far I haven't come across an issue with this change, although someone may prefer not to allocate a new intermediate variable before the return statement.
Hopefully that is enough information, so on to my questions...
Has anyone else experienced this issue?
Am I doing something wrong and should I be serializing Objects a different way?
If this is a suitable fix/implementation of TypeSerializer, what are the chances of this being addressed in an update to ServiceStack.Text? I would raise an issue on GitHub but the ServiceStack.Text repo doesn't let me raise issues.
Thanks in advance.
We recently upgraded to Json.NET 10.0r2 from 6.0.1 and since upgrading I noticed that one of our unit tests throws a Stack Overflow Exception when trying to deserialize invalid Json. The purpose of the test was to ensure handling of invalid Json. This same test used to throw a JsonSerializationException, but now is bringing down nUnit with the StackOverflow.
I've replicated it in Json.NET's own unit testing project with this test:
[Test]
public void FailOnInvalidJSON( )
{
string json = #"{'Row' : ";
Assert.Throws<JsonSerializationException>(()=>JsonConvert.DeserializeXmlNode(json, "ROOT"));
}
Any ideas on work a work-around?
Thanks!
Update
And promptly fixed in change set 822c3f0. Should be in the next release after 10.0.2.
Original Answer
It looks like a change to JsonTextReader in version 8.0.1 may have uncovered a bug in XmlNodeConverter.
In 7.0.1, when the unexpected end of file is reached, JsonReader.TokenType becomes JsonToken.None after the next attempt to Read(), which causes DeserializeNode() to throw an Unexpected JsonToken when deserializing node: None exception. But in 8.0.1 and later the TokenType appears to stay stuck at the type of the last encountered token, namely JsonToken.PropertyName, which causes the infinite recursion.
The correct fix would be, in XmlNodeConverter.DeserializeNode() around line 2171, to check the return from reader.Read():
case JsonToken.PropertyName:
if (currentNode.NodeType == XmlNodeType.Document && document.DocumentElement != null)
{
throw JsonSerializationException.Create(reader, "JSON root object has multiple properties. The root object must have a single property in order to create a valid XML document. Consider specifying a DeserializeRootElementName.");
}
string propertyName = reader.Value.ToString();
// Need to check the return from reader.Read() here:
if (!reader.Read())
{
throw JsonSerializationException.Create(reader, "Unexpected end of file when deserializing property: " + propertyName );
}
... And it appears there are a few more places in XmlNodeConverter.cs where the return from reader.Read() needs to be checked, for instance in ReadAttributeElements(JsonReader reader, XmlNamespaceManager manager) around line 1942.
You could report an issue if you want.
In the meantime, your options for a workaround would be:
Corrupt the JSON in a different manner, for instance like so:
string json = #"{'Row' : }";
And check for the more general exception JsonException.
Pre-parse the JSON into a JToken:
Assert.Throws<JsonException>(()=>JsonConvert.DeserializeXmlNode(JToken.Parse(json).ToString(), "ROOT"));
I'm trying to use Redis to store some cache data for my entity, which has different types of fields inside, for example,
public class Job {
private String id;
private Date createTime; //Long
private String submitterName;
private JobDefinition jobDef; //Another class
}
There are more fields and due to the fact that several fields are updated more frequently than others, I decided to save this job as a Hashmap in Redis with each field as a key. Here the nested object like jobDef is not important so I used Jackson2JsonRedisSerializer as hashValueSerializer for RedisTemplate and the jobDef obj will just be serialized as a long JSON string, which is totally fine in my case.
But I don't know how can I effectively deserialize the whole job object back from Redis. The type I set to deserializer is like Jackson2JsonRedisSerializer(Map.class) but it complains when deserializing String keys and values.
So is this an invalid usage with RedisTemplate or how should I configure my serializer for it?
EDIT:
Adding more code details,
#Autowired
private StringRedisTemplate redisTemplate; //Here I'm using a String template as I need to use the same redisTemplate for some key-value/list operations too
Map jobHash= new ObjectMapper().convertValue(job, Map.class);
redisTemplate.setHashValueSerializer(new Jackson2JsonRedisSerializer(Map.class));
redisTemplate.opsForHash().putAll("job:"+job.getId(), jobHash); //After this the job hash shows up in Redis as I expected, while the jobDef member is serialized and saved as a JSON string
Map jobMap = redisTemplate.opsForHash().entries("job:" + job.getId()); //But this won't work as it'll throw exception complaining cannot deserialize a String value to Map. But when I set Jackson2JsonRedisSerializer(String.class) it throws exception that cannot resolve the byte code
2nd EDIT:
If using JdkSerializationRedisSerializer as HashValueSerializer in RedisTemplate then the deserialization works fine, however the downside for using this one is the value stored in Redis is not the same human readable string value as when using Jackson2JsonRedisSerializer.
The Jackson2JsonRedisSerializer does not include mapping information into the actual hash structure.
The resulting Redis HASH results in something like:
127.0.0.1:6379> hgetall job:1
1) "id"
2) "\"1\""
3) "createTime"
4) "1455778716799"
5) "submitterName"
6) "\"Jon Snow\""
7) "jobDef"
8) "{\"def\":\"nightwatch\"}"
The ObjectMapper produces a LinkedHashMap for the JobDefinition entry which fails to deserialize as the type is unknown.
Using the GenericJackson2JsonRedisSerializer includes type information so the resulting Redis HASH looks like this:
127.0.0.1:6379> hgetall job:1
1) "id"
2) "\"1\""
...
7) "jobDef"
8) "{\"#class\":\"java.util.LinkedHashMap\",\"def\":\"nightwatch\"}"
This allows to deserialize values correctly.
Another approach would be to NOT use a specific HashValueSerializer but instead use a DecoratingStringHashMapper along with the StringRedisTemplate.
DecoratingStringHashMapper mapper = new DecoratingStringHashMapper<Job>(
new JacksonHashMapper<Job>(Job.class));
template.opsForHash().putAll("job:" + job.id, mapper.toHash(job));
Map jobMap = template.opsForHash().entries("job:" + job.id);
The DecoratingStringHashMapper will produce a Redis Hash as follows:
127.0.0.1:6379> hgetall job:1
1) "id"
2) "1"
3) "createTime"
4) "1455780810643"
5) "submitterName"
6) "Jon Snow"
7) "jobDef"
8) "{def=nightwatch}"
Unfortunately there is no Jackson2HashMapper. Please vote for DATAREDIS-423 and help us prioritize.
I'm stumped as to what may be the problem. I'm using GSON (2.2.4) to serialize/deserialize simple json objects and arrays, which I display in a JTable. Everything works fine but when I feed a json object like this:
{"1":{"336":"#1700EB","17":"#EB0000","10":"#EB0000","26":"#1700EB","3":"#1700EB","1":"#EB0000"}}
it throws this error:
com.google.gson.JsonSyntaxException: java.io.EOFException: End of input at line 1 column 71
at com.google.gson.Gson.fromJson(Gson.java:813)
at com.google.gson.Gson.fromJson(Gson.java:768)
at com.google.gson.Gson.fromJson(Gson.java:717)
When I remove the pound signs, it functions normally.
Here's the code that does the conversion:
Map<String, String> conv = null;
Type type = new TypeToken<Map<String, String>>(){}.getType();
ListMultimap<String, Object> returnMap = ArrayListMultimap.create();
try {
conv = g.fromJson(parse, type);
for(String key : conv.keySet()) {
returnMap.put("Key", key);
returnMap.put("Value", conv.get(key));
}
} catch (JsonSyntaxException jse) {
type = new TypeToken<Map<String, Object>>(){}.getType();
conv = g.fromJson(parse, type);
for(Object key : conv.keySet()) {
returnMap.put("Key", key);
returnMap.put("Value", conv.get(key));
}
}
Please note that I am working on a "legacy" application and I have little control over the values that come to the part of the code I'm working on; which is why I have that odd try-catch block.
Most users of this application are not savvy enough to treat their strings/jsons with the express purpose of avoiding tripping exceptions like the one outlined here (e.g. not including the # sign when passing it through the application; but adding it back when they need it), so I'd really like to fix it within the code.
Thanks!
Edit: I forgot to add an important detail. What I'm doing with the code is display data in tabular form. When a user selects a cell, it handles according to context. In the context of a cell containing a json object or array, it uses the code above to extract the values from the json and passes it as the new table data for the table's model. So, the sample json object should ideally come out like this (imagine table form)
336 | #1700EB
17 | #EB0000
10 | #EB0000
26 | #1700EB
3 | #1700EB
1 | #EB0000
...but it doesn't get there. The previous table that had the cell with the json object looked like this:
1 | {336=#1700EB, 17=#EB0000, 10=#EB0000, 26=#1700EB, 1=#EB0000}
Does this form have anything to do with the error? I understand that the json object form should be like this (at least the ones I work with): {"336":"#1700EB"...}. That's probably my only hunch as to what may be wrong.
Thanks for trying guys. I was able to fix the problem though, honestly, I still don't know the underlying cause. Anyway, I refactored the code to not do the initial attempt to map using Type. Apparently, it caused the problem further downstream.
Here's the amended code for the curious:
Map<String, Object> conv = null;
Type type = new TypeToken<Map<String, Object>>(){}.getType();
ListMultimap<String, Object> returnMap = ArrayListMultimap.create();
conv = g.fromJson(parse, type);
for(Object key : conv.keySet()) {
returnMap.put("Key", key);
Object check = conv.get(key.toString());
if ((check.toString().startsWith("{") && check.toString().endsWith("}")) ||
(check.toString().startsWith("[") && check.toString().endsWith("]")))
check = g.toJson(check);
returnMap.put("Value", check);
}
return returnMap;