I'm currently struggling handling serialization exceptions properly in a kafka stream application. Using the latest version. Exception handling for deserialization and production exceptions are fine. There I'm using
props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG, MyCustomDeserializationExceptionHandler.class);
props.put(StreamsConfig.DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG, MyCustomProductionExceptionHandler.class);
I'm using AVRO as output format and having define such a serde:
private SpecificAvroSerde<MyClass> avroSerde(Properties envProps) {
SpecificAvroSerde<MyClass> avroSerde = new SpecificAvroSerde<>();
final HashMap<String, String> serdeConfig = new HashMap<>();
serdeConfig.put(SCHEMA_REGISTRY_URL_CONFIG, envProps.getProperty("schema.registry.url"));
avroSerde.configure(serdeConfig, false);
return avroSerde;
}
If for example a required value is not set we obviously get an exception. But as we are dynamically mapping input values it's not really in our hand if we get correct values. An exception looks like
Error encountered sending record to topic NAME for task 0_0 due to:
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
org.apache.kafka.streams.errors.StreamsException: Error encountered sending record to topic NAME for task 0_0 due to:
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:167)
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:129)
at org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:91)
Finally the question/problem. How can I handle such serialization exceptions properly? I would like to log the error and not having the stream failing.
Thanks in advance!
Related
I keep getting following exception while reading data from cache.
org.springframework.data.redis.serializer.SerializationException: Could not read JSON: Cannot construct instance of `java.time.Instant` (no Creators, like default construct, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
It started as soon as I introduced new variable of type java.time.Instant
You can use JsonSerializer and JsonDeserializer to serialize Instant object either as milliseconds or custom text format.
For example implementation follow the answer section of How to set format of string for java.time.Instant using objectMapper?
in c# I validate a json with a json schema, using the library Newtonsoft.Json.
JsonSchema schema = JsonSchema.Parse(schemaJson);
JObject Json= JObject.Parse(json);
bool valid = Json.IsValid(schema);
It works fine, but the problem is that I don't understand where the errors occur in a simple way. My 'json' is only in one Line and I have a lot of property so it isn't immediate to find the position of the proprierty where the error occurs.
For example, I got the error:
'Invalid type. Expected String but got Null. Line 1, position 2221'
Is there a way to find the name of the property of my json where the error occurs using the library Newtonsoft.Json?
Thanks a lot
So, I am battling and trying to figure this out.
I have Serilog and using EventHub to log errors.
It took a while to find but I needed it to be serialized into JSON so I used this:
logger = new LoggerConfiguration().WriteTo.Sink(new AzureEventHubSink(eventHubClient, new JsonFormatter()))
.CreateLogger();
Great. Now, when I write the exception:
logger.Error(ex, "An Error Occurred");
It writes it BUT the exception is written in 1 field (big long strong).
Is there a way to tell SeriLog to write each property of the exception in its own field (think of it as a SQL table with fields)?
How about changing the outputTemplate but still using JsonFormatter, as there is no overload to accept the output template?
I am using Stream Analytics to do some querying and it makes it better (MUCH better) to have each exception property as its own field column rather than just 1 field with the entire JSON string in there, and I need to do cross joins on another data source.
Thank you.
Seems the only way is to inherit from JsonFormatter and then override the WriteException method and write the Property in question....
WriteJsonProperty("ClassName", exception.GetType(), ref delim, output);
WriteJsonProperty("Message", exception.Message, ref delim, output);
I'm working with data via an IBM MQ call where a 0 is returned as +0.0 when this is serialized by ColdFusion 10 (10,0,11,285437) it results in invalid JSON and cannot be deserialized.
stPolicy = { "prem": "+0.0" };
serializedData = serializeJSON(stPolicy);
writeDump(isJSON(serializedData));
writeDump(deserializeJSON(serializedData));
This outputs NO for isJSON and an error when trying to deserialize JSON parsing failure at character 9:'+' in {"prem":+0.0}.
I'm able to work around this by replacing all +0 to 0, but I'm guessing this is a bug in ColdFusion. Has anyone else had this issue and implemented a better fix?
Bug report filed: Bug #3632972
Adobe has the bug listed as fixed in their bug database, but it has not yet been released. Here's the fix I came up with. Simply replace the +0 with 0
private string function serializeJSONFixCF10(required string serializedData) {
return replace(arguments.serializedData,'+0','0','all');
}
I'm facing following error while trying unmarshal java.util.Date from JSON in grails controller.
org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object '2011-10-07 10:24:40' with class 'java.lang.String' to class 'java.util.Date'**
Also, I've tried the following method but still no luck, actually I've doubt wether I've implemented following in right way or not because when i put println statements in following method:
public CustomDateBinder(List formats)
Nothing prints on console.
Grails Date unmarshalling
According to the description of your error message you are trying to convert a String to Date, if you want to do it manually you can use the following method in your Controller (since Grails 2)
def val = params.date('myDate', 'dd-MM-yyyy') //Obviously you need to change the format
Check the following post for more info: http://mrhaki.blogspot.com/2012/01/grails-goodness-date-request-parameter.html