Serializing Manatee.Json in .NET Core 3 - json

Background
I want to provide some JsonSchema from my .NET Core 3 application, as well as other objects serialized into JSON. Since Manatee.Json is frequently updated and provides good support for JsonSchema, they are my preferred choice. At the same time, .NET Core 3 provides excellent support for returning objects with magic that transform them into JSON documents.
Example:
public ActionResult<MyFancyClass> MyFancyAction()
{
return new MyFancyClass {
Property1 = "property 1 content",
Property2 = "property 2 content",
};
}
Output:
{
"Property1": "property 1 content",
"Property2": "property 2 content"
}
Problem
.NET Core 3 has internal support for JSON with its System.Text.Json which is used in the previous example. If I try to serialize Manatee.Json.Schema.JsonSchema, its internal structure are serialized, not the json schema itself.
Example:
public ActionResult<MyFancyClass2> MyFancyAction2()
{
return new MyFancyClass2 {
Property1 = "property 1 content",
Property1Schema = new JsonSchema()
.Type(JsonSchemaType.String)
};
}
Output:
{
"Property1": "property 1 content",
"Property1Schema": [{
"name":"type",
"supportedVersions":15,
"validationSequence":1,
"vocabulary": {
"id":"https://json-schema.org/draft/2019-09/vocab/validation",
"metaSchemaId":"https://json-schema.org/draft/2019-09/meta/validation"
}
}]
}
I expect this:
{
"Property1": "property 1 content",
"Property1Schema": {
"type": "string",
}
}
Manatee.Json.JsonValue also have a conflicting inner structure, where System.Text.Json.JsonSerializer fails to access internal get methods correctly and I get for instance this exception message:
Cannot access value of type Object as type Boolean.
Discovery
Manatee.Json.Schema.JsonSchema has a .ToJson() method that can be used to get the correct json schema as a JsonValue, but then I get the problem I just mentioned with serializing Manatee.Json.JsonValue.
Question
Does anyone know a way to enable System.Text.Json to serialize Manatee.Json structures?
Sidemark
Another way forward is to replace System.Text.Json altogether (take a look at this question).

.NET Core 3 json serialization comes with a lot of configuration options. One of them is to add converters that specify how different types should be serialized. One way forward is to create a JsonConverter for JsonSchema and another for JsonValue.
For JsonSchema we can implement a JsonSchemaConverter that when serializing/writing, extracts the json schema as a JsonValue and ask the serializer to serialize that JsonValue instead. Like this:
public class JsonSchemaConverter : JsonConverter<JsonSchema>
{
public JsonSchemaConverter()
{
_manateeSerializer = new ManateeSerializer();
}
private ManateeSerializer _manateeSerializer { get; set; }
public override JsonSchema Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
var jsonText = reader.GetString();
var jsonValue = JsonValue.Parse(jsonText);
return _manateeSerializer.Deserialize<JsonSchema>(jsonValue);
}
public override void Write(Utf8JsonWriter writer, JsonSchema value, JsonSerializerOptions options)
{
var schemaAsJson = value.ToJson(_manateeSerializer);
try
{
System.Text.Json.JsonSerializer.Serialize<JsonValue>(writer, schemaAsJson, options);
}
catch (Exception e)
{
Log.Information($"Failed to serialize JsonSchema ({e.Message});");
writer.WriteNullValue();
}
}
}
For JsonValue we can change it into something System.Text.Json understands, since it is json after all. One unfortunate approach is to serialize the JsonValue to a string, parsing it with for instance JsonDocument.Parse(string) and serialize its RootElement property. It feels so unnecessary to go via JsonDocument, so if anyone finds a better solution that would be great!
A possible implementation can look like this:
public class JsonValueConverter : JsonConverter<JsonValue>
{
public override JsonValue Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
var json = reader.GetString();
return JsonValue.Parse(json);
}
public override void Write(Utf8JsonWriter writer, JsonValue value, JsonSerializerOptions options)
{
string content = value.ToString();
try
{
var jsonDocument = JsonDocument.Parse(content);
JsonSerializer.Serialize<JsonElement>(writer, jsonDocument.RootElement, options);
}
catch (Exception e)
{
Log.Warning($"JsonDocument.Parse(JsonValue) failed in JsonValueConverter.Write(,,).\n{e.Message}");
writer.WriteNullValue();
}
}
}
They must be registered at Startup.cs like this:
public void ConfigureServices(IServiceCollection services)
{
services.AddControllers()
.AddJsonOptions(options =>
{
options.JsonSerializerOptions.Converters.Add(new JsonValueConverter());
options.JsonSerializerOptions.Converters.Add(new JsonSchemaConverter());
});
}

Related

Distinguish between NULL and not present using JSON Merge Patch with NetCore WebApi and System.Text.Json

I want to support partial updates with JSON Merge Patch. The domain model is based on the always valid concept and has no public setters. Therefore I can't just apply the changes to the class types. I need to translate the changes to the specific commands.
Since classes have nullable properties I need to be able to distinguish between properties set to null and not provided.
I'm aware of JSON Patch. I could use patch.JsonPatchDocument.Operations to go through the list of changes. JSON Patch is just verbose and more difficult for the client. JSON Patch requires to use Newtonsoft.Json (Microsoft states an option to change Startup.ConfigureServices to only use Newtonsoft.Json for JSON Patch (https://learn.microsoft.com/en-us/aspnet/core/web-api/jsonpatch?view=aspnetcore-6.0).
Newtonsoft supports IsSpecified-Properties that can be used as a solution for JSON Merge Patch in the DTO classes (How to make Json.NET set IsSpecified properties for properties with complex values?). This would solve the problem, but again requires Newtonsoft. System.Text.Json does not support this feature. There is an open issue for 2 years (https://github.com/dotnet/runtime/issues/40395), but nothing to expect.
There is a post that describes a solution with a custom JsonConverter for Web API (https://github.com/dotnet/runtime/issues/40395). Would this solution still be usable for NetCore?
I was wondering if there is an option to access the raw json or a json object inside the controller method after the DTO object was filled. Then I could manually check if a property was set. Web Api closes the stream, so I can't access the body anymore. It seems there are ways to change that behavior (https://gunnarpeipman.com/aspnet-core-request-body/#comments). It seems quite complicated and feels like a gun that is too big. I also don't understand what changes were made for NetCore 6.
I'm surpised that such a basic problem needs one to jump through so many loops. Is there an easy way to accomplish my goal with System.Text.Json and NetCore 6? Are there other options? Would using Newtonsoft have any other bad side effects?
With the helpful comments of jhmckimm I found
Custom JSON serializer for optional property with System.Text.Json. DBC shows a fantastic solution using Text.Json and Optional<T>. This should be in the Microsoft docs!
In Startup I added:
services.AddControllers()
.AddJsonOptions(o => o.JsonSerializerOptions.DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingDefault)
.AddJsonOptions(o => o.JsonSerializerOptions.Converters.Add(new OptionalConverter()));
Since we use <Nullable>enable</Nullable> and <WarningsAsErrors>nullable</WarningsAsErrors> I adapted the code for nullables.
public readonly struct Optional<T>
{
public Optional(T? value)
{
this.HasValue = true;
this.Value = value;
}
public bool HasValue { get; }
public T? Value { get; }
public static implicit operator Optional<T>(T value) => new Optional<T>(value);
public override string ToString() => this.HasValue ? (this.Value?.ToString() ?? "null") : "unspecified";
}
public class OptionalConverter : JsonConverterFactory
{
public override bool CanConvert(Type typeToConvert)
{
if (!typeToConvert.IsGenericType) { return false; }
if (typeToConvert.GetGenericTypeDefinition() != typeof(Optional<>)) { return false; }
return true;
}
public override JsonConverter CreateConverter(Type typeToConvert, JsonSerializerOptions options)
{
Type valueType = typeToConvert.GetGenericArguments()[0];
return (JsonConverter)Activator.CreateInstance(
type: typeof(OptionalConverterInner<>).MakeGenericType(new Type[] { valueType }),
bindingAttr: BindingFlags.Instance | BindingFlags.Public,
binder: null,
args: null,
culture: null
)!;
}
private class OptionalConverterInner<T> : JsonConverter<Optional<T>>
{
public override Optional<T> Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
T? value = JsonSerializer.Deserialize<T>(ref reader, options);
return new Optional<T>(value);
}
public override void Write(Utf8JsonWriter writer, Optional<T> value, JsonSerializerOptions options) =>
JsonSerializer.Serialize(writer, value.Value, options);
}
}
My test DTO looks like this:
public class PatchGroupDTO
{
public Optional<Guid?> SalesGroupId { get; init; }
public Optional<Guid?> AccountId { get; init; }
public Optional<string?> Name { get; init; }
public Optional<DateTime?> Start { get; init; }
public Optional<DateTime?> End { get; init; }
}
I can now access the fields and check with .HasValue if the value was set. It also works for writing and allows us to stripe fields based on permission.

Deserialization difference in Newtonsoft.Json.6.0.8 vs Newtonsoft.Json.12.0.3

It seems there exists a difference in the way of deserialization process of Newtonsoft.Json.6.0.8 vs Newtonsoft.Json.12.0.3.
The following is model in our C# project :
public class WebServiceConfigModel
{
public string RestoreFile { get; set; }
public string RestoreFileDescription { get; set; }
}
The action method defined in the controller is as follows:
[HttpPost]
public void Restore(WebServiceConfigModel request)
{
}
The input JSON text which was provided to the method is as follows:
{
"RestoreFile": "SampleFile",
"RestoreFileDescription": {
"ID": "DatasetDescription",
"Label": "Description"
}
}
This was deserialized successfully (the request object contains values), even if there exists a deserialization error and we were able to read the RestoreFile property value in the C# while using the Newtonsoft.Json.6.0.8.
After upgrading the version Newtonsoft.Json to 12.0.3, the request object in C# seems to be null and the deserialization error still exists. It works properly if we change the "RestoreFileDescription" property to a string value.
Is there any way to get the deserialized object even if some of the property has a contract mismatch?
These docs may be helpful.
It appears that as of 12.0.1, you can handle Json Deserialization errors in two ways:
JsonSerializerSettings.Error event
The [OnError] attribute
In the first instance, the JsonSerializerSettings have been set so as to handle a non-date string, and this is handled as per their docs:
The event handler has logged these messages and Json.NET has continued on deserializing the JSON because the errors were marked as handled.
List<DateTime> c = JsonConvert.DeserializeObject<List<DateTime>>(#"[
'2009-09-09T00:00:00Z',
'I am not a date and will error!',
[
1
],
'1977-02-20T00:00:00Z',
null,
'2000-12-01T00:00:00Z'
]",
new JsonSerializerSettings
{
Error = delegate(object sender, ErrorEventArgs args)
{
errors.Add(args.ErrorContext.Error.Message);
args.ErrorContext.Handled = true;
},
Converters = { new IsoDateTimeConverter() }
});
The second way is to create a method, and decorate it with the [OnError] attribute as follows:
[OnError]
internal void OnError(StreamingContext context, ErrorContext errorContext)
{
errorContext.Handled = true;
}
PersonError person = new PersonError
{
Name = "George Michael Bluth",
Age = 16,
Roles = null,
Title = "Mister Manager"
};
So when Roles is required, this provides the following result:
string json = JsonConvert.SerializeObject(person, Formatting.Indented);
Console.WriteLine(json);
//{
// "Name": "George Michael Bluth",
// "Age": 16,
// "Title": "Mister Manager"
//}

How to validate JSON using System.Text.Json before deserialization

In .NET Core 5.0, using System.Text.Json.JsonSerializer Deserialize(someJsonFile) i get:
System.Text.Json.JsonException: 'The JSON value could not be converted to System.Guid. Path: $.someGuid | ..
which is expected, since the the someGuid property is of type System.Guid and the value of someGuid in the JSON file/string is:
{
"someGuid": ""
}
which can not be deserialized properly .. (since it's not Guid.Empty)..
To my question.
What's a good and generic implementation to validate the Json before deserializing it (in general)? like TryParse or JsonDocument.Parse? sure, try-catch but that's dirty (imho).
btw: i don't want to use Newtonsoft
thanks for your suggestions (and critics of course).
I created a custom converter using the example is this answer: The JSON value could not be converted to System.Int32
public class StringToGuidConverter : JsonConverter<Guid>
{
public override Guid Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
{
if (reader.TokenType == JsonTokenType.String)
{
ReadOnlySpan<byte> span = reader.ValueSpan;
if (Utf8Parser.TryParse(span, out Guid guid, out int bytesConsumed) && span.Length == bytesConsumed)
{
return guid;
}
if (Guid.TryParse(reader.GetString(), out guid))
{
return guid;
}
}
return Guid.Empty;
}
public override void Write(Utf8JsonWriter writer, Guid value, JsonSerializerOptions options)
{
writer.WriteStringValue(value.ToString());
}
}
In my case my model to deserialize to can't take Nullable Guid so I return an empty GUID and then validate this in my logic.
Because I'm create a web api using .Net standard, I can't register this in the services at the startup class. But you can register the custom converter using the JsonSerializerOptions property when calling the Deserialize method like this:
var options = new JsonSerializerOptions
{
PropertyNameCaseInsensitive = true,
Converters = { new NERDS.API.Helpers.StringToGuidConverter() }
};
StreamReader reader = new StreamReader(HttpContext.Current.Request.InputStream);
string json = reader.ReadToEnd();
return JsonSerializer.Deserialize<T>(json, options);

Webapi with entity framework - use customer JSON converter to serialized DbGeography not working

I am using webapi with DbGeography spatial data and want to serialize to json.
By default, DbGeography serializes to null. So I implemented my own converter for it.
Here is what I have so far, but it doesn't seem to work.
Basically, with the following code, my DbGeographyConverter.WriteJson method is never under debug and the Location property is serialized as null
Customer converter:
public class DbGeographyConverter : JsonConverter
{
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
DbGeography contextObj = value as DbGeography;
writer.WriteStartObject();
writer.WritePropertyName("Lat");
serializer.Serialize(writer, contextObj.Latitude);
writer.WritePropertyName("Long");
serializer.Serialize(writer, contextObj.Longitude);
writer.WriteEndObject();
}
public override bool CanConvert(Type objectType)
{
if (objectType == typeof(DbGeography))
{
return true;
}
return false;
}
public override bool CanRead
{
get
{
return true;
}
}
public override bool CanWrite
{
get
{
return true;
}
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
throw new NotImplementedException();
}
Add convert in Global.ascx.cs
protected void Application_Start()
{
GlobalConfiguration.Configuration.Formatters.JsonFormatter.SerializerSettings.Converters.Add(
new DbGeographyConverter()
}
Finally, apply converter to the data model class property
public DataModelClass1
{
[JsonConverter (typeof(DbGeographyConverter))]
public DbGeography Location { get; set; }
}
First, since you're adding your custom converter to the SerializerSettings.Converters collection, you don't need to decorate your DataModelClass1's Location property with the JsonConverterAttribute- The JsonFormatter will run through the aforementioned collection until it finds the derived JsonConverter you added without the attribute.
Now back to your question- which browser are you testing this in and how? If I were to speculate, I'd say you're using either Chrome or Firefox with GET requests, both of which prioritize application/xml over application/json in the accept header they send to the server. For that reason Web API will see that the browsers prefer XML over JSON and the JsonFormatter will never be touched, let alone your custom JsonConverter.
There are a few workarounds to this. On the browser side the easiest way is to make ajax GET requests with jQuery and specify that you want JSON back. On the server side, you can remove application/xml from the SupportedMediaTypes.
I spent quite awhile on this. You'd only use the write method if you wanted to change the default output format of the Json for DbGeography from this
"geography": {
"coordinateSystemId": 4326,
"wellKnownText": "POINT (77.6599502563474 12.9602302518557)"
}
to something else like "77.22, 12.8" - just a single string.
If you're looking to convert a string like that to DbGeography on reading Json from the request, the code below is what you're after
public class DbGeographyConverter : JsonConverter
{
public override bool CanConvert ( Type objectType )
{
return objectType.IsAssignableFrom( typeof( string ) );
}
public override object ReadJson ( JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer )
{
if ( reader.Value == null ) {
return null;
}
return Parser.ToDbGeography( reader.Value.ToString() );
}
public override void WriteJson ( JsonWriter writer, object value, JsonSerializer serializer )
{
// Base serialization is fine
serializer.Serialize( writer, value );
}
}
This is the code for the Converter if you're passing in a string value - 12,99
Which will be you're lat and lng
This sample application has the answer you're looking for.
https://code.msdn.microsoft.com/windowsazure/HTML-ASPNET-Web-API-Bing-58c97f9f
The Converter code from this can be found here
https://github.com/Azure-Samples/SQLDatabase-Spatial-WebAPI-BingMaps/blob/master/SpatialTypesWithWebAPI/Models/DbGeographyConverter.cs

#JsonFilter throws "JsonMappingException: Can not resolve BeanPropertyFilter"

Is it possible to selectively determine when the #JsonFilter annotation gets used at runtime?
I'm getting JsonMappingException exception (see below) when I don't provide the filter.
Background:
I learned from a recent StackOverflow post that I can use #JsonFilter to dynamically filter the bean properties getting serialized. This works great. After adding #JsonFilter("apiFilter") to my domain class and with the addition of this code in my jax-rs service (using the CXF implementation), I am able to dynamically filter the properties returned by my RESTful API:
// shortened for brevity
FilterProvider filters = new SimpleFilterProvider().addFilter("apiFilter", SimpleBeanPropertyFilter.filterOutAllExcept(filterProperties));
return mapper.filteredWriter(filters).writeValueAsString(user);
The problem is there are different service calls where I don't want to apply the filter at all. In those cases I want to return the entire domain class without filtering any properties. In the case where I just try to return the domain class I'm getting an exception as follows:
Caused by: org.codehaus.jackson.map.JsonMappingException: Can not resolve BeanPropertyFilter with id 'apiFilter'; no FilterProvider configured
at org.codehaus.jackson.map.ser.BeanSerializer.findFilter(BeanSerializer.java:252)
at org.codehaus.jackson.map.ser.BeanSerializer.serializeFieldsFiltered(BeanSerializer.java:216)
at org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:140)
I know it's already been answered but for any newcommers Jackson has actually added the ability to not fail on missing filters (JACKSON-650):
You just need to call
SimpleFilterProvider.setFailOnUnknownId(false) and you won't get this exception.
For Spring Boot / Jackson configuration just add:
#Configuration
public class JacksonConfiguration {
public JacksonConfiguration(ObjectMapper objectMapper) {
objectMapper.setFilterProvider(new SimpleFilterProvider().setFailOnUnknownId(false));
}
}
I think you could trick the filtered writer defining an empty serialize filter for the cases where you want all the properties seralized:
FilterProvider filters = new SimpleFilterProvider().addFilter("apiFilter", SimpleBeanPropertyFilter.serializeAllExcept(emptySet));
This way, when the engine looks for the "apiFilter" filter defined at the #JsonFilter anotation, it finds it, but it will not have any effect (as will serialize all the properties).
EDIT
Also, you can call the factory method writer() instead of filteredWriter():
ObjectWriter writer=null;
if(aplyFilter) {
FilterProvider filters = new SimpleFilterProvider().addFilter("apiFilter", SimpleBeanPropertyFilter.filterOutAllExcept(filterProperties));
writer=mapper.filteredWriter(filters);
} else {
writer=mapper.writer();
}
return writer.writeValueAsString(user);
I think this last solution is way cleaner, and indeed better.
I had a similar issue getting the same Exception, but the accepted answer didn't really help in my case. Here's the solution that worked for me:
In my setup I was using a custom JacksonSerializer like this:
#JsonSerialize(using = MyCustomSerializer.class)
private Object someAttribute;
And that serializer was implemented like this:
public class MyCustomSerializer extends JsonSerializer<Object> {
#Override
public void serialize(Object o, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonProcessingException {
if (o != null) {
jgen.writeObject(o);
}
}
}
The problem with this is, that as long as you don't use any filters, it works. It also works if you serialize primitives, so for instance if you use jgen.writeString(..). If you use filters, that code is wrong, because the filters are stored somewhere inside of the SerializerProvider, not in the JsonGenerator. If in that case you use the jsongenerator directly, a new SerializerProvider, that doesn't know about the filters, is created internally. So instead of the shorter jgen.writeObject(o) you need to call provider.defaultSerializeValue(o, jgen). That will ensure that the filters don't get lost and can be applied.
I have applied the same solution as mentioned accepted solution but when i am returning writer.writeValueAsString(course) as a Rest service response then i am getting response in below format
{ "status": "OK", "data": "[{\"name\":\"JPA in Use\",\"reviews\":[{\"id\":4081,\"rating\":\"4\",\"description\":\"Fine\"},{\"id\":4084,\"rating\":\"4\",\"description\":\"Ok\"}]},{\"name\":\"Spring in Use\",\"reviews\":[{\"id\":4003,\"rating\":\"3\",\"description\":\"Nice Course\"}]}]" }
But My expected Response is
{ "status": "OK", "data": [ { "name": "JPA in Use", "reviews": [ { "id": 4081, "rating": "4", "description": "Fine" }, { "id": 4082, "rating": "5", "description": "Great" } ] }, { "name": "Spring in Use", "reviews": [ { "id": 4003, "rating": "3", "description": "Nice Course" } ] } ] }
For getting my response i have applied converted the jsonstring to specfic object type
List<Course> resultcourse = mapper.readValue(writeValueAsString,List.class);
Note: Course has id,name and reviews as field and i want to suppress the id
I am providing the code snippet hope it is helpful to some.
#GetMapping("/courses")
public ResponseEntity<JpaResponse> allCourse() throws Exception {
JpaResponse response = null;
ObjectMapper mapper = new ObjectMapper();
mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
List<Course> course = service.findAllCourse();
SimpleBeanPropertyFilter filter = SimpleBeanPropertyFilter.filterOutAllExcept("name","reviews");
FilterProvider filterProvider = new SimpleFilterProvider().addFilter("jpafilter", filter).setFailOnUnknownId(false);
ObjectWriter writer = mapper.writer(filterProvider);
String writeValueAsString = writer.writeValueAsString(course);
List<Course> resultcourse = mapper.readValue(writeValueAsString,List.class);
response = new JpaResponse(HttpStatus.OK.name(),resultcourse);
return new ResponseEntity<>(response, HttpStatus.OK);
}
public class JpaResponse {
private String status;
private Object data;
public JpaResponse() {
super();
}
public JpaResponse(String status, Object data) {
super();
this.status = status;
this.data = data;
}
}
This is what I did for Springboot, no more logic is needed to filter those fields from all the REST responses in your application, if you need to filter more POJOs just add them to the FilterProvider:
Add a configuration class with the filter:
#Configuration
public class JacksonConfiguration {
public JacksonConfiguration(ObjectMapper objectMapper) {
SimpleFilterProvider simpleFilterProvider = new SimpleFilterProvider();
FilterProvider filters = simpleFilterProvider.addFilter("PojoFilterDTO",
SimpleBeanPropertyFilter.serializeAllExcept("field1", "field2")).setFailOnUnknownId(false);
objectMapper.setFilterProvider(filters);
objectMapper.configure(SerializationFeature.FAIL_ON_SELF_REFERENCES, false);
}
}
Add the JsonFilter annotation to your POJO:
#JsonInclude(JsonInclude.Include.NON_NULL)
#JsonFilter("PojoFilterDTO")
public class PojoDTO {
}