Deserializing F# Map by Json.Net - json

I have one simply question: Is it possible to parse F# Map type from json? Because when I try it (With F# Map<string, string>), it is easy to serialize and it looks how it have to, but when I try to deserialize it is throwing an exception.
Newtonsoft.Json.JsonSerializationException: Unable to find a default constructor to use for type Microsoft.FSharp.Collections.FSharpMap`2[System.Int32,System.String]. Path '1', line 2, position 7.
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewDictionary (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonDictionaryContract contract, System.Boolean& createdFromNonDefaultConstructor) [0x00000] in <filename unknown>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00000] in <filename unknown>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00000] in <filename unknown>:0
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType, Boolean checkAdditionalContent) [0x00000] in <filename unknown>:0
And it is deserializing from classic:
Map.ofList [ ("1", "one"); ("2", "two"); ("3", "three") ]
The resulting JSON looks like C# dictionary
{
"1": "one",
"2": "two",
"3": "three"
}
It is serializing without settings (Only indentation). So is it possible to serialize this, or is there some working workaround?
Thanks for answer

You can make your own converter to do this. It's a lot of reflection and constructing appropriate generic types, but it can be done.
You first deserialize to a Dictionary<Key, Val>, then create and fill a List<Tuple<Key, Val>> manually via reflection (because the Map constructor requires Tuples, not KeyValuePairs), then finally pass that into the Map constructor.
Not sure if there's an easier way, but this is what I came up with:
open System
open System.Collections
open System.Collections.Generic
open Newtonsoft.Json
let mapConverter = {
new JsonConverter() with
override x.CanConvert(t:Type) =
t.IsGenericType && t.GetGenericTypeDefinition() = typedefof<Map<_, _>>
override x.WriteJson(writer, value, serializer) =
serializer.Serialize(writer, value)
override x.ReadJson(reader, t, _, serializer) =
let genArgs = t.GetGenericArguments()
let generify (t:Type) = t.MakeGenericType genArgs
let tupleType = generify typedefof<Tuple<_, _>>
let listType = typedefof<List<_>>.MakeGenericType tupleType
let create (t:Type) types = (t.GetConstructor types).Invoke
let list = create listType [||] [||] :?> IList
let kvpType = generify typedefof<KeyValuePair<_, _>>
for kvp in serializer.Deserialize(reader, generify typedefof<Dictionary<_, _>>) :?> IEnumerable do
let get name = (kvpType.GetProperty name).GetValue(kvp, null)
list.Add (create tupleType genArgs [|get "Key"; get "Value"|]) |> ignore
create (generify typedefof<Map<_, _>>) [|listType|] [|list|]
}
Once you have your converter, then you just pass it into the DeserializeObject method and JsonConvert will use it wherever appropriate.
let str = JsonConvert.SerializeObject (Map<_, _> [333, 1234])
JsonConvert.DeserializeObject<Map<int, int>>(str, mapConverter)
The nice thing about doing it this way is that if you've got a big/deep record where your Map is just a single field, then it'll work with that too--you don't have to go changing your record structure to use Dictionaries just to support serialization.

This functionality became part of JSON.Net in version 6.0.3. (April 30th, 2014)
But, if you are stuck for some reason using an earlier version then a simplified (and more efficient as less reflection) version of Dax Fohl's version could be:
type mapConvert<'f,'t when 'f : comparison>() =
static member readJson (reader:JsonReader, serializer:JsonSerializer) =
serializer.Deserialize<Dictionary<'f, 't>> (reader)
|> Seq.map (fun kv -> kv.Key, kv.Value)
|> Map.ofSeq
let mapConverter = {
new JsonConverter() with
override __.CanConvert (t:Type) =
t.IsGenericType && t.GetGenericTypeDefinition() = typedefof<Map<_, _>>
override __.WriteJson (writer, value, serializer) =
serializer.Serialize(writer, value)
override __.ReadJson (reader, t, _, serializer) =
let converter =
typedefof<mapConvert<_,_>>.MakeGenericType (t.GetGenericArguments())
let readJson =
converter.GetMethod("readJson")
readJson.Invoke(null, [| reader; serializer |])
}

The problem is that json.net can't construct a Map<int,string>. However if you deserialize to a regular .net Dictionary<int,string> it will work, as the json is the same.

You can't serialize F#'s Map directly, since it has no default contructor (constructor with no parameter) at all.
This is the original documentation of F# map: (from http://msdn.microsoft.com/en-us/library/ee353686%28v=vs.110%29.aspx)
[<Sealed>]
type Map<[<EqualityConditionalOnAttribute>] 'Key,[<ComparisonConditionalOnAttribute>] [<EqualityConditionalOnAttribute>] 'Value (requires comparison)> =
class
interface IEnumerable
interface IComparable
interface IEnumerable
interface ICollection
interface IDictionary
new Map : seq<'Key * 'Value> -> Map< 'Key, 'Value>
member this.Add : 'Key * 'Value -> Map<'Key, 'Value>
member this.ContainsKey : 'Key -> bool
member this.Remove : 'Key -> Map<'Key, 'Value>
member this.TryFind : 'Key -> 'Value option
member this.Count : int
member this.IsEmpty : bool
member this.Item ('Key) : 'Value
end
As you see above, Map doesn't have default constructor but the serializer need a class with default constructor.
The best way to serialize a map is mapping the map to be regular .NET dictionary, but then the new dictionary doesn't have all of the advantages of F#'s Map, especially the immutability of F#'s Map.

Related

Using a KClass reference as a reified parameter to deserialize from JSON

I'm trying to implement a general serialization framework to convert outgoing and incoming messages to json using the kotlinx serialialization. I'm developing a multiplatform app, so I'm trying to get it to run on KotlinJVM and KotlinJS.
For this, I add a type field to every message and use a map that maps each type string to a KClass. What's the type for that map? It contains KClass<> objects whose classes extend the Message class, therefore in java I'd specify my map as
Map<KClass<? extends Message>, String>.
How can I do that in Kotlin?
Afterwards I need to serialize and deserialize the message based on its key and therefore type. Java frameworks take a Class parameter for the type of the object I want to deserialize/instantiate (e.g. gson.fromJson(ClientMessage.class)). In Kotlin this is done using reified parameters Json.decodeFromString<Type>. I do not know the type of the message at compile time though and just have a reference to a KClass, how can I instantiate an object based on that?
#Serializable
open class Message(val type: String) {
companion object {
val messageTypes: Map<KClass<out Message>, String> = mapOf(
ClientLoginMessage::class to "clientLoginMessage",
Message::class to "message"
)
inline fun <reified T> getMessageTypeByClass(): String = messageTypes[T::class]!! // utility for defining the type in the constructors of the individual messages
}
fun toJson() = Json.encodeToString(this)
fun fromJson(json: String): Message? {
val plainMessage = Json.decodeFromString<Message>(json) // get type string from json
return messageTypes.entries.find { it.value == plainMessage.type }?.let {
// how can I use the KClass from it.key as reified parameter?
Json.decodeFromString<?????>(json)
}
}
}
#Serializable
class ClientLoginMessage
: Message(Message.getMessageTypeByClass<ClientLoginMessage>()) {}
Create a map of serializers like for types:
val serializers: Map<KClass<out Message>, KSerializer<out Message>> = mapOf(
ClientLoginMessage::class to ClientLoginMessage.serializer(),
Message::class to Message.serializer()
)
Pass in the serializer needed to Json.decodeFromString like this:
fun fromJson(json: String): Message? {
val plainMessage = Json.decodeFromString<Message>(json) // get type string from json
return messageTypes.entries.find { it.value == plainMessage.type }?.let {
// how can I use the KClass from it.key as reified parameter?
Json.decodeFromString(serializers.get(plainMessage.type)!!, json)
}
}
You might also want to have a look at the Kotlin built in handling of polymorphic classes: https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/polymorphism.md

F# Custom JsonConverter for Option

I'm trying to create an option type customer serialiser here is the JsonConverter
I've got:
type OptionConverter<'T> =
inherit JsonConverter<'T>
override __.WriteJson (writer: Utf8JsonWriter, value: 'T option, serializer: JsonSerializerOptions) : unit =
match value with
| Some optionValue ->
JsonSerializer.Serialize(writer, optionValue, serializer)
| None ->
writer.WriteNullValue()
override __.ReadJson (reader: byref<Utf8JsonWriter>, typeToConvert: Type, serializer: JsonSerializerOptions): 'T =
failwith "Not implemented"
type OptionConverterFactory =
inherit JsonConverterFactory
override __.CanConvert(typeToConvert: Type) =
let isOption = if typeToConvert.GetGenericTypeDefinition() = typeof<option<_>> then true else false
isOption
override __.CreateConverter(typeToConvert: Type, options: JsonSerializerOptions) : JsonConverter =
let optionType = typeToConvert.GetGenericArguments().[0]
let converter = Activator.CreateInstance(typeof<OptionConverter<_>>.MakeGenericType([| optionType |])) :?> JsonConverter
converter
I get red line underneath WriteJson saying no abstract or interface member found that corresponds with this override...
Also, if I change WriteJson and ReadJson to member then OptionConvert<'T> gets red underline saying that WriteJson and ReadJson have not been implemented...
How do I fix this?
I think there are two problems here:
The wrong methods are being overloaded
You need to constrain your 'T to be an option.
This compiled:
open System
open System.Text.Json
open System.Text.Json.Serialization
type OptionConverter<'T> =
inherit JsonConverter<Option<'T>>
override _.Write(writer, value, serializer) =
match value with
| Some optionValue ->
JsonSerializer.Serialize(writer, optionValue, serializer)
| None ->
writer.WriteNullValue()
override _.Read(reader, typeToConvert, options) =
failwith "Not implemented"

Convert ByteArrayOutputStream to json in Kotlin

I'm trying to create a resource for 2 services, 1 in application/x-www-form-urlencoded and string payload and the other application/json format with json body.
I have this code:
#POST #Path("/test")
fun test(#Context request: ContainerRequest): Response {
val baos = ByteArrayOutputStream()
request.entityStream.use { it.copyTo(baos) }
val ipnRawData = baos.toString()
var map : Map<String,Any>
map = when (request.headers.getFirst("Content-Type")) {
"application/json" -> objectMapper.convertValue(ipnRawData,Map::class.java) as Map<String,Any>
"application/x-www-form-urlencoded" -> LinkedHashMap()
else -> throw UnsupportedOperationException()
}
//....handle the map
return Response.status(200).build()
}
But when I try to run it with the json option, and body: {"name" :"test"}), I get an error:
"java.lang.IllegalArgumentException: Can not construct instance of java.util.LinkedHashMap: no String-argument constructor/factory method to deserialize from String value ('{
"name" :"test"}')"
Thanks for any help, Yoel
You should use mapper.readValue to deserialize JSON into an object.
Using raw Jackson without the Jackson-Kotlin module:
val map: Map<String, String> = JSON.readValue("""{"name" :"test"}""",
object : TypeReference<Map<String, String>>() {})
This passes in an object expression with superclass TypeReference specifying the the type you are wanting to create with full generics still intact (you method suffers from type erasure).
Instead, if you are using the Jackson-Kotlin module you only need:
val map: Map<String, String> = JSON.readValue("""{"name" :"test"}""")
Since it has helper/extension functions to hide some of the uglier things like TypeReference creation.
You should always use the Jackson-Kotlin module with Kotlin code so that you can instantiate any type of Kotlin object including data classes that have all val parameters and no default constructors, have it understand nullability, and also deal with default values for constructor parameters. A simple stand-alone example:
import com.fasterxml.jackson.module.kotlin.*
val JSON = jacksonObjectMapper() // creates ObjectMapper() and adds Kotlin module in one step
val map: Map<String, String> = JSON.readValue("""{"name" :"test"}""")
Notice the import .* so that it picks up all the extension functions otherwise you need to explicitly import: com.fasterxml.jackson.module.kotlin.readValue
Or in your case the modified code would be:
import com.fasterxml.jackson.module.kotlin.readValue
val objectMapper = jacksonObjectMappe() // instead of ObjectMapper()
...
#POST #Path("/test")
fun test(#Context request: ContainerRequest): Response {
val bodyAsString = request.entityStream.bufferedReader().readText()
val map: Map<String, Any> = when (request.headers.getFirst("Content-Type")) {
"application/json" -> objectMapper.readValue(bodyAsString)
"application/x-www-form-urlencoded" -> LinkedHashMap()
else -> throw UnsupportedOperationException()
}
//....handle the map
return Response.status(200).build()
}
The code has also been cleaned up a little to remove the use of a var and to read the entity stream in a more Kotlin friendly way.
Also note that the Content-Type header may be more complicated, it could contain encoding as well such as:
Content-type: application/json; charset=utf-8
So you may want a utility function that checks if the header is "equal to application/json or starts with application/json;" instead of only an equality check.
Lastly you could pass the request.entityStream directly to objectMapper.readValue and never copy it into a string at all. There are various overloads for readValue that are helpful for these types of inputs.

Converting DateTime to a JSON string

I want to convert a case class with an Option[DateTime] parameter to a spray-json object which can be served by an API. Using spray-json I have a custom JsonFormat as such
object JsonImplicits extends DefaultJsonProtocol {
implicit object PostJsonFormat extends RootJsonFormat[Post] {
def write(p: Post) = JsObject(
"title" -> JsString(p.title),
"content" -> JsString(p.content),
"author" -> JsString(p.author),
"creationDate" -> JsString(p.creationDate.getOrElse(DateTime.now))
)
}
}
But I get:
overloaded method value apply with alternatives:
(value: String)spray.json.JsString <and>
(value: Symbol)spray.json.JsString
cannot be applied to (com.github.nscala_time.time.Imports.DateTime)
"creationDate" -> JsString(p.creationDate.getOrElse(DateTime.now))
when I try to compile it and no matter what I try I can't seem to convert the DateTime object to a string. For instance, when I try calling toString I get
ambiguous reference to overloaded definition,
both method toString in class AbstractDateTime of type (x$1: String, x$2: java.util.Locale)String
and method toString in class AbstractDateTime of type (x$1: String)String
match expected type ?
"creationDate" -> JsString(p.creationDate.getOrElse(DateTime.now.toString)))
You have several problems here.
First, the toString() method in AbstractDateTime requires one or several arguments see here.
But I would advise you against this path and recommend using properly Spray-Json.
Spray-json does not know how to serialize Option[DateTime], therefore you have to provide a RootJsonFormat for it.
This is what I am doing.
implicit object DateJsonFormat extends RootJsonFormat[DateTime] {
private val parserISO : DateTimeFormatter = ISODateTimeFormat.dateTimeNoMillis();
override def write(obj: DateTime) = JsString(parserISO.print(obj))
override def read(json: JsValue) : DateTime = json match {
case JsString(s) => parserISO.parseDateTime(s)
case _ => throw new DeserializationException("Error info you want here ...")
}
}
Adapt it as you want if you do not want to use ISO formatting.

How to convert an object containing DateTime fields to JSON in Dart?

I try to convert an object to JSON.
var obj = { "dt": new DateTime.now() };
var s = stringify(obj);
The runtime throws an exception: "Calling toJson method on object failed."
That's expected since DateTime class doesn't have toJson method.
But what should I do in this case?
Javascript's JSON.stringify function has an optional argument replacer which allows me to provide my own way of serialization of any object even if the object has no toJson method. Is there any similar facility in Dart or maybe I can extend somehow DateTime class with my own toJson method?
JSON conversion only works with maps, lists, strings, numbers, booleans, or null. So what if your object contains another type like DateTime?
DateTime → JSON
Let's start with the following object:
class Person {
Person(this.name, this.birthdate);
String name;
DateTime birthdate;
}
You can convert it to a map like this:
final person = Person('Bob', DateTime(2020, 2, 25));
Map<String, dynamic> map = {
'name': person.name,
'birthdate': person.birthdate,
};
If you tried to encode this right now with jsonEncode (or json.encode), you would get an error because the DateTime is not directly serializeable. There are two solutions.
Solution 1
You could serialize it yourself first like this:
Map<String, dynamic> map = {
'name': person.name,
'birthdate': person.birthdate.toIso8601String(),
};
final jsonString = json.encode(map);
Note:
Here is the difference between toString and toIso8601String:
2020-02-25 14:44:28.534 // toString()
2020-02-25T14:44:28.534 // toIso8601String()
The toIso8601String doesn't have any spaces so that makes it nicer for conversions and sending over APIs that might not deal with spaces well.
Solution 2
You could use the optional toEncodable function parameter on jsonEncode.
import 'dart:convert';
void main() {
final person = Person('Bob', DateTime(2020, 2, 25));
Map<String, dynamic> map = {
'name': person.name,
'birthdate': person.birthdate,
};
final toJson = json.encode(map, toEncodable: myDateSerializer);
}
dynamic myDateSerializer(dynamic object) {
if (object is DateTime) {
return object.toIso8601String();
}
return object;
}
The toEncodable function just converts the input to a string or something that jsonEncode can covert to a string.
JSON → DateTime
There is nothing special here. You just have to parse the string into the type that you need. In the case of DateTime you can use its parse or tryParse methods.
final myMap= json.decode(jsonString);
final name = myMap['name'];
final birthdateString = myMap['birthdate'];
final birthdate = DateTime.parse(birthdateString);
final decodedPerson = Person(name, birthdate);
Note that parse will throw an exception if the format of the string cannot be parsed into a DateTime object.
As a model class
class Person {
Person(this.name, this.birthdate);
String name;
DateTime birthdate;
Person.fromJson(Map<String, dynamic> json)
: name = json['name'],
birthdate = DateTime.tryParse(json['birthdate']),
Map<String, dynamic> toJson() {
return {
'name': name,
'birthdate': birthdate.toIso8601String(),
};
}
}
This will not throw an exception is the date is malformatted, but birthdate would be null.
Notes
See my fuller answer here.
Thanks to this answer for pointing me in the right direction.
Zdeslav Vojkovic's answer is outdated now.
The JSON.encode() method in dart:convert has an optional toEncodable method that is invoked for objects that are not natively serializable to JSON. It's then up to the user to provide a closure that returns an appropriate serialization of the DateTime.
IMO, it's a flaw in dart:json library that stringify doesn't support additional callback to serialize types lacking the implementation of toJson. Strangely enough, parse function does support reviver argument to customize the deserialization process. Probably the reason was along the lines that user can add toJson for their on types, and core library will take care of 'native' types, but DateTime was omitted (maybe because date serialization in JSON is not really a straightforward story).
Maybe the goal of the team was to use custom types instead of Maps (which is nice), but the drawback here is that if your custom type contains 10 other properties and 1 which is of DateTime type, you need to implement toJson which serializes all of them, even integers and strings. Hopefully, once when Mirror API is ready, there will be libraries that implement serialization 'out-of-band' by reading the reflected info on type, see lower for an example. The promise of Dart is that I would be able to use my types both on server and client, but if I have to manually implement serialization/deserialization for all my models then it is too awkward to be usable.
it also doesn't help that DateTime itself is not very flexible with formatting, as there are no other methods besides toString which returns the format useless for serialization.
So here are some options:
wrap (or derive from) DateTime in your own type which provides toJson
patch json.stringifyJsonValue and submit the change or at least submit an issue :)
use some 3-rd party library like json-object (just an example, it also doesn't support DateTime serialization, AFAIK
I am not really happy with any of them :)
I've added a new package to Dart pub.dev that allows json serialization of objects within a structure. This package Jsonize serialize and deserialize custom classes, and handles DateTime out of the box:
List<dynamic> myList = [1, "Hello!", DateTime.now()];
var jsonRep = Jsonize.toJson(myList);
var myDeserializedList = Jsonize.fromJson(jsonRep);
So will do with your example
var obj = { "dt": new DateTime.now() };
var s = Jsonize.toJson(obj);
var obj2 = Jsonize.fromJson(s);
but can do also this
var obj = DateTime.now();
var s = Jsonize.toJson(obj);
var dt = Jsonize.fromJson(s);