Newtonsoft deserializing Json incorrectly - json

I have encountered a case where Newtonsoft is taking perfectly valid JSON text, but deserializing it incorrectly. I have an object that contains an embedded class that consists of members Year, Month, Week, and DayOfWk. The JSON looks like this:
"openDate": {
"Year": 1997,
"Month": 12,
"Week": 5,
"DayOfWk": 5
},
But the data that comes back after deserialization is Year = 1, Month = 1, Week = 1, and DayOfWk = 1, regardless of the input JSON.
Here is the code (it's in F#, but should be easily readable):
let jsonText = File.ReadAllText( #"..\..\..\..\Dependencies\ADBE.dat")
let dailyData = JsonConvert.DeserializeObject<DailyAnalysis[]>(jsonText)
DailyAnalysis is defined as:
type DailyAnalysis = {
openDate: TradeDate
openPrice: Decimal
closeDate: TradeDate
closePrice: Decimal
gainPercentage: Decimal
}
TradeDate is the class in question - it is an F# class that exposes properties Year, Month, Week, and DayOfWk. Year, Month, and Week are int's; DayOfWeek is a DayOfWeek enum. All the other fields in the DailyAnalysis objects come back with the correct values.
How can this problem be resolved?
Note that if I don't include the type in the DeserializeObject call, it does get the correct data, but simply returns it as an object, and converting to the correct type is very difficult (i.e., I don't know how to do it in F#).
Can anybody point out something obvious (or even obscure) I'm overlooking, or point me to other resources?
Update: note that the constructor for TradeDate takes a single DateTime argument.

Assuming that your TradeDate is immutable (as typically happens in f#), then Json.NET is able to deserialize such a type by finding a single constructor which is parameterized, then invoking it by matching constructor arguments to JSON properties by name, modulo case. Arguments that do not match are given a default value. If TradeDate actually takes a single DateTime as input, you will get the behavior you are seeing.
For instance, if we take a simplified version like so:
type TradeDate(date : DateTime) =
member this.Year = date.Year
member this.Month = date.Month
member this.DayOfMonth = date.Day
And then round-trip it using Json.NET as follows:
let t1 = new TradeDate(new DateTime(1997, 12, 25))
let json1 = JsonConvert.SerializeObject(t1)
let t2 = JsonConvert.DeserializeObject<TradeDate>(json1)
let json2 = JsonConvert.SerializeObject(t2)
printfn "\nResult after round-trip:\n%s" json2
The result becomes:
{"Year":1,"Month":1,"DayOfMonth":1}
Which is exactly what you are seeing. Demo fiddle #1 here.
So, what are your options? Firstly, you could modify TradeDate to have the necessary constructor, and mark it with JsonConstructor. It could be private as long as the attribute is applied:
type TradeDate [<JsonConstructor>] private(year : int, month : int, dayOfMonth: int) =
member this.Year = year
member this.Month = month
member this.DayOfMonth = dayOfMonth
new(date : DateTime) = new TradeDate(date.Year, date.Month, date.Day)
Demo fiddle #2 here.
Secondly, if you cannot modify TradeDate or add Json.NET attributes to it, you could introduce a custom JsonConverter for it:
[<AllowNullLiteral>] type private TradeDateDTO(year : int, month : int, dayOfMonth : int) =
member this.Year = year
member this.Month = month
member this.DayOfMonth = dayOfMonth
type TradeDateConverter () =
inherit JsonConverter()
override this.CanConvert(t) = (t = typedefof<TradeDate>)
override this.ReadJson(reader, t, existingValue, serializer) =
let dto = serializer.Deserialize<TradeDateDTO>(reader)
match dto with
| null -> raise (new JsonSerializationException("null TradeDate"))
| _ -> new TradeDate(new DateTime(dto.Year, dto.Month, dto.DayOfMonth)) :> Object
override this.CanWrite = false
override this.WriteJson(writer, value, serializer) =
raise (new NotImplementedException());
And deserialize as follows:
let converter = new TradeDateConverter()
let t2 = JsonConvert.DeserializeObject<TradeDate>(json1, converter)
Demo fiddle #3 here.
Notes:
Your question did not include code for TradeDate, in particular the code for converting between a DateTime and the year/month/week of month/day of week representation. This turns out to be slightly nontrivial so I did not include it in the answer; see Calculate week of month in .NET and Calculate date from week number for how this might be done.
For details on how Json.NET chooses which constructor to invoke for a type with multiple constructors, see How does JSON deserialization in C# work.

Related

Jsony newHook has `SIGSEGV: Illegal storage access. (Attempt to read from nil?)` when deserializing into ref-objects

I am writing a web-application and am deserializing via jsony into norm-model-object types.
Norm-model-types are always ref objects. Somehow my code which is very similar to the default example in jsony's github documentation does not compile. Instead I receive the error SIGSEGV: Illegal storage access. (Attempt to read from nil?).
See here my code sample
import std/[typetraits, times]
import norm/[pragmas, model]
import jsony
const OUTPUT_TIME_FORMAT* = "yyyy-MM-dd'T'HH:mm:ss'.'ffffff'Z'"
type Character* {.tableName: "wikientries_character".} = ref object of Model
name*: string
creation_datetime*: DateTime
update_datetime*: DateTime
proc parseHook*(s: string, i: var int, v: var DateTime) =
##[ jsony-hook that is automatically called to convert a json-string to datetime
``s``: The full JSON string that needs to be serialized. Your type may only be a part of this
``i``: The index on the JSON string where the next section of it starts that needs to be serialized here
``v``: The variable to fill with a proper value]##
var str: string
s.parseHook(i, str)
v = parse(s, OUTPUT_TIME_FORMAT, utc())
proc newHook*(entry: var Character) =
let currentDateTime: DateTime = now()
entry.creation_datetime = currentDateTime # <-- This line is listed as the reason for the sigsev
entry.update_datetime = currentDateTime
entry.name = ""
var input = """ {"name":"Test"} """
let c = input.fromJson(Character)
I don't understand what the issue appears to be here, as the jsony-example on its github page looks pretty similar:
type
Foo5 = object
visible: string
id: string
proc newHook*(foo: var Foo5) =
# Populates the object before its fully deserialized.
foo.visible = "yes"
var s = """{"id":"123"}"""
var v = s.fromJson(Foo5)
doAssert v.id == "123"
doAssert v.visible == "yes"
How can I fix this?
The answer lies in the fact that norm-object-types are ref objects, not normal (value) objects (Thanks to ElegantBeef, Rika and Yardanico from the nim-discord to point this out)! If you do not explicitly 'create' a ref-type at one point, the memory for it is never allocated since the code doesn't do the memory allocation for you unlike with value types!
Therefore, you must initialize/create a ref-object first before you can use it, and Jsony does not take over initialization for you!
The correct way to write the above newHook thus looks like this:
proc newHook*(entry: var Character) =
entry = new(Character)
let currentDateTime: DateTime = now()
entry.creation_datetime = currentDateTime
entry.update_datetime = currentDateTime
entry.name = ""

Reading a row with a NULL column causes an exception in slick

I have a table with a column type date. This column accepts null values,
therefore, I declared it as an Option (see field perDate below). When I
run the select query through the application code I get the following exception
slick.SlickException: Read NULL value (null) for ResultSet column
problem.This
is the Slick table definition:
import java.sql.Date
import java.time.LocalDate
class FormulaDB(tag: Tag) extends Table[Formula](tag, "formulas") {
def sk = column[Int]("sk", O.PrimaryKey, O.AutoInc)
def formula = column[Option[String]]("formula")
def notes = column[Option[String]]("notes")
def periodicity = column[Int]("periodicity")
def perDate = column[Option[LocalDate]]("per_date")(localDateColumnType)
def * =
(sk, name, descrip, formula, notes, periodicity, perDate) <>
((Formula.apply _).tupled, Formula.unapply)
implicit val localDateColumnType = MappedColumnType.base[Option[LocalDate], Date](
{
case Some(localDate) => Date.valueOf(localDate)
case None => null
}, { sqlDate =>
if (sqlDate != null) Some(sqlDate.toLocalDate) else None
}
)
}
Your mapped column function just needs to provide the LocalDate to Date conversion. Slick will automatically handle Option[LocalDate] if it knows how to handle LocalDate.
That means changing your localDateColumnType to be:
implicit val localDateColumnType = MappedColumnType.base[LocalDate, Date](
Date.valueOf(_), _.toLocalDate
)
Chapter 5 of Essential Slick covers some of this, as does the section on User Defined Features in the Manual.
I'm not 100% sure why you're seeing the run-time error: my guess is that the column is being treated as an Option[Option[LocalDate]] or similar, and there's a level of null in there that's being missed.
BTW, your def * can probably be:
def * = (sk, name, descrip, formula, notes, periodicity, perDate).mapTo[Formula]
...which is a little nicer to read. The mapTo was added in Slick 3 at some point.

How to access json schema info in SparkDataset Api when using plain map/reduce functions?

Given there is a dataset of messages, defined by following code:
case class Message(id: Int, value: String)
var messages = Seq(
(0, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}"""),
(1, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}""")
).toDF("id", "value").as[Message]
var schema = new StructType().add("action", StringType).add("timestamp", TimestampType)
var res = messages.select(
from_json(col("value").cast("string"), schema)
)
+------------------------------------+
|jsontostructs(CAST(value AS STRING))|
+------------------------------------+
| [update,2017-10-0...|
| [update,2017-10-0...|
What is the best way to access the schema information in a plain map function. The function itself returns a row which has lost all the Type infos. In order to reach to the values one has to specify the type again e.g
res.head().getStruct(0).getValuesMap[TimestampType](Seq("timestamp"))
=> Map[String,org.apache.spark.sql.types.TimestampType] = Map(timestamp -> 2017-10-06 01:01:19.0)
or
res.head().getStruct(0).getString(0)
=> res20: String = update
Is there some better way to access the raw json data without spark sql aggregation functions?
As a rule of thumb:
To use collection API (map, flatMap, mapPartitions, groupByKey, etc.) use strongly typed API - define record type (case class works the best) which reflects the schema and use Encoders to convert things back and forth:
case class Value(action: String, timestamp: java.sql.Timestamp)
case class ParsedMessage(id: Int, value: Option[Value])
messages.select(
$"id", from_json(col("value").cast("string"), schema).alias("value")
).as[ParsedMessage].map(???)
With Dataset[Row] stay with high level SQL / DataFrame API (select, where, agg, groupBy)

Order of fields in a type for FileHelpers

I'm reading a simple CSV file with Filehelpers - the file is just a key, value pair. (string, int64)
The f# type I've written for this is:
type MapVal (key:string, value:int64) =
new()= MapVal("",0L)
member x.Key = key
member x.Value = value
I'm missing something elementary here, but FileHelpers always assumes the order of fields to be the reverse of what I've specified - as in Value, Key.
let dfe = new DelimitedFileEngine(typeof<MapVal>)
let recs = dfe.ReadFile(#"D:\e.dat")
recs |> Seq.length
What am I missing here?
The order of primary constructor parameters doesn't necessarily determine the order that fields occur within a type (in fact, depending on how the parameters are used, they may not even result in a field being generated). The fact that FileHelpers doesn't provide a way to use properties instead of fields is unforunate, in my opinion. If you want better control over the physical layout of the class, you'll need to declare the fields explicitly:
type MapVal =
val mutable key : string
val mutable value : int64
new() = { key = ""; value = 0L }
new(k, v) = { key = k; value = v }
member x.Key = x.key
member x.Value = x.value
The library uses the order of the field in the declaration, but looks like that F# words different, in the last the last stable version of the library you can use the [FieldOrder(1)] attribute to provide the order of the fields.
http://teamcity.codebetter.com/viewLog.html?buildId=lastSuccessful&buildTypeId=bt66&tab=artifacts&guest=1
Cheers

ActionScript: Is there ever a good reason to use 'as' casting?

From what I understand of ActionScript, there are two kinds of casts:
var bar0:Bar = someObj as Bar; // "as" casting
var bar1:Bar = Bar(someObj); // "class name" casting (for want of a better name)
Also, and please correct me if I'm wrong here, as casting will either return an instance of the class or null, while "class name" casting will either return an instance of the class or raise an exception if the cast is impossible – other than this, they are identical.
Given this, though, as casting seems to be a massive violation of the fail-fast-fail-early principle... And I'm having trouble imagining a situation where it would be preferable to use an as cast rather than a class name cast (with, possibly, an instanceof thrown in there).
So, my question is: under what circumstances would it be preferable to use as casting?
There are a couple of points in this discussion worth noting.
There is a major difference in how the two work, Class() will attempt to cast the object to the specified Class, but on failure to do so will (sometimes, depends on datatype) throw a runtime error. On the other hand using object as Class will preform a type check first, and if the specified object cannot be cast to the indicated Class a null value is returned instead.
This is a very important difference, and is a useful tool in development. It allows us to do the following:
var o:MyClass = myArray[i] as MyClass;
if(o)
{
//do stuff
}
I think the usefulness of that is pretty obvious.
"as" is also more consistent with the rest of the language (ie: "myObject is MyClass").
The MyClass() method has additional benefits when working with simple data types (int, Number, uint, string) Some examples of this are:
var s:String = "89567";
var s2:String = "89 cat";
var n:Number = 1.9897;
var i:int = int(s); // i is = 89567, cast works
var i2:int = int(s2); //Can't convert so i2 is set to 0
var i3:int = int(n); // i = 1
var n2:Number = Number(s2); // fails, n2 = NaN
//when used in equations you'll get very different results
var result:int = int(n) * 10; //result is 10
var result:int = n * 10; //result is 19.89700
var result:int = int(s2) * 10; //result is 0
trace(s2 as Number); //outputs null
trace(s2 as int); //outputs null
trace(Number(s2)); //outputs NaN
This is a good and important topic, as a general rule I use "as" when working with Objects and Cast() when using simpler data types, but that's just how I like to structure my code.
You need to use as to cast in two scenarios: casting to a Date, and casting to an Array.
For dates, a call to Date(xxx) behaves the same as new Date().toString().
For arrays, a call to Array(xxx) will create an Array with one element: xxx.
The Class() casting method has been shown to be faster than as casting, so it may be preferable to as when efficiency matters (and when not working with Dates and Arrays).
import flash.utils.*;
var d = Date( 1 );
trace( "'" + d, "'is type of: ",getQualifiedClassName( d ) );
var a:Array = Array( d );
trace( "'" + a, "' is type of: ", getQualifiedClassName( a ) );
//OUTPUT
//'Mon Jun 15 12:12:14 GMT-0400 2009 'is type of: String
//'Mon Jun 15 12:12:14 GMT-0400 2009 ' is type of: Array
//COMPILER ERRORS/WARNINGS:
//Warning: 3575: Date(x) behaves the same as new Date().toString().
//To cast a value to type Date use "x as Date" instead of Date(x).
//Warning: 1112: Array(x) behaves the same as new Array(x).
//To cast a value to type Array use the expression x as Array instead of Array(x).
`
They actually do different things...when you say
myvar as ClassName
You are really just letting the compiler know that this object is either a ClassName or a subclass of ClassName
when you say:
ClassName(myvar)
It actually tries to convert it to that type of object.
so if your object is a or a descent of the class and you do not need to convert it you would use as
examples:
var myvar:String = '<data/>';
var othervar:XML = XML(myvar); //right
var myvar:String = '<data/>';
var othervar:XML = (myvar as XML); //wrong
var myvar:XML = <data/>;
var othervar:XML = myvar as XML; // right
Use 'as' with arrays.
var badArray:Array;
badArray = Array(obj);
Will yield an array of length one with the original array in the first element. If you use 'as' as follows, you get the exptected result.
var goodArray:Array;
goodArray = obj as Array;
Generally, 'as' is preferable to 'Class()' in ActionScript as it behaves more like casting in other languages.
I use it when I have an ArrayCollection of objects and need to enumerate through them, or use a selector function.
e.g.
var abc:mytype = mycollection.getItemAt(i) as mytype