Delphi - Parse TNullable (record) to JSON - json

We are using the Spring4D nullable types (which are records, not objects) in some of our business objects that need to be parsed to JSON.
When the nullable type field has no value, there are 2 options that would be okay in our case:
The field is not present in the JSON
The field is present in the JSON with value null
I am trying to make this work by using a TJSONInterceptor subclass.
For example for the TNullableInteger:
I want to create an interceptor that will be used when the field has a NullableIntegerAttribute (derived from JsonReflectAttribute), in which case my TNullableIntegerInterceptor will be used.
The problem is that I don't quite know which convertertype and revertertype to use in this case because the nullable types are record types and not object types.
Does anyone have any experience with parsing record types in Delphi?
Or are there other ways to achieve this?
Any guidance would be much appreciated.
UPDATE
In the meantime I have managed to make interceptors for most of the TNullable types that we use.
Example for TNullableInteger:
{ JsonNullableIntegerAttribute }
constructor JsonNullableIntegerAttribute.Create;
begin
inherited Create(ctObject, rtString, TNullableIntegerInterceptor);
end;
{ TNullableIntegerInterceptor }
constructor TNullableIntegerInterceptor.Create;
begin
inherited;
ConverterType := ctObject;
ReverterType := rtString;
end;
function TNullableIntegerInterceptor.ObjectConverter(Data: TObject; Field: string): TObject;
var
ctx: TRTTIContext;
Int: TNullableInteger;
begin
Int := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TNullableInteger>;
If Int.HasValue
then begin
StringProxy.Value := IntToStr(Int.GetValueOrDefault);
Result := StringProxy;
end
else Result := nil;
end;
procedure TNullableIntegerInterceptor.StringReverter(Data: TObject; Field, Arg: string);
var
ctx: TRTTIContext;
Value: TValue;
Int: Integer;
begin
If TryStrToInt(Arg, Int)
then begin
TValue.Make<TNullableInteger>(Int, Value);
ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, Value);
end;
end;
The only type that is still a problem is TNullableBoolean.
For the unmarshalling the procedure TJONUnMarshal.PopulateFields checks if there is a reverter available but then, depending on the revertertype, it only handles TJSONArray, TJSONObject, TJSONString and TJSONNull. Boolean values in JSON are of type TJSONBool and are not handled. Instead I get the error 'JSON object expected for field fieldname in JSON true'
Also, for the serialization, I don't know how I can achieve that my JSON looks like
"FieldName": true
instead of
"FieldName": "true"
Passing a boolean as a string in JSON is not allowed by our backend software. So for now I am using a rather stupid workaround in my ObjectConverter method by returning "BOOLEAN_TRUE_IDENTIFIER" or "BOOLEAN_FALSE_INDENTIFIER" which I replace by true or false in the resulting JSON just before sending it to our backend.
If anyone would have a solution for this, it would make me very happy :-)
In the meantime we will try to avoid using NullableBoolean variables.

Related

Memory leak When Returning TJSONVALUE from Function

I have a JSON array of objects that I iterate. From each element of the array I retrieve further json data via a function that returns a JSONValue. I then must add that returned jsonvalue to the specific array element.
It works perfectly except that Delphi is reporting a memory leak on shutdown and I just can't find it. I'm pretty new to JSON and I've spent all day on this. Can someone help by suggesting the right way to go about this. Simplified code below. Many Thanks.
function TForm1.Function1(ThisProductJSON: String): Integer;
Var
MultiObj : TJSONObject;
OtherValues : TJSONVALUE;
ThisObject : TJSONObject;
I : Integer;
jsArr : TJSONARRAY;
S:String;
Stockcode : String;
begin
MultiObj:=TJSONObject.Create(nil);
s:='[{"ThisElement":1,"Thiscode":12345,"ThisBarcode":"2345678901231","Price":1.00,"Measure":"EA","Description":"Some Description"},'+
'{"ThisElement":2,"Thiscode":21345,"ThisBarcode":"3124567890123","Price":2.00,"Measure":"EA","Description":"Some Description"},'+
'{"ThisElement":3,"Thiscode":31345,"ThisBarcode":"6123457890123","Price":3.00,"Measure":"EA","Description":"Some Description"},'+
'{"ThisElement":4,"Thiscode":41345,"ThisBarcode":"9123456780123","Price":4.00,"Measure":"EA","Description":"Some Description"},'+
'{"ThisElement":5,"Thiscode":51345,"ThisBarcode":"8234567901235","Price":5.00,"Measure":"EA","Description":"Some Description"}]';
ThisProductJSON:=S;
try
try
MultiObj.AddPair('ThisName','ThisValue');
MultiObj.AddPair('AnotherName','AnotherValue');
I:=0;
JSArr:=TJSONObject.ParseJSONValue(ThisProductJSON).AsType<TJSONARRAY>;
//Process all products in the array with a call to detailed Rest for each product
begin
for I := 0 to jsArr.count-1 do //iterate through the array
begin
Stockcode:= jsArr.Items[I].GetValue<string>('Thiscode');
ThisObject:=TJSONObject(jsArr[i]); // Make ThisObject Point to the jsArr[i] as an object so I can add the Other Values to this array element
OtherValues :=(GetOtherDetails(Stockcode)); // Call the below function to return the JSONVALUE containing addtional data
ThisObject.AddPair('OtherDetails',OtherValues); // Add all of the additional data in OtherValues JSONVALUE to this element of array via the object
end;
MultiObj.AddPair('WoWMultiProduct',JSARR); // This MultiObj hokds the consolidated data
end;
except
On E: Exception do
begin
//Errror handling here
end;
end;
finally
MultiObj.Free;
OtherValues.Free; // <<- I thought this would free the Result of the function but I think its leaking
JSARR.Free;
if assigned(ThisObject) then ThisObject.Free;
end;
end;
function TForm1.GetOtherDetails(Stockcode: String): TJSONVALUE;
Var
ThisResponseObject, MYRESULT : TJSONOBJECT;
vNIP, vCOO : TJSONVALUE;
begin
DetailedRESTRequest.execute; //<-- This REST Service Returns aditional JSON data
MyResult:=TJSONObject.Create;
if DetailedRESTResponse.StatusCode=200 then
begin
Try
Try
ThisResponseObject := TJSONObject.ParseJSONValue(DetailedRESTResponse.content)as TJSONObject; // Resd the entire JSON Response into TJSONObject
vCOO:=ThisResponseObject.Getvalue('COO'); // Get First JSON Value I need
vNIP:=ThisResponseObject.Getvalue('Nip'); // Get Next JSON Value I Need
MyResult.AddPair('NIPInfo',vNip); // Thiis is the only way I know how to Add the Values
MyResult.AddPair('COO',vCOO); // So that I can get them both together in the Result
Result:= TJSONObject.ParseJSONValue(MyResult.ToJSON)as TJSONValue; // Get back the JSONValue from the function
Except
On E:Exception do
begin
// Some Error Handling Here e.Classname+' ' +E.Message ;
end;
End;
Finally
If Assigned(ThisResponseObject) then ThisResponseObject.Free;
// NOTE IF I FREE MYRESULT OBJECT I GET ACCESS VIOLATION WHEN I TRY TO USE THE RESULT OF THE FUNCTION
End;
end;
end;
JSON objects are a tree-like data structure. When you request a value from a node, with getvalue (for example), it is actually giving you a reference to that node's object.
so, when you do the following in the "GetOtherDetails" function:
vCOO:=ThisResponseObject.Getvalue('COO');
MyResult.AddPair('COO',vCOO);
You make ThisResponseObject and MyResult share nodes (memory locations), so when you free one of them the other will try to free memory locations that no longer exist and generate the access violation
ThisResponseObject.free;
MyResult.Free; //access violation
Similarly, on "Function1" when doing:
ThisObject:=TJSONObject(jsArr[i]);
OtherValues :=(GetOtherDetails(Stockcode));
ThisObject.AddPair('OtherDetails',OtherValues);
You're making ThisObject contain OtherValues object... so when you try to free the two objects you're going to run into memory problems.

Why does TJSONObject.AddPair results Self?

I've noticed that TJSONObject.AddPair functions results Self instead of the newly created object:
For example, in System.JSON unit I see the following code:
function TJSONObject.AddPair(const Str: string; const Val: string): TJSONObject;
begin
if (not Str.IsEmpty) and (not Val.IsEmpty) then
AddPair(TJSONPair.Create(Str, Val));
Result := Self;
end;
I was expecting something like that:
function TJSONObject.AddPair(const Str: string; const Val: string): TJSONObject;
begin
if (not Str.IsEmpty) and (not Val.IsEmpty) then
Result := AddPair(TJSONPair.Create(Str, Val));
else
Result := nil;
end;
I find this very unusual, is it a Delphi XE7 bug or is there any technical/practical reason why they did that?
Returning Self is common coding pattern called fluent interface.
It allows you to continue with calls to the same object, creating chain of methods without the need to reference object variable for every call. That makes code more readable, on the other hand it is harder to debug.
var
Obj: TJSONObject;
begin
Obj := TJSONObject.Create
.AddPair('first', 'abc')
.AddPair('second', '123')
.AddPair('third', 'aaa');
...
end;
would be equivalent of
var
Obj: TJSONObject;
begin
Obj := TJSONObject.Create;
Obj.AddPair('first', 'abc');
Obj.AddPair('second', '123');
Obj.AddPair('third', 'aaa');
...
end;
And the generated JSON object will look like:
{
"first": "abc",
"second": "123",
"third": "aaa"
}
That kind of coding style is more prevalent in languages with automatic memory management, because you don't need to introduce intermediate variables.
For instance, if you need JSON string you would use following construct:
var
s: string;
begin
s := TJSONObject.Create
.AddPair('first', 'abc')
.AddPair('second', '123')
.AddPair('third', 'aaa')
.Format(2);
...
end;
The problem with the above code in Delphi is that it creates memory leak as you don't have ability to release intermediate object. Because of that, it is more common to use fluent interface pattern in combination with reference counted classes where automatic memory management will handle releasing of any intermediary object instances.

Why TJson.ObjectToJsonObject/ObjectToJsonString represents record fields as a JSON array?

There is an inconsistency of how SuperObject and TJson.ObjectToJsonObject represent certain parts of a class (i.e. Record fields). Let's have the following code snippet:
Uses rest.json, superobject;
type
TSimplePersonRec = record
FirstName: string;
LastName: string;
Age: byte;
end;
TSimplePerson = class
protected
FPersonRec: TSimplePersonRec;
public
property personRecord: TSimplePersonRec read FPersonRec write FPersonRec;
end;
// ...
{ Public declarations }
function toJson_SO(simplePerson: TSimplePerson): string;
function toJson_Delphi(simplePerson: TSimplePerson): string;
// ...
function TForm1.toJson_Delphi(simplePerson: TSimplePerson): string;
begin
result := tjson.Format(TJson.ObjectToJsonObject(simplePerson));
end;
function TForm1.toJson_SO(simplePerson: TSimplePerson): string;
var
so: ISuperObject;
ctx: TSuperRttiContext;
begin
ctx := TSuperRttiContext.Create;
try
so := ctx.AsJson<TSimplePerson>( simplePerson );
finally
ctx.Free;
end;
result := so.AsJSon(true, true);
end;
// ...
procedure TForm1.Button3Click(Sender: TObject);
var
spr: TSimplePersonRec;
sp: TSimplePerson;
begin
spr.FirstName := 'John';
spr.LastName := 'Doe';
spr.Age := 39;
sp := TSimplePerson.Create;
sp.personRecord := spr;
memo1.Lines.Add(#13'--- SuperObject ---'#13);
memo1.Lines.Add(toJson_SO(sp));
memo1.Lines.Add(#13'--- Delphi ---'#13);
memo1.Lines.Add(toJson_Delphi(sp));
end;
The OUTPUT is:
--- SuperObject ---
{
"FPersonRec": {
"LastName": "Doe",
"Age": 39,
"FirstName": "John"
}
}
--- Delphi ---
{
"personRec":
[
"John",
"Doe",
39
]
}
What's the reason for Delphi to represent records as JSON array? Is there a public standard or suggestion leading to this?
Note:
For me it's more natural to represent records with {key: value} notation instead of array. Not knowing the key name to witch the value belongs may have strange results during deserialization. For Example during deserialization I could pass a new class with the same layout, containing a record with different memory layout. In this case the values will be randomly assigned or an AV could occur?
UPDATE:
I'm using Delphi XE7.
Also I found this of json.org:
JSON is built on two structures:
A collection of name/value pairs. In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed
list, or associative array.
An ordered list of values. In most
languages, this is realized as an array, vector, list, or sequence.
So probably the question is more about is this a bug in TJson unit?
The Delphi output is legal JSON. Internally, REST.TJson is hard-coded to marshal a record as a JSON array. That is simply how it is implemented, it is by design, not a bug. It is just another way to represent data. SuperObject chooses to be more explicit. That is also OK. Two different implementations, two different representations. The JSON spec allows for both.

Crossplatform JSON Parsing

Good evening all.
I'm currently developing a cross-platform compatible version of my product WinFlare. The issue I'm facing is that SuperObject still isn't cross-platform compatible with Firemonkey. By all means, I used it in the original version of the product, but now I want to create a cross-platform version as opposed to one limited to just Windows, I'm finding it to be a hassle.
DBXJSON is the only cross-platform solution I've been able to find after extensive hours of research, but that's proving to be frustrating to try and deal with. Most all of the examples I've found for it either don't apply for my situation, or they're too complicated to gleam anything useful from. There's lots of discussion, but I'm just struggling to get to grips with what was such a simple task with SuperObject. I've spent the best part of this evening trying to find something that works to build from, but everything I've tried has just led me back to square one.
Ideally, I'd like to fix up SuperObject, but I lack the knowledge to go so in depth as to make it cross-platform compatible with OS X (and ready for the mobile studio). I'd welcome any suggestions on that, but as I imagine no one's got the time to go through such a huge task, it looks like DBXJSON is my only option.
The JSON layout I'm dealing with is still the same;
{
response: {
ips: [
{
ip: "xxx.xxx.xxx.xxx",
classification: "threat",
hits: xx,
latitude: xx,
longitude: xx,
zone_name: "domain-example1"
},
{
ip: "yyy.yyy.yyy.yyy",
classification: "robot",
hits: yy,
latitude: xx,
longitude: xx,
zone_name: "domain-example2"
}
]
}
result : "success",
msg: null
}
There can be hundreds of results in the ips array. Let's say I want to parse through all of the items in the array and extract every latitude value. Let's also assume for a second, I'm intending to output them to an array. Here's the sort of code template I'd like to use;
procedure ParseJsonArray_Latitude(SInput : String);
var
i : Integer;
JsonArray : TJsonArray;
Begin
// SInput is the retrieved JSON in string format
{ Extract Objects from array }
for i := 0 to JsonArray.Size-1 do
begin
Array_Latitude[i] := JsonArray.Item[i].ToString;
end;
end;
Essentially, where it says { Extract Objects from array }, I'd like the most basic solution using DBXJSON that would solve my problem. Obviously, the calls I've shown related to JsonArray in the template above might not be correct - they're merely there to serve as an aid.
First, parse the string to get an object.
var
obj: TJsonObject;
obj := TJsonObject.ParseJsonValue(SInput) as TJsonObject;
That gives you an object with three attributes, response, result, and msg. Although ParseJsonValue is a method of TJsonObject, and your particular string input happens to represent an object value, it can return instances of any TJsonValue descendant depending on what JSON text it's given. Knowing that's where to start is probably the hardest part of working with DbxJson.
Next, get the response attribute value.
response := obj.Get('response').JsonValue as TJsonObject;
That result should be another object, this time with one attribute, ips. Get that attribute, which should have an array for a value.
ips := response.Get('ips').JsonValue as TJsonArray;
Finally, you can get the values from the array. It looks like you're expecting the values to be numbers, so you can cast them that way.
for i := 0 to Pred(ips.Size) do
Array_Latitude[i] := (ips.Get(i) as TJsonObject).Get('latitude').JsonValue as TJsonNumber;
Remember to free obj, but not the other variables mentioned here, when you're finished.
For completion, since the question stated that there was no alternative to DBXJSON for cross-platform, I would like to point out two Open Source alternatives, which appeared since the initial question.
XSuperObject has an API very close to SuperObject, but is cross-platform;
Our SynCrossPlatformJSON.pas unit, which is lighter and much faster than both DBXJSON and XSuperObject.
SynCrossPlatformJSON is able to create schema-less objects or arrays, serialize and unserialize them as JSON, via a custom variant type, including late-binding to access the properties.
For your problem, you could write:
var doc: variant;
ips: PJSONVariantData; // direct access to the array
i: integer;
...
doc := JSONVariant(SInput); // parse JSON Input and fill doc custom variant type
if doc.response.result='Success' then // easy late-binding access
begin
ips := JSONVariantData(doc.response.ips); // late-binding access into array
SetLength(Arr_Lat,ips.Count);
for i := 0 to ips.Count-1 do begin
Arr_lat[i] := ips.Values[i].latitude;
Memo1.Lines.add(ips.Values[i].latitude);
end;
end;
... // (nothing to free, since we are using variants for storage)
Late-binding and variant storage allow pretty readable code.
Thanks to assistance from Rob Kennedy, I managed to build a solution that solved the problem;
var
obj, response, arrayobj : TJSONObject;
ips : TJSONArray;
JResult : TJsonValue;
i : Integer;
Arr_Lat : Array of string;
begin
try
Memo1.Lines.Clear;
obj := TJsonObject.ParseJSONValue(SInput) as TJSONObject;
response := Obj.Get('response').JsonValue as TJSONObject;
ips := response.Get('ips').JsonValue as TJSONArray;
SetLength(Arr_Lat, ips.Size-1);
for i := 0 to ips.Size-1 do
begin
arrayobj := ips.Get(i) as TJSONObject;
JResult := arrayobj.Get('latitude').JsonValue;
Arr_lat[i] := JResult.Value;
Memo1.Lines.Add(JResult.Value);
end;
finally
obj.Free;
end;
This will add the results to both the array (Arr_Lat), and output them to the memo (Memo1).

Custom marshaling TDictionary in Delphi

I need to custom marshal/unmarchal a TDictionary in Delphi (XE). The dictionary is declared as:
TMyRecord = record
key11: integer;
key12: string;
...
end;
TMyDict: TDictionary<string, TMyRecord>;
Now, if i marshal the dictionary without registering a custom converter, the marshaller will put all kind of fields in the JSON string - FOnValueNotify, FKeyCollection, FItems, etc.
What i need is some sort of associative array of associative arrays, i.e.
{"key1":{"key11":"val1","key12":"val2"},"key2":{"key11":"val3","key12":"val4"}}
Unfortunately, i don't know how to write the custom converter and reverter. I'm using Delphi XE and the built in TJSONMarshal and TJSONUnMarshal.
Note: The use of TDictionary for this task is not required. I just cant come with something better.
For a simple case like yours, I tend to use a custom method to represent my object in JSON. But, if you want to create reverter and converter, you should read this article:
http://www.danieleteti.it/?p=146
Another option is TSuperObject which has the ability to marshal to/from JSON using RTTI:
type
TData = record
str: string;
int: Integer;
bool: Boolean;
flt: Double;
end;
var
ctx: TSuperRttiContext;
data: TData;
obj: ISuperObject;
begin
ctx := TSuperRttiContext.Create;
try
data := ctx.AsType<TData>(SO('{str: "foo", int: 123, bool: true, flt: 1.23}'));
obj := ctx.AsJson<TData>(data);
finally
ctx.Free;
end;
end;