What happened to the JSON deserializer of Delphi Sydney (10.4.1)?
After the migration from Delphi Seattle to Sydney, the standard marshal has problems with the deserialization of simple records.
Here is an example and simplified representation of my problem:
Data structure - Interation 1:
TAnalysisAdditionalData=record {order important for marshaling}
ExampleData0:Real; {00}
ExampleData1:Real; {01}
ExampleData2:String; {02}
end;
JSON representation:
"AnalysisAdditionalData":[0,1,"ExampleString"]
Data structure - Interation x, 5 years later:
TAnalysisAdditionalData=record {order important for marshaling}
ExampleData0:Real; {00}
ExampleData1:Real; {01}
ExampleData2:String; {02}
ExampleData3:String; {03} {since version 2016-01-01}
ExampleData4:String; {04} {since version 2018-01-01}
ExampleData5:String; {05}
end;
JSON representation:
"AnalysisAdditionalData":[0,1,"ExampleString0","ExampleString1","ExampleString2","ExampleString3"]
After interation 1, three string fields have been added.
If I now confront the standard marshal of Delphi Sydney (no custom converter, reverter, etc.) with an old dataset, so concretely with the data "AnalysisAdditionalData":[0,1, "ExampleString"], Sydney throws an EArgumentOutOfBoundsException because the 3 strings are expected - the deserialization fails.
Exit point is in Data.DBXJSONReflect in method TJSONUnMarshal.JSONToTValue - location marked below:
function TJSONUnMarshal.JSONToTValue(JsonValue: TJSONValue;
rttiType: TRttiType): TValue;
var
tvArray: array of TValue;
Value: string;
I: Integer;
elementType: TRttiType;
Data: TValue;
recField: TRTTIField;
attrRev: TJSONInterceptor;
jsonFieldVal: TJSONValue;
ClassType: TClass;
Instance: Pointer;
begin
// null or nil returns empty
if (JsonValue = nil) or (JsonValue is TJSONNull) then
Exit(TValue.Empty);
// for each JSON value type
if JsonValue is TJSONNumber then
// get data "as is"
Value := TJSONNumber(JsonValue).ToString
else if JsonValue is TJSONString then
Value := TJSONString(JsonValue).Value
else if JsonValue is TJSONTrue then
Exit(True)
else if JsonValue is TJSONFalse then
Exit(False)
else if JsonValue is TJSONObject then
// object...
Exit(CreateObject(TJSONObject(JsonValue)))
else
begin
case rttiType.TypeKind of
TTypeKind.tkDynArray, TTypeKind.tkArray:
begin
// array
SetLength(tvArray, TJSONArray(JsonValue).Count);
if rttiType is TRttiArrayType then
elementType := TRttiArrayType(rttiType).elementType
else
elementType := TRttiDynamicArrayType(rttiType).elementType;
for I := 0 to Length(tvArray) - 1 do
tvArray[I] := JSONToTValue(TJSONArray(JsonValue).Items[I],
elementType);
Exit(TValue.FromArray(rttiType.Handle, tvArray));
end;
TTypeKind.tkRecord, TTypeKind.tkMRecord:
begin
TValue.Make(nil, rttiType.Handle, Data);
// match the fields with the array elements
I := 0;
for recField in rttiType.GetFields do
begin
Instance := Data.GetReferenceToRawData;
jsonFieldVal := TJSONArray(JsonValue).Items[I]; <<<--- Exception here (EArgumentOutOfBoundsException)
// check for type reverter
ClassType := nil;
if recField.FieldType.IsInstance then
ClassType := recField.FieldType.AsInstance.MetaclassType;
if (ClassType <> nil) then
begin
if HasReverter(ClassType, FIELD_ANY) then
RevertType(recField, Instance,
Reverter(ClassType, FIELD_ANY),
jsonFieldVal)
else
begin
attrRev := FieldTypeReverter(recField.FieldType);
if attrRev = nil then
attrRev := FieldReverter(recField);
if attrRev <> nil then
try
RevertType(recField, Instance, attrRev, jsonFieldVal)
finally
attrRev.Free
end
else
recField.SetValue(Instance, JSONToTValue(jsonFieldVal,
recField.FieldType));
end
end
else
recField.SetValue(Instance, JSONToTValue(jsonFieldVal,
recField.FieldType));
Inc(I);
end;
Exit(Data);
end;
end;
end;
// transform value string into TValue based on type info
Exit(StringToTValue(Value, rttiType.Handle));
end;
Of course, this may make sense for people who, for example, only work with Sydney, or at least with Delphi versions above Seattle, or have started with these versions. I, on the other hand, have only recently been able to make the transition from Seattle to Sydney (Update 1).
Delphi Seattle has no problems with the missing record fields. Why should it, when they can be left untouched as default? Absurdly, however, Sydney has no problems with excess data.
Is this a known Delphi Sydney bug? Can we expect a fix? Or can the problem be worked around in some other way, i.e. compiler directive, Data.DBXJSONReflect.TCustomAttribute, etc.? Or, is it possible to write a converter/reverter for records? If so, is there a useful guide or resource that explains how to do this?
I, for my part, have unfortunately not found any useful information in this regard, only many very poorly documented class descriptions.
Addendum: Yes, it looks like it is a Delphi bug, and in my opinion a very dangerous one. Luckily, and I'm just about to deploy a major release, I discovered the bug while testing after porting to Sydney. But that was only by chance, because I had to deal with old datasets. I could have easily overlooked the flaw.
You should check if your projects are also affected. For me, the problem is a neckbreaker right now.
I have just written a very simple test program for the Embarcadero support team. If you want, you can have a look at it and test if your code is also affected.
Below are the instructions and the code:
Create a new project.
Creates two buttons and a memo on the main form.
Assign the two OnClick events for the buttons for load and save accordingly
Runs the program and clicks the save button.
Opens the .TXT in the application directory and delete e.g. the last entry of the record.
Click the load button and an EArgumentOutOfBoundsException is thrown.
unit main;
interface
uses
System.SysUtils, System.Types, System.UITypes, System.Classes, System.Variants,
FMX.Types, FMX.Controls, FMX.Forms, FMX.Graphics, FMX.Dialogs,
FMX.Memo.Types, FMX.StdCtrls, FMX.Controls.Presentation, FMX.ScrollBox,
FMX.Memo;
type
TAnalysisAdditionalData=record {order important for marshaling}
ExampleData0:Real; {00}
ExampleData1:Real; {01}
ExampleData2:String; {02}
ExampleData3:String; {03} {since version 2016-01-01}
ExampleData4:String; {04} {since version 2018-01-01}
ExampleData5:String; {05}
end;
TSHCustomEntity=class(TPersistent)
private
protected
public
GUID:String;
end;
TSHAnalysis=class(TSHCustomEntity)
private
protected
public
AnalysisResult:String;
AnalysisAdditionalData:TAnalysisAdditionalData;
end;
TMainform = class(TForm)
Memo_Output: TMemo;
Button_Save: TButton;
Button_Load: TButton;
procedure Button_SaveClick(Sender: TObject);
procedure Button_LoadClick(Sender: TObject);
private
Analysis:TSHAnalysis;
procedure Marshal(Filename:String);
procedure Unmarshal(Filename:String);
function GetApplicationPath: String;
function GetFilename: String;
protected
procedure AfterConstruction;override;
public
Destructor Destroy;override;
property ApplicationPath:String read GetApplicationPath;
property Filename:String read GetFilename;
end;
var
Mainform: TMainform;
implementation
{$R *.fmx}
uses
DBXJSON,
DBXJSONReflect,
System.JSON;
{ TMainform }
procedure TMainform.AfterConstruction;
begin
inherited;
self.Analysis:=TSHAnalysis.Create;
self.Analysis.GUID:='6ed61388-cdd4-28dd-6efe-24461c4df3cd';
self.Analysis.AnalysisAdditionalData.ExampleData0:=0.5;
self.Analysis.AnalysisAdditionalData.ExampleData1:=0.9;
self.Analysis.AnalysisAdditionalData.ExampleData2:='ExampleString0';
self.Analysis.AnalysisAdditionalData.ExampleData3:='ExampleString1';
self.Analysis.AnalysisAdditionalData.ExampleData4:='ExampleString2';
self.Analysis.AnalysisAdditionalData.ExampleData5:='ExampleString3';
end;
destructor TMainform.Destroy;
begin
self.Analysis.free;
inherited;
end;
function TMainform.GetApplicationPath: String;
begin
RESULT:=IncludeTrailingPathDelimiter(ExtractFilePath(paramStr(0)));
end;
function TMainform.GetFilename: String;
begin
RESULT:=self.ApplicationPath+'6ed61388-cdd4-28dd-6efe-24461c4df3cd.txt';
end;
procedure TMainform.Button_SaveClick(Sender: TObject);
begin
self.Marshal(self.Filename);
end;
procedure TMainform.Button_LoadClick(Sender: TObject);
begin
if Analysis<>NIL then
FreeAndNil(Analysis);
self.Unmarshal(self.Filename);
self.Memo_Output.Text:=
self.Analysis.GUID+#13#10+
FloatToStr(self.Analysis.AnalysisAdditionalData.ExampleData0)+#13#10+
FloatToStr(self.Analysis.AnalysisAdditionalData.ExampleData1)+#13#10+
self.Analysis.AnalysisAdditionalData.ExampleData2+#13#10+
self.Analysis.AnalysisAdditionalData.ExampleData3+#13#10+
self.Analysis.AnalysisAdditionalData.ExampleData4+#13#10+
self.Analysis.AnalysisAdditionalData.ExampleData5;
end;
procedure TMainform.Marshal(Filename:String);
var
_Marshal:TJSONMarshal;
_Strings:TStringlist;
_Value:TJSONValue;
begin
_Strings:=TStringlist.Create;
try
_Marshal:=TJSONMarshal.Create;
try
_Value:=_Marshal.Marshal(Analysis);
_Strings.text:=_Value.ToString;
finally
if _Value<>NIL then
_Value.free;
_Marshal.free;
end;
_Strings.SaveToFile(Filename);
finally
_Strings.free;
end;
end;
procedure TMainform.Unmarshal(Filename:String);
var
_Strings:TStrings;
_UnMarshal:TJSONUnMarshal;
_Value:TJSONValue;
begin
if FileExists(Filename) then begin
_Strings:=TStringlist.create;
try
_Strings.LoadFromFile(Filename);
try
_Value:=TJSONObject.ParseJSONValue(_Strings.Text);
_UnMarshal:=TJSONUnMarshal.Create;
try
try
self.Analysis:=_UnMarshal.Unmarshal(_Value) as TSHAnalysis;
except
on e:Exception do
self.Memo_Output.text:=e.Message;
end;
finally
_UnMarshal.free;
end;
finally
if _Value<>NIL then
_Value.free;
end;
finally
_Strings.free;
end;
end;
end;
end.
To solve the problem temporarily, I have the following quick solution for you:
Make a copy of the standard library Data.DBXJSONReflect and name it e.g. Data.TempFix.DBXJSONReflect.
Change all includes/uses in your project accordingly.
After that navigate in Data.TempFix.DBXJSONReflect to line 2993:
jsonFieldVal := TJSONArray(JsonValue).Items[I];
And replace it with the following code:
try
jsonFieldVal := TJSONArray(JsonValue).Items[I];
except
on e:Exception do
if e is EArgumentOutOfRangeException then
continue
else
raise;
end;
After that the whole method should look like this:
function TJSONUnMarshal.JSONToTValue(JsonValue: TJSONValue; rttiType: TRttiType): TValue;
var
tvArray: array of TValue;
Value: string;
I: Integer;
elementType: TRttiType;
Data: TValue;
recField: TRTTIField;
attrRev: TJSONInterceptor;
jsonFieldVal: TJSONValue;
ClassType: TClass;
Instance: Pointer;
begin
// null or nil returns empty
if (JsonValue = nil) or (JsonValue is TJSONNull) then
Exit(TValue.Empty);
// for each JSON value type
if JsonValue is TJSONNumber then
// get data "as is"
Value := TJSONNumber(JsonValue).ToString
else if JsonValue is TJSONString then
Value := TJSONString(JsonValue).Value
else if JsonValue is TJSONTrue then
Exit(True)
else if JsonValue is TJSONFalse then
Exit(False)
else if JsonValue is TJSONObject then
// object...
Exit(CreateObject(TJSONObject(JsonValue)))
else
begin
case rttiType.TypeKind of
TTypeKind.tkDynArray, TTypeKind.tkArray:
begin
// array
SetLength(tvArray, TJSONArray(JsonValue).Count);
if rttiType is TRttiArrayType then
elementType := TRttiArrayType(rttiType).elementType
else
elementType := TRttiDynamicArrayType(rttiType).elementType;
for I := 0 to Length(tvArray) - 1 do
tvArray[I] := JSONToTValue(TJSONArray(JsonValue).Items[I],
elementType);
Exit(TValue.FromArray(rttiType.Handle, tvArray));
end;
TTypeKind.tkRecord, TTypeKind.tkMRecord:
begin
TValue.Make(nil, rttiType.Handle, Data);
// match the fields with the array elements
I := 0;
for recField in rttiType.GetFields do
begin
Instance := Data.GetReferenceToRawData;
try
jsonFieldVal := TJSONArray(JsonValue).Items[I];
except
on e:Exception do
if e is EArgumentOutOfRangeException then
continue
else
raise;
end;
// check for type reverter
ClassType := nil;
if recField.FieldType.IsInstance then
ClassType := recField.FieldType.AsInstance.MetaclassType;
if (ClassType <> nil) then
begin
if HasReverter(ClassType, FIELD_ANY) then
RevertType(recField, Instance,
Reverter(ClassType, FIELD_ANY),
jsonFieldVal)
else
begin
attrRev := FieldTypeReverter(recField.FieldType);
if attrRev = nil then
attrRev := FieldReverter(recField);
if attrRev <> nil then
try
RevertType(recField, Instance, attrRev, jsonFieldVal)
finally
attrRev.Free
end
else
recField.SetValue(Instance, JSONToTValue(jsonFieldVal,
recField.FieldType));
end
end
else
recField.SetValue(Instance, JSONToTValue(jsonFieldVal,
recField.FieldType));
Inc(I);
end;
Exit(Data);
end;
end;
end;
// transform value string into TValue based on type info
Exit(StringToTValue(Value, rttiType.Handle));
end;
Here's the case:
uses
System.JSON;
{$R *.dfm}
procedure TForm1.Button1Click(Sender: TObject);
var
dd:Double;
aJsonObj:TJSONObject;
begin
dd := 100.0;
aJsonObj := TJSONObject.Create(TJSONPair.Create('DecimalValue',TJSONNumber.Create(dd)));
ShowMessage(aJsonObj.ToString);
end;
which shows
{"DecimalValue":100}
What I need is
{"DecimalValue":100.0}
I've tried to change JsonFormatSettings but I couldn't find a way to add the fractional part.
One of TJSONNumber constructors accepts a raw string value that will be used in resulting JSON string without further processing. It allows you to pre-format the value yourself:
procedure TForm1.Button1Click(Sender: TObject);
var
dd: Double;
aJsonObj: TJSONObject;
begin
dd := 100.0;
aJsonObj := TJSONObject.Create(
TJSONPair.Create('DecimalValue', TJSONNumber.Create(FormatFloat('0.0###', dd, GetJSONFormat))));
try
ShowMessage(aJsonObj.ToString);
finally
aJsonObj.Free;
end;
end;
Using the above snippet the value shows as:
{"DecimalValue":100.0}
You can even create 'whatever' as number:
aJsonObj := TJSONObject.Create(TJSONPair.Create('DecimalValue', TJSONNumber.Create('whatever')));
to get this (invalid) JSON:
{"DecimalValue":whatever}
Side note: you are responsible for releasing root JSON objects you create as they have no parent object to manage their lifetime. Use try..finally to Free the root object when you're done with it as you can see in the first code snippet. Too bad that the documentation itself tempts you not to cleanup.
I'm attempting to import JSON from a TMemo to a TFDMemTable. Upon execution I get a "Exception EAccessViolation..." error. The line of code that appears to be causing the problem is FDMemTable1.FieldByName('userId').AsString := oProd.GetValue('PutRequest.Item.userId').Value; which is where I'm appending my first entry to the FDMemTable. I checked the JSON and it appears to be formatted correctly.
Here's my JSON array:
{"Jobs":[{"PutRequest":{"Item":{"userId":{"S":"1"},"WorkOrder":{"S":"29236"},"ServiceDate":{"S":"4/12/2019"}}}},{"PutRequest":{"Item":{"userId":{"S":"1"},"WorkOrder":{"S":"29237"},"ServiceDate":{"S":"4/12/2019"}}}}]}
Here's a snippet from my code:
uses
System.JSON;
procedure TForm1.FormCreate(Sender: TObject);
var
oJson: TJSONObject;
oArr: TJsonArray;
oPair: TJSONPair;
i: Integer;
oProd: TJSONObject;
begin
FDMemTable1.FieldDefs.Add('userId', ftString, 5);
FDMemTable1.FieldDefs.Add('WorkOrder', ftString, 5);
FDMemTable1.FieldDefs.Add('ServiceDate', ftString, 10);
FDMemTable1.Active := True;
oJson := TJSONObject.ParseJSONValue(TEncoding.ASCII.GetBytes(Memo1.Lines.Text), 0) as TJSONObject;
try
oArr := oJson.Get('Jobs').JsonValue as TJSONArray;
for i := 0 to oArr.Count - 1 do begin
oProd := oArr.Items[i] as TJSONObject;
FDMemTable1.Append;
FDMemTable1.FieldByName('userId').AsString := oProd.GetValue('PutRequest.Item.userId').Value;
FDMemTable1.FieldByName('WorkOrder').AsString := oProd.GetValue('PutRequest.Item.WorkOrder').Value;
FDMemTable1.FieldByName('ServiceDate').AsString := oProd.GetValue('PutRequest.Item.ServiceDate').Value;
FDMemTable1.Post;
end;
finally
oJson.Free;
end;
end;
I have checked the assignments and values of "oArr" and "oProd" and they appear correct. I'm guessing that I'm not addressing the JSON properly when I'm trying to assign the value to the FDMemTable. How do I resolve this?
You forgot to add
`FDMemTable1.CreateDataSet;`
under
FDMemTable1.FieldDefs.Add('ServiceDate', ftString, 10);
which means your dataSet Fields are not created yet and thus the call to
FDMemTable1.FieldByName('userId').AsString
will return nil and throw an EV when assigning to it
add that line and you are good to go
see this guide by Jim McKeeth (the code under the video) for proper way to use TMemTable and what it can do.
JSON-Document:
{"asks":
[["0.01129999",0.9997237],["0.01130000",594.26412711],["0.01130826",0.23004724],["0.01130999",0.25231793]],
"bids":
[["0.01129800",174.30861783],["0.01128867",0.13287659],["0.01128817",0.0097447],["0.01127559",241.94881004]],
"isFrozen":"0","seq":254407265
}
This is my first expirience of work with JSON-documents. I'm done that as I may. It work. But I think that it was wrong way. Because in my realization absent TJSONPair class, unlike many examples in net. Is there other variant of realisation comprising TJSONPair in my case?
const
ArrString : array[0..1] of String = ('asks','bids');
var
JSONObject: TJSONObject;
JSONArray: TJSONArray;
i,j: integer;
...
JSONObject:=TJSONObject.ParseJSONValue(JSONString) as TJSONObject;
if Assigned(JSONObject) then
begin
for i := 0 to High(ArrString) do
begin
JSONArray := JSONObject.Get(ArrString[i]).JsonValue as TJSONArray;
for j := 0 to JSONArray.Size-1 do
begin
if j = 0 then Memo1.Lines.Add(ArrString[i]+':');
Memo1.Lines.Add(TJSONString(TJSONArray(JSONArray.Get(j)).Get(0)).Value+ ' ' +TJSONNumber(TJSONArray(JSONArray.Get(j)).Get(1)).ToString);
end;
end;
I would advise you to ditch the inbuilt JSON classes, and instead use the excellent SuperObject classes from Henry Gourvest. They've been around longer, are better tested, more powerful, and easier to use.
With those, your code would become:
procedure ParseJSON(const JSONString: string);
const ArrString: array of string = ['asks', 'bids'];
var JSON, Element: ISuperObject;
CurrentString: string;
begin
JSON := SO(JSONString);
for CurrentString in ArrString do
begin
Memo1.Lines.Add(CurrentString + ':');
for Element in JSON[CurrentString] do
Memo1.Lines.Add(Element.AsArray.S[0] + ' ' + Element.AsArray.D[1].ToString);
end;
end;
I have a code that changes a value of a determinated pair in a existent JSON file and works perfectly. Now i need add one object and one pair to this file, using great part this same code. So how do this?
Thank you.
uses
System.Json, ShFolder, System.IOUtils;
...
function GetSpecialFolderPath(folder : integer) : string;
const
SHGFP_TYPE_CURRENT = 0;
var
path: array [0..MAX_PATH] of char;
begin
if SUCCEEDED(SHGetFolderPath(0,folder,0,SHGFP_TYPE_CURRENT,#path[0])) then
Result := path
else
Result := '';
end;
procedure ChangeChromeSetting(const ATarget, Avalue: string);
var
specialfolder: integer;
caminhochrome: String;
JSONObj, Obj: TJSONObject;
JSONPair: TJSONPair;
OldValue: string;
begin
specialFolder := CSIDL_LOCAL_APPDATA;
caminhochrome := GetSpecialFolderPath(specialFolder);
caminhochrome := caminhochrome + '\Google\Chrome\User Data\Local State';
if fileexists(caminhochrome) then
begin
Obj := TJSONObject.Create;
JSONObj := TJSONObject.ParseJSONValue(TFile.ReadAllText(caminhochrome)) as TJSONObject;
if not Assigned(JSONObj) then raise Exception.Create('Cannot read file: ' + caminhochrome);
try
OldValue := JSONObj.GetValue<string>(ATarget);
if not SameText(OldValue, Avalue) then
begin
JSONPair := JSONObj.Get(ATarget);
JSONPair.JsonValue.Free;
JSONPair.JsonValue := TJSONString.Create(Avalue);
///////////////////////////////////////////////////
Obj.AddPair('enabled', TJSONBool.Create(false)); // Trying add pair
JSONObj.AddPair('hardware_acceleration_mode', Obj); // Trying add object
//////////////////////////////////////////////////
TFile.WriteAllText(caminhochrome, JSONObj.ToJSON); // Don't add object and pair
end;
finally
JSONObj.Free;
end;
end;
end;
procedure TForm1.Button1Click(Sender: TObject);
begin
ChangeChromeSetting('hardware_acceleration_mode_previous', 'false');
end;
This is result that i'm waiting
"hardware_acceleration_mode":{"enabled":false}
Your code is somewhat confusing since you pass in some of the names as arguments, but then hard code others inside the function. Abstracting functionality is good practise but before you can abstract you really need to ensure the code works correctly. I'm going to show code that does not attempt to be abstract. Once you are satisfied it behaves as you need, then feel free to abstract away.
This code does what I believe is your intent:
var
root: TJSONObject;
value: TJSONObject;
prev: string;
begin
root := TJSONObject.ParseJSONValue(TFile.ReadAllText(FileName)) as TJSONObject;
try
prev := root.GetValue<string>('hardware_acceleration_mode_previous');
if not SameText(prev, 'false') then
begin
// remove existing value, if it exists
root.RemovePair('hardware_acceleration_mode').Free;
// create a new object, and initialise it
value := TJSONObject.Create;
value.AddPair('enabled', 'false');
// add the object at the root level
root.AddPair('hardware_acceleration_mode', value);
// save to file
TFile.WriteAllText(FileName, root.ToJSON);
end;
finally
root.Free;
end;
end;
Note that I have ensured that there are no memory leaks. I've used RemovePair to make sure that if there is an existing value named hardware_acceleration_mode it is first removed.