Crossplatform JSON Parsing - json

Good evening all.
I'm currently developing a cross-platform compatible version of my product WinFlare. The issue I'm facing is that SuperObject still isn't cross-platform compatible with Firemonkey. By all means, I used it in the original version of the product, but now I want to create a cross-platform version as opposed to one limited to just Windows, I'm finding it to be a hassle.
DBXJSON is the only cross-platform solution I've been able to find after extensive hours of research, but that's proving to be frustrating to try and deal with. Most all of the examples I've found for it either don't apply for my situation, or they're too complicated to gleam anything useful from. There's lots of discussion, but I'm just struggling to get to grips with what was such a simple task with SuperObject. I've spent the best part of this evening trying to find something that works to build from, but everything I've tried has just led me back to square one.
Ideally, I'd like to fix up SuperObject, but I lack the knowledge to go so in depth as to make it cross-platform compatible with OS X (and ready for the mobile studio). I'd welcome any suggestions on that, but as I imagine no one's got the time to go through such a huge task, it looks like DBXJSON is my only option.
The JSON layout I'm dealing with is still the same;
{
response: {
ips: [
{
ip: "xxx.xxx.xxx.xxx",
classification: "threat",
hits: xx,
latitude: xx,
longitude: xx,
zone_name: "domain-example1"
},
{
ip: "yyy.yyy.yyy.yyy",
classification: "robot",
hits: yy,
latitude: xx,
longitude: xx,
zone_name: "domain-example2"
}
]
}
result : "success",
msg: null
}
There can be hundreds of results in the ips array. Let's say I want to parse through all of the items in the array and extract every latitude value. Let's also assume for a second, I'm intending to output them to an array. Here's the sort of code template I'd like to use;
procedure ParseJsonArray_Latitude(SInput : String);
var
i : Integer;
JsonArray : TJsonArray;
Begin
// SInput is the retrieved JSON in string format
{ Extract Objects from array }
for i := 0 to JsonArray.Size-1 do
begin
Array_Latitude[i] := JsonArray.Item[i].ToString;
end;
end;
Essentially, where it says { Extract Objects from array }, I'd like the most basic solution using DBXJSON that would solve my problem. Obviously, the calls I've shown related to JsonArray in the template above might not be correct - they're merely there to serve as an aid.

First, parse the string to get an object.
var
obj: TJsonObject;
obj := TJsonObject.ParseJsonValue(SInput) as TJsonObject;
That gives you an object with three attributes, response, result, and msg. Although ParseJsonValue is a method of TJsonObject, and your particular string input happens to represent an object value, it can return instances of any TJsonValue descendant depending on what JSON text it's given. Knowing that's where to start is probably the hardest part of working with DbxJson.
Next, get the response attribute value.
response := obj.Get('response').JsonValue as TJsonObject;
That result should be another object, this time with one attribute, ips. Get that attribute, which should have an array for a value.
ips := response.Get('ips').JsonValue as TJsonArray;
Finally, you can get the values from the array. It looks like you're expecting the values to be numbers, so you can cast them that way.
for i := 0 to Pred(ips.Size) do
Array_Latitude[i] := (ips.Get(i) as TJsonObject).Get('latitude').JsonValue as TJsonNumber;
Remember to free obj, but not the other variables mentioned here, when you're finished.

For completion, since the question stated that there was no alternative to DBXJSON for cross-platform, I would like to point out two Open Source alternatives, which appeared since the initial question.
XSuperObject has an API very close to SuperObject, but is cross-platform;
Our SynCrossPlatformJSON.pas unit, which is lighter and much faster than both DBXJSON and XSuperObject.
SynCrossPlatformJSON is able to create schema-less objects or arrays, serialize and unserialize them as JSON, via a custom variant type, including late-binding to access the properties.
For your problem, you could write:
var doc: variant;
ips: PJSONVariantData; // direct access to the array
i: integer;
...
doc := JSONVariant(SInput); // parse JSON Input and fill doc custom variant type
if doc.response.result='Success' then // easy late-binding access
begin
ips := JSONVariantData(doc.response.ips); // late-binding access into array
SetLength(Arr_Lat,ips.Count);
for i := 0 to ips.Count-1 do begin
Arr_lat[i] := ips.Values[i].latitude;
Memo1.Lines.add(ips.Values[i].latitude);
end;
end;
... // (nothing to free, since we are using variants for storage)
Late-binding and variant storage allow pretty readable code.

Thanks to assistance from Rob Kennedy, I managed to build a solution that solved the problem;
var
obj, response, arrayobj : TJSONObject;
ips : TJSONArray;
JResult : TJsonValue;
i : Integer;
Arr_Lat : Array of string;
begin
try
Memo1.Lines.Clear;
obj := TJsonObject.ParseJSONValue(SInput) as TJSONObject;
response := Obj.Get('response').JsonValue as TJSONObject;
ips := response.Get('ips').JsonValue as TJSONArray;
SetLength(Arr_Lat, ips.Size-1);
for i := 0 to ips.Size-1 do
begin
arrayobj := ips.Get(i) as TJSONObject;
JResult := arrayobj.Get('latitude').JsonValue;
Arr_lat[i] := JResult.Value;
Memo1.Lines.Add(JResult.Value);
end;
finally
obj.Free;
end;
This will add the results to both the array (Arr_Lat), and output them to the memo (Memo1).

Related

Why does TJSONObject.AddPair results Self?

I've noticed that TJSONObject.AddPair functions results Self instead of the newly created object:
For example, in System.JSON unit I see the following code:
function TJSONObject.AddPair(const Str: string; const Val: string): TJSONObject;
begin
if (not Str.IsEmpty) and (not Val.IsEmpty) then
AddPair(TJSONPair.Create(Str, Val));
Result := Self;
end;
I was expecting something like that:
function TJSONObject.AddPair(const Str: string; const Val: string): TJSONObject;
begin
if (not Str.IsEmpty) and (not Val.IsEmpty) then
Result := AddPair(TJSONPair.Create(Str, Val));
else
Result := nil;
end;
I find this very unusual, is it a Delphi XE7 bug or is there any technical/practical reason why they did that?
Returning Self is common coding pattern called fluent interface.
It allows you to continue with calls to the same object, creating chain of methods without the need to reference object variable for every call. That makes code more readable, on the other hand it is harder to debug.
var
Obj: TJSONObject;
begin
Obj := TJSONObject.Create
.AddPair('first', 'abc')
.AddPair('second', '123')
.AddPair('third', 'aaa');
...
end;
would be equivalent of
var
Obj: TJSONObject;
begin
Obj := TJSONObject.Create;
Obj.AddPair('first', 'abc');
Obj.AddPair('second', '123');
Obj.AddPair('third', 'aaa');
...
end;
And the generated JSON object will look like:
{
"first": "abc",
"second": "123",
"third": "aaa"
}
That kind of coding style is more prevalent in languages with automatic memory management, because you don't need to introduce intermediate variables.
For instance, if you need JSON string you would use following construct:
var
s: string;
begin
s := TJSONObject.Create
.AddPair('first', 'abc')
.AddPair('second', '123')
.AddPair('third', 'aaa')
.Format(2);
...
end;
The problem with the above code in Delphi is that it creates memory leak as you don't have ability to release intermediate object. Because of that, it is more common to use fluent interface pattern in combination with reference counted classes where automatic memory management will handle releasing of any intermediary object instances.

JSON with SuperObject: is element an array or an object?

I get JSON from API and it have a quirk: usually it returns "tags" element as object {"x":"y"}, but if ther are no tags, it returns empty array [] instead.
I parse JSON with SuperObject, and use this code:
var
JsonObject: ISuperObject;
item: TSuperAvlEntry;
temp: TStringList;
begin
{...}
for item in JsonObject.O['tags'].AsObject do
begin
temp.Add(item.Name);
end;
{...}
It works wonderfully for objects, but it crashes with Access Violation error if it's an array.
As well, if I try something like:
if JSONObject['tags'].AsArray.Length=0 then
it works fine for empty array, but crashes if it is an object.
I don't know for sure that elements may be in "tags" and thus don't know how can I use Exists() in this case.
Any ideas?
Well, looks like I found the answer myself, so I will share it.
ISuperObject has a property "DataType" which you can check, like this:
if JsonObject['tags'].DataType = stObject then
begin
for item in JsonObject.O['tags'].AsObject do
begin
temp.Add(item.Name);
end;
end;
stObject and stArray are most useful to check, but there's also: stBoolean, stDouble, stCurrency, stInt and stMethod.

Does the ulkJSON library have limitations when dealing with base64 in Delphi 7?

I'm working on a project that is using Delphi 7 to consume RESTful services. We are creating and decoding JSON with the ulkJSON library. Up to this point I've been able to successfully build and send JSON containing a base64 string that exceed 5,160kb. I can verify that the base64 is being received by the services and verify the integrity of the base64 once its there. In addition to sending, I can also receive and successfully decode JSON with a smaller (~ 256KB or less) base64.
However I am experiencing some issues on the return trip when larger (~1,024KB+) base64 is involved for some reason. Specifically when attempting to use the following JSON format and function combination:
JSON:
{
"message" : "/9j/4AAQSkZJRgABAQEAYABgAAD...."
}
Function:
function checkResults(JSONFormattedString: String): String;
var
jsonObject : TlkJSONObject;
iteration : Integer;
i : Integer;
x : Integer;
begin
jsonObject := TlkJSONobject.Create;
// Validate that the JSONFormatted string is not empty.
// If it is empty, inform the user/programmer, and exit from this routine.
if JSONFormattedString = '' then
begin
result := 'Error: JSON returned is Null';
jsonObject.Free;
exit;
end;
// Now that we can validate that this string is not empty, we are going to
// assume that the string is a JSONFormatted string and attempt to parse it.
//
// If the string is not a valid JSON object (such as an http status code)
// throw an exception informing the user/programmer that an unexpected value
// has been passed. And exit from this routine.
try
jsonObject := TlkJSON.ParseText(JSONFormattedString) as TlkJSONobject;
except
on e:Exception do
begin
result := 'Error: No JSON was received from web services';
jsonObject.Free;
exit;
end;
end;
// Now that the object has been parsed, lets check the contents.
try
result := jsonObject.Field['message'].value;
jsonObject.Free;
exit;
except
on e:Exception do
begin
result := 'Error: No Message received from Web Services '+e.message;
jsonObject.Free;
exit;
end;
end;
end;
As mentioned above when using the above function, I am able to get small (256KB and less) base64 strings out of the 'message' field of a JSON object. But for some reason if the received JSON is larger than say 1,024kb the following line seems to just stop in its tracks:
jsonObject := TlkJSON.ParseText(JSONFormattedString) as TlkJSONobject;
No errors, no results. Following the debugger, I can go into the library, and see that the JSON string being passed is not considered to be JSON despite being in the format listed above. The only difference I can find between calls that work as expected and calls that do not work as expect appears to be the size of base64 being transmitted.
Am I missing something completely obvious and should be shot for my code implementation (very possible)? Have I missed some notation regarding the limitations of the ulkJSON library? Any input would be extremely helpful. Thanks in advance stack!
So after investigating this for hours over the course of some time, I did discover that the library indeed was working properly and there was no issue.
The issue came down to the performance of my machine as it was taking on average 215802 milliseconds (3.5967 minutes) to process a moderately sized image (1.2 meg) in base64 format. This performance scaled according to the size of the base64 string (faster for smaller, longer for larger).

Parsing JSON into TListBox

Good evening guys!
I'm currently trying to put together a CloudFlare client for the desktop. I've connected to their API and successfully retrieved the JSON results with a POST request (the results of which have been output into a TMemo). I'm now wanting to parse these results into a TListBox (see bolded area for example). The project is being designed in Firemonkey.
Here's the formatted layout of the response with some example content;
{
- response: {
|- ips: [
|- {
ip: "xxx.xxx.xxx.xxx",
classification: "threat",
hits: xx,
latitude: null,
longitude: null,
zone_name: "domain-example1"
},
- {
ip: "yyy.yyy.yyy.yyy",
classification: "robot",
hits: yy,
latitude: null,
longitude: null,
zone_name: "domain-example2"
}
]
}
result : "success",
msg: null
}
I've tried several different components - SuperObject, Paweł Głowacki's JSON Designtime Parser, Tiny-JSON, LKJSON and the built in DBXJSON. However, i've no experience with JSON at all and i can't seem to find the most basic of examples that i can get started from. Many of them show sample data, but all the ones i've tried don't seem to work as i'd expect, most likely because i'm misunderstanding them. I'd assume the components work, so i need guidance on getting started.
There are hundreds, often thousands, of results in the ips "array" (i apologise if that's not correct, i'd assume it's known as an array but again, i'm completely new to JSON).
What i'm really looking for is some sort of extremely basic sample code which i can build from (along with what component it uses for parsing and such).
For example, if i wanted to grab every ip from the JSON results, and put each one as a separate item into a TListBox (using TListBox.add method), how would i go about achieving this?
When i say ip, i mean the value (in the formatted layout above, this would be xxx.xxx.xxx.xxx or yyy.yyy.yyy.yyy).
Additionally, if i wanted to find a "record" (?) by it's IP from the JSON results and output the data to a delphi array - e.g.;
Result : Array of String = ['"xxx.xxx.xxx.xxx"','"threat"','xx','null','null','"domain-example1"'];
is that possible with JSON? (If this is seen as a separate question or too unrelated, please feel free to edit it out rather than close the question as a whole).
The closest i got to this had not only the ip's, but every other piece of data in a seperate TListItem (i.e. response, ips, ip, classification, xxx.xxx.xxx.xxx and everything else had it's own item, along with several empty items in between each non-empty item).
I'm sure it's extremely simple to do, but there's so much information on JSON that it's a little overwhelming for people new to the format.
Best Regards,
Scott Pritchard.
JSON is very simple and easy to figure out, once you understand the basic concepts. Have a look at http://json.org, where it explains things.
There are 4 basic concepts in JSON:
A value is any JSON element: a basic string or number, an array, or an object. (Anything but a pair.)
An array should be a familiar concept: an ordered list of values. The main difference from Delphi arrays is that JSON arrays don't have a defined type for the elements; they're simply "an array of JSON values."
A pair is a key-value pair. The key can be a string or a number, and the value can be any JSON value.
An object is an associative map of JSON pairs. You can think of it conceptually as a TDictionary<string, JSON value>.
So if I wanted to take a JSON array of data like that, and put it in a TListBox, I'd do something like this (DBXJSON example, warning: not tested):
procedure TMyForm.LoadListBox(response: TJSONObject);
var
i: integer;
ips: TJSONArray;
ip: TJSONObject;
pair: TJSONPair;
begin
ListBox.Clear;
pair := response.Get('ips');
if pair = nil then
Exit;
ips := pair.value as TJSONArray;
for i := 0 to ips.size - 1 do
begin
ip := ips.Get(i) as TJSONObject;
pair := ip.Get('ip');
if pair = nil then
ListBox.AddItem('???', ip.Clone)
else ListBox.AddItem(pair.JsonString, ip.Clone);
end;
end;
Then you have a list of IP addresses, and associated objects containing the full record that you can get at if the user selects one. (If you wanted to put the entire contents of each record into the list control, have a look at TListView. It works better than TListBox for that.)
And if you want to build an array of strings containing all the values, do something like this:
function JsonObjToStringArray(obj: TJsonObject): TArray<string>;
var
i: integer;
begin
SetLength(result, obj.Size);
for i := 0 to obj.Size - 1 do
result[i] := obj.Get(i).JsonValue.ToString;
end;
This is all just sample code, of course, but it should give you something to build on.
EDIT2: AV Fixed with extreme ease.
EDIT: After further examining my own code, i realised it would cause a massive amount of memory leaks. However, i have since switched over to SuperObject and found the same result can be achieved in 2 lines of code with only 2 variables and no memory leaks;
Procedure ParseIPs;
ISO : ISuperObject;
MyItem : ISuperObject;
begin
ISO := SO(RetrievedJSON);
for MyItem in ISO['response.ips'] do Memo2.Lines.Add(MyItem.S['ip']);
end;
RetrievedJSON is simply a string containing the unparsed, plaintext JSON (i.e. not a JSONString but an actual string).
I've left the original code underneath for sake of continuity.
With assistance from Mason Wheeler in an earlier answer, as well as an answer provided by "teran" on question 9608794, i successfully built the following to parse down to the actual level (i.e. the "array" containing the data) i needed to access, and then output all items with a specific JSONString.Value into a listbox (named LB1 in the sample below);
Procedure ParseIP;
var
o, Jso, OriginalObject : TJSONObject;
ThePair, JsPair : TJSONPair;
TheVal, jsv : TJSONValue;
jsArr : TJsonArray;
StrL1 : String;
i, num : Integer;
begin
num := 0;
o := TJSONObject.ParseJSONValue(TEncoding.ASCII.GetBytes(Memo1.Text), 0) as TJSONObject;
ThePair := o.Get('response');
TheVal := ThePair.JsonValue;
STRL1 := TheVal.ToString;
JSV := TJSONObject.ParseJSONValue(STRL1);
OriginalObject := JSV as TJSONObject;
JSPair := OriginalObject.Get('ips');
JSARR := JSPair.JsonValue as TJSONArray;
for i := 0 to JsArr.Size-1 do
begin
JSO := JSArr.Get(i) as TJSONObject;
for JSPAIR in JSO do
begin
num := num+1;
if JSPAIR.JsonString.Value = 'ip' then
begin
LB1.Items.Add(JSPair.JsonValue.Value);
end
else null;
end;
end;
ShowMessage('Items in listbox: ' + IntToStr(LB1.Items.Count));
ShowMessage('Items in JSON: ' + IntToStr(num div JSO.Size));
Jsv.Free;
end;
While this is an extremely round-about way of doing it, it allows me to look at each individual step, and see where it's iterating down through the JSON and with extreme ease, and to change it into a function where i can output any piece or range of data as a result based on one of multiple criteria. For the sake of verifying i got the correct number of items, i added 2 ShowMessage routines at the end; One for the items in the listbox, and one for the number of instances of "ip" data that i was parsing.
This code was specifically tested in Firemonkey with CloudFlare API JSON results which were output into a TMemo exactly as they were retrieved (on an &calls_left&a=zone_ips&class=t&geo=1 API call, of course with your zone, token and email appended in addition). It should be relatively easy to modify it to work with other results from the numerous other API calls too.
To clarify, i did try Mason's code, but unfortunately i couldn't get it working. However, i have accepted his answer for the time being on the basis that the explanation he gave on the basics was worthy of it and assisted me in getting to an end-solution and coming up with something i can build from and teach myself.

Custom marshaling TDictionary in Delphi

I need to custom marshal/unmarchal a TDictionary in Delphi (XE). The dictionary is declared as:
TMyRecord = record
key11: integer;
key12: string;
...
end;
TMyDict: TDictionary<string, TMyRecord>;
Now, if i marshal the dictionary without registering a custom converter, the marshaller will put all kind of fields in the JSON string - FOnValueNotify, FKeyCollection, FItems, etc.
What i need is some sort of associative array of associative arrays, i.e.
{"key1":{"key11":"val1","key12":"val2"},"key2":{"key11":"val3","key12":"val4"}}
Unfortunately, i don't know how to write the custom converter and reverter. I'm using Delphi XE and the built in TJSONMarshal and TJSONUnMarshal.
Note: The use of TDictionary for this task is not required. I just cant come with something better.
For a simple case like yours, I tend to use a custom method to represent my object in JSON. But, if you want to create reverter and converter, you should read this article:
http://www.danieleteti.it/?p=146
Another option is TSuperObject which has the ability to marshal to/from JSON using RTTI:
type
TData = record
str: string;
int: Integer;
bool: Boolean;
flt: Double;
end;
var
ctx: TSuperRttiContext;
data: TData;
obj: ISuperObject;
begin
ctx := TSuperRttiContext.Create;
try
data := ctx.AsType<TData>(SO('{str: "foo", int: 123, bool: true, flt: 1.23}'));
obj := ctx.AsJson<TData>(data);
finally
ctx.Free;
end;
end;