Using Delphi 10.4.1 on Windows and Mobile. I have a memory leak using this code in Windows. Can anyone point out what I'm doing wrong?
oResultJO := TJSONObject.Create;
try
oSecondJO := TJSONObject.Create;
oFirstJO := TJSONObject.Create;
oFirstJO.AddPair('TYPE_1', TJSONNumber.Create(1));
oFirstJO.AddPair('TYPE_1', TJSONNumber.Create(2));
oSecondJO.AddPair('TYPE_2', TJSONNumber.Create(3));
if (oFirstJO.Count > 0) then
oResultJO.AddPair('FIRST', oFirstJO)
else
oFirstJO.Free;
if (oSecondJO.Count > 0) then
begin
oResultJO.AddPair('SECOND', oSecondJO);
end else
oSecondJO.Free;
showmessage(oResultJO.ToString);
finally
oResultJO.Free;
end;
Related
i created a function in MariaDB which its body is:
BEGIN
SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'MY_ERROR_MESSAGE';
RETURN 1;
END
Then in Golang, I tried to select this function, which must be done with an error.
dbResult := db.QueryRow("SELECT `function1`()")
if dbResult.Err() != nil {
fmt.Printf("error (%s)", dbResult.Err().Error())
}
but the problem is that Err() is nil !!!!!!!!!! What is the problem !?
and fmt.Printf is not executed.
The funny part is that i created a procedure this time
BEGIN
SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'MY_ERROR_MESSAGE';
END
and i used the same golang code to call it
dbResult := db.QueryRow("CALL `procedure1`()")
if dbResult.Err() != nil {
fmt.Printf("error (%s)", dbResult.Err().Error())
}
but this time everything is OK !!! and .Err is not nil any more ! Why ???? What is the problem
I am having a device added to the rocket chip, it has its control & status registers and also an internal ram. To have the ability to access it with software I have added it into the regmap() in the next way :
val mem = Module(new SinglePortMemory(32,128, 7))
// init //
mem.io.re := false.B
mem.io.we := false.B
mem.io.addr := 0.U
mem.io.din := 0.S
val dout = RegNext(mem.io.dout)
regmap((
Seq(
0x0 -> Seq(RegField(1,enable,RegFieldDesc("enable","This bit enable my device")))
++ Seq.tabulate(128) {j =>
(0x50 + (j*4)) -> Seq(RegField(32,
RegReadFn(ready => {
when(ready) {
mem.io.addr := j.asUInt
mem.io.re := true.B
}
(true.B,dout.asUInt)
}),
RegWriteFn((valid,data) => {
when(valid) {
mem.io.addr := j.asUInt
mem.io.we := true.B
mem.io.din := data.asSInt
}
true.B
}),
RegFieldDesc(s"mem_${j}","")
))}
):_*)
Now this works , but I'm not sure if that is the best way to do that (I mean adding the memory to regmap). Can someone advise on a different / better way to do that ?
rocketchip has some built in memories for TileLink and AXI (AHB/APB I don't recall, but can use the others as an example and/or add converters)
https://github.com/chipsalliance/rocket-chip/blob/master/src/main/scala/tilelink/SRAM.scala
You can then connect the new TLRAM to a Xbar in addition to your normal memory mapped registers.
Notice:
Original post title
Why multithreaded JSON parser from DWScript does not scale with number of threads?
was changed because this problem is not related to processing JSON data with DWScript.
The problem is in default memory manager in Delphi XE2 to XE7 ( tested were XE2 and trial XE7 ), but problem appeared first in such type of application.
I have multithreaded Win32/Win64 vcl application which process JSON data in Delphi XE2.
Each thread parses JSON data using TdwsJSONValue.ParseString(sJSON) from DWScript, reads values using DWScript methods and stores result as records.
For testing purposes I process same JSON data in each thread.
Single thead run takes N seconds within thread to process data. Increasing number of threads to M lineary (approx. M * N) increases time within single thread necessary to process same data.
In result there is no speed improvment. Other parts of this applications ( JSON data delivery, storing results in target environment ) - scale as expected.
What could be a reason ? Any ideas appreciated.
Supplemental information:
Tested on Win7/32 and Win7/64, Win8/64 from 2-core to 12-core (w/w-out HT) systems
DWScript was choosen as fastest available (tested a bunch, among them: Superobject, build-in Delphi). SO behaves similar as JSON unit from DWS.
Below is complete console app illustrating the problem. To run it we need sample json data available here: https://www.dropbox.com/s/4iuv87ytpcdugk6/json1.zip?dl=0 This file contains data json1.dat for first thread. For threads up to 16 just copy json1.dat to json2.dat...json16.dat.
Program and data shoule be in the same folder. To run: convert.exe N, where N is number of threads.
Program writes time of execution in msecs to stout - spent in thread, time of parsing data and time of releasing (Destroy) TdwsJSONValue object.
Statement _dwsjvData.Destroy; does not scale.
program Convert;
{$APPTYPE CONSOLE}
{$R *.res}
uses
System.SysUtils,
System.Diagnostics,
System.Classes,
dwsJSON in 'dwsJSON.pas',
dwsStrings in 'dwsStrings.pas',
dwsUtils in 'dwsUtils.pas',
dwsXPlatform in 'dwsXPlatform.pas';
type
TWorkerThread = class (TThread)
private
_iUid: Integer;
_swWatch: TStopwatch;
_lRunning: Boolean;
_sFileJSonData: String;
_fJsonData: TextFile;
protected
constructor Create (AUid: Integer);
procedure Execute; override;
published
property Running: Boolean read _lRunning;
end;
TConverter = class (TObject)
private
_swWatch0, _swWatch1, _swWatch2: TStopwatch;
_dwsjvData: TdwsJSONValue;
protected
constructor Create;
destructor Destroy; override;
function Calculate (AUid: Integer; AJSonData: String; var AParse, ADestroy: Integer): Integer;
end;
const
MAX_THREADS = 16;
var
iHowMany: Integer;
athWorker: array [1..MAX_THREADS] of Pointer;
aiElapsed: array [1..MAX_THREADS] of Integer;
aiElapsedParse: array [1..MAX_THREADS] of Integer;
aiElapsedDestroy: array [1..MAX_THREADS] of Integer;
aiFares: array [1..MAX_THREADS] of Integer;
swWatchT, swWatchP: TStopwatch;
constructor TWorkerThread.Create (AUid: Integer);
begin
inherited Create (True);
_iUid := AUid;
_swWatch := TStopwatch.Create;
_sFileJSonData := ExtractFilePath (ParamStr (0)) + 'json' + Trim (IntToStr (_iUid)) + '.dat';
_lRunning := False;
Suspended := False;
end;
procedure TWorkerThread.Execute;
var
j: Integer;
sLine: String;
slLines: TStringList;
oS: TConverter;
begin
_lRunning := True;
oS := TConverter.Create;
slLines := TStringList.Create;
System.AssignFile (_fJsonData, _sFileJSonData);
System.Reset (_fJsonData);
j := 0;
repeat
System.Readln (_fJsonData, sLine);
slLines.Add (sLine);
Inc (j);
until (j = 50);
// until (System.Eof (_fJsonData));
System.Close (_fJsonData);
Sleep (1000);
_swWatch.Reset;
_swWatch.Start;
aiFares [_iUid] := 0;
aiElapsedParse [_iUid] := 0;
aiElapsedDestroy [_iUid] := 0;
for j := 1 to slLines.Count do
aiFares [_iUid] := aiFares [_iUid] + oS.Calculate (_iUid, slLines.Strings [j - 1], aiElapsedParse [_iUid], aiElapsedDestroy [_iUid]);
_swWatch.Stop;
slLines.Free;
os.Destroy;
aiElapsed [_iUid] := _swWatch.ElapsedMilliseconds;
_lRunning := False;
end;
constructor TConverter.Create;
begin
inherited Create;
_swWatch0 := TStopwatch.Create;
_swWatch1 := TStopwatch.Create;
_swWatch2 := TStopwatch.Create;
end;
destructor TConverter.Destroy;
begin
inherited;
end;
function TConverter.Calculate (AUid: Integer; AJSonData: String; var AParse, ADestroy: Integer): Integer;
var
jFare, jTotalFares, iElapsedParse, iElapsedDestroy, iElapsedTotal: Integer;
begin
_swWatch0.Reset;
_swWatch0.Start;
_swWatch1.Reset;
_swWatch1.Start;
_dwsjvData := TdwsJSONValue.ParseString (AJSonData);
_swWatch1.Stop;
iElapsedParse := _swWatch1.ElapsedMilliseconds;
if (_dwsjvData.ValueType = jvtArray) then
begin
_swWatch2.Reset;
_swWatch2.Start;
jTotalFares := _dwsjvData.ElementCount;
for jFare := 0 to (jTotalFares - 1) do
if (_dwsjvData.Elements [jFare].ValueType = jvtObject) then
begin
_swWatch1.Reset;
_swWatch1.Start;
_swWatch1.Stop;
end;
end;
_swWatch1.Reset;
_swWatch1.Start;
_dwsjvData.Destroy;
_swWatch1.Stop;
iElapsedDestroy := _swWatch1.ElapsedMilliseconds;
_swWatch0.Stop;
iElapsedTotal := _swWatch0.ElapsedMilliseconds;
Inc (AParse, iElapsedParse);
Inc (ADestroy, iElapsedDestroy);
result := jTotalFares;
end;
procedure MultithreadStart;
var
j: Integer;
begin
for j := 1 to iHowMany do
if (athWorker [j] = nil) then
begin
athWorker [j] := TWorkerThread.Create (j);
TWorkerThread (athWorker [j]).FreeOnTerminate := False;
TWorkerThread (athWorker [j]).Priority := tpNormal;
end;
end;
procedure MultithreadStop;
var
j: Integer;
begin
for j := 1 to MAX_THREADS do
if (athWorker [j] <> nil) then
begin
TWorkerThread (athWorker [j]).Terminate;
TWorkerThread (athWorker [j]).WaitFor;
TWorkerThread (athWorker [j]).Free;
athWorker [j] := nil;
end;
end;
procedure Prologue;
var
j: Integer;
begin
iHowMany := StrToInt (ParamStr (1));
for j := 1 to MAX_THREADS do
athWorker [j] := nil;
swWatchT := TStopwatch.Create;
swWatchT.Reset;
swWatchP := TStopwatch.Create;
swWatchP.Reset;
end;
procedure RunConvert;
function __IsRunning: Boolean;
var
j: Integer;
begin
result := False;
for j := 1 to MAX_THREADS do
result := result or ((athWorker [j] <> nil) and TWorkerThread (athWorker [j]).Running);
end;
begin
swWatchT.Start;
MultithreadStart;
Sleep (1000);
while (__isRunning) do
Sleep (500);
MultithreadStop;
swWatchT.Stop;
Writeln (#13#10, 'Total time:', swWatchT.ElapsedMilliseconds);
end;
procedure Epilogue;
var
j: Integer;
begin
for j := 1 to iHowMany do
Writeln ( #13#10, 'Thread # ', j, ' tot.time:', aiElapsed [j], ' fares:', aiFares [j], ' tot.parse:', aiElapsedParse [j], ' tot.destroy:', aiElapsedDestroy [j]);
Readln;
end;
begin
try
Prologue;
RunConvert;
Epilogue;
except
on E: Exception do
Writeln (E.ClassName, ': ', E.Message);
end;
end.
Have you tried my scaleable memory manager? Because Delphi (with fastmm internally) does not scale well with strings and other memory related stuff:
https://scalemm.googlecode.com/files/ScaleMM_v2_4_1.zip
And you could also try both profiler modes of my profiler to see which part is the bottleneck:
https://code.google.com/p/asmprofiler/
I did a (re)test of the FastCode MM Challenge, and the results were not that good for TBB (also out of memory exception in block downsize test).
In short: ScaleMM2 and Google TCmalloc are the fastest in this complex test, Fastmm and ScaleMM2 use the least memory.
Average Speed Performance: (Scaled so that the winner = 100%)
XE6 : 70,4
TCmalloc : 89,1
ScaleMem2 : 100,0
TBBMem : 77,8
Average Memory Performance: (Scaled so that the winner = 100%)
XE6 : 100,0
TCmalloc : 29,6
ScaleMem2 : 75,6
TBBMem : 38,4
FastCode Challenge: https://code.google.com/p/scalemm/source/browse/#svn%2Ftrunk%2FChallenge
TBB 4.3: https://www.threadingbuildingblocks.org/download
The solution is exchange default Delphi XE2 or XE7 memory manager with Intel® Threading Building Blocks memory manager. In example application it scales ca. lineary with number of threads up to 16 when app is 64 bits.
update: with assumption that number of threads running is less than number of cores
This was tested on machines from 2cores/4ht to 12cores/24ht running KVM virtualized Windows 7 with 124GB RAM
Interesting thing is virtualizing Win 7. memory allocation and deallocation is from 2 x faster as in native Win 7.
Conclusion: if you do a lot of memory allocation / deallocation operations of 10kB-10MB blocks in threads of multithreaded ( more than 4-8 threads) application - use only memory manager from Intel.
#André: thanks for tip pointing me to right direction!
Here is unit with TBB memory manager taken for tests, it has to appear as 1st on unit list in main project file .dpr
unit TBBMem;
interface
function ScalableGetMem (ASize: NativeInt): Pointer; cdecl; external 'tbbmalloc' name 'scalable_malloc';
procedure ScalableFreeMem (APtr: Pointer); cdecl; external 'tbbmalloc' name 'scalable_free';
function ScalableReAlloc (APtr: Pointer; Size: NativeInt): Pointer; cdecl; external 'tbbmalloc' name 'scalable_realloc';
implementation
Function TBBGetMem (ASize: Integer): Pointer;
begin
result := ScalableGetMem (ASize);
end;
Function TBBFreeMem (APtr: Pointer): Integer;
begin
ScalableFreeMem (APtr);
result := 0;
end;
Function TBBReAllocMem (APtr: Pointer; ASize: Integer): Pointer;
begin
result := ScalableRealloc (APtr, ASize);
end;
const
TBBMemoryManager: TMemoryManager = ( GetMem: TBBGetmem;
FreeMem: TBBFreeMem;
ReAllocMem: TBBReAllocMem; );
var
oldMemoryManager: TMemoryManager;
initialization
GetMemoryManager (oldMemoryManager);
SetMemoryManager (TBBMemoryManager);
finalization
SetMemoryManager (oldMemoryManager);
end.
I am trying to make a Android/Ios app that connects to MySQL through a DataSnap server.
I want to make this as a Thread. It works fine when I don't use a Thread.
In some articles it is mentioned that when using COM objects in a Thread it is importen to use CoInitialize and CoUninitialize. (But I don't get this to work)
Is this correct for FireMonkey app Android/Ios?
My Thread code:
Constructor TDMThread.Create(CreateSuspended: Boolean; ServerClassName, ProviderName:String; var ds:TclientDataset; n1:String=''; p1:String=''; n2:String=''; p2:String=''; n3:String='';p3:String='';n4:String='';p4:String='');
begin
Inherited Create(CreateSuspended);
FreeOnTerminate := False;
iServerClassName:=ServerClassName;
iProvName := ProviderName;
ip1 := p1;
in1 := n1;
ip2 := p2;
in2 := n2;
ip3 := p3;
in3 := n3;
ip4 := p4;
in4 := n4;
OutDS := ds;
end;
Destructor TDMThread.Destroy;
begin
inherited Destroy;
end;
procedure TDMThread.Execute;
var
par1,par2,par3,par4:Tparam;
begin
SQLConnection1 := TSQLConnection.Create(Nil);
SQLConnection1.DriverName := 'DataSnap';
SQLConnection1.Params.Values['HostName'] := 'localhost';
SQLConnection1.Params.Values['Port'] := '211';
SQLConnection1.Params.Values['DSAuthenticationPassword'] := '******';
SQLConnection1.Params.Values['DSAuthenticationUser'] := '*******';
SQLConnection1.Params.Values['DriverUnit'] := 'Data.DBXDataSnap';
SQLConnection1.Params.Values['CommunicationProtocol'] := 'tcp/ip';
SQLConnection1.Params.Values['DatasnapContext'] := 'datasnap/';
SQLConnection1.Params.Values['DriverAssemblyLoader'] := 'Borland.Data.TDBXClientDriverLoader,Borland.Data.DbxClientDriver,Version=19.0.0.0,Culture=neutral,PublicKeyToken=91d62ebb5b0d1b1b';
DSProviderConnection1:=TDSProviderConnection.Create(NIL);
DSProviderConnection1.SQLConnection := SQLConnection1;
DSProviderConnection1.ServerClassName := iServerClassName;
SQLConnection1.Connected:=True;
ClientDataSet1 := TClientDataSet.Create(Nil);
ClientDataSet1.RemoteServer := DSProviderConnection1;
ClientDataSet1.ProviderName := iProvName;
ClientDataSet1.Close;
ClientDataSet1.Open;
ClientDataset1.FindFirst;
OutDS.CloneCursor(ClientDataSet1,False,True);
// Some more code ...
end;
Somebody have any thoughts? Examples that works?
I have XE5.1 and working on a Windows 8.1.
Update information...
It is running now.
I have made this change at the end:
procedure TDMThread.Execute;
var
par1,par2,par3,par4:Tparam;
begin
SQLConnection1 := TSQLConnection.Create(Nil);
SQLConnection1.DriverName := 'DataSnap';
SQLConnection1.Params.Values['HostName'] := 'localhost';
SQLConnection1.Params.Values['Port'] := '211';
SQLConnection1.Params.Values['DSAuthenticationPassword'] := '******';
SQLConnection1.Params.Values['DSAuthenticationUser'] := '*******';
SQLConnection1.Params.Values['DriverUnit'] := 'Data.DBXDataSnap';
SQLConnection1.Params.Values['CommunicationProtocol'] := 'tcp/ip';
SQLConnection1.Params.Values['DatasnapContext'] := 'datasnap/';
SQLConnection1.Params.Values['DriverAssemblyLoader'] := 'Borland.Data.TDBXClientDriverLoader,Borland.Data.DbxClientDriver,Version=19.0.0.0,Culture=neutral,PublicKeyToken=91d62ebb5b0d1b1b';
DSProviderConnection1:=TDSProviderConnection.Create(NIL);
DSProviderConnection1.SQLConnection := SQLConnection1;
DSProviderConnection1.ServerClassName := iServerClassName;
SQLConnection1.Connected:=True;
ClientDataSet1 := TClientDataSet.Create(Nil);
ClientDataSet1.RemoteServer := DSProviderConnection1;
ClientDataSet1.ProviderName := iProvName;
ClientDataSet1.Close;
ClientDataSet1.Open;
ClientDataset1.FindFirst;
OutDS.CloneCursor(ClientDataSet1,False,True);
// This is new
while not terminated do
Begin
Sleep(100);
end;
//
// Some more code ...
end;
I found the solution here: XE5 Android TBitmap.LoadFromStream fail inside a thread
As you can see in:
XE5 Android TBitmap.LoadFromStream fail inside a thread
its bug in XE5 - the author's solution (sleep in loop) is not valid form of waiting for the thread...
dont use localhost as hostname. on mobile device it is wrong. you need to use actual IP in your local network of machine where server is running
I have a very good DirectMySQL unit, which is ready to be used and i want it to be a TDataset descendant so i can use it with QuickReport, i just want MySQL Query with DirectMySQL which descendant from TDataset.
Everything was ok until i tried to access a big table with 10.000 rows and more. It was unstable, the error was unpredictable and not always shown but it likely happened after you played with other tables.
It happened in GetFieldData(Field: TField; Buffer: Pointer): boolean; which used to get the field value from MySQL rows.
Here's the code,
function TMySQLQuery.GetFieldData(Field: TField; Buffer: Pointer): Boolean;
var
I, CT: Integer;
Row: TMySQL_Row;
TBuf: PChar;
FD: PMySQL_FieldDef;
begin
UpdateCursorPos; ------------> This code is after i got the error but no result
Resync([]); ------------> This code is after i got the error but no result
Result := false;
Row := oRecordset.CurrentRow;
I := Field.FieldNo-1;
FD := oRecordset.FieldDef(I);
if Not Assigned(FD) then
FD := oRecordset.FieldDef(I);
TBuf := PP(Row)[i];
Try
CT := MySQLWriteFieldData(fd.field_type, fd.length, fd.decimals, TBuf, PChar(Buffer));
Result := Buffer <> nil;
Finally
Row := nil; ------------> This code is after i got the error but no result
FD := nil; ------------> This code is after i got the error but no result
TBuf := nil; ------------> This code is after i got the error but no result
Buffer := nil; ------------> This code is after i got the error but no result
End;
end;
{
These codes below are to translate the data type
from MySQL Data type to a TDataset data type
and move mysql row (TBuf) to TDataset buffer to display.
And error always comes up from this function
when moving mysql row to buffer.
}
function TMySQLQuery.MySQLWriteFieldData(AType: byte;
ASize: Integer; ADec: cardinal; Source, Dest: PChar): Integer;
var
VI: Integer;
VF: Double;
VD: TDateTime;
begin
Result := MySQLDataSize(AType, ASize, ADec);
case AType of
FIELD_TYPE_TINY, FIELD_TYPE_SHORT, FIELD_TYPE_LONG, FIELD_TYPE_LONGLONG,
FIELD_TYPE_INT24:
begin
if Source <> '' then
VI := StrToInt(Source)
else
VI := 0;
Move(VI, Dest^, Result);
end;
FIELD_TYPE_DECIMAL, FIELD_TYPE_NEWDECIMAL:
begin
if source <> '' then
VF := internalStrToCurr(Source)
else
VF := 0;
Move(VF, Dest^, Result);
end;
FIELD_TYPE_FLOAT, FIELD_TYPE_DOUBLE:
begin
if Source <> '' then
VF := InternalStrToFloat(Source)
else
VF := 0;
Move(VF, Dest^, Result);
end;
FIELD_TYPE_TIMESTAMP:
begin
if Source <> '' then
VD := InternalStrToTimeStamp(Source)
else
VD := 0;
Move(VD, Dest^, Result);
end;
FIELD_TYPE_DATETIME:
begin
if Source <> '' then
VD := InternalStrToDateTime(Source)
else
VD := 0;
Move(VD, Dest^, Result);
end;
FIELD_TYPE_DATE:
begin
if Source <> '' then
VD := InternalStrToDate(Source)
else
VD := 0;
Move(VD, Dest^, Result);
end;
FIELD_TYPE_TIME:
begin
if Source <> '' then
VD := InternalStrToTime(Source)
else
VD := 0;
Move(VD, Dest^, Result);
end;
FIELD_TYPE_STRING, FIELD_TYPE_VAR_STRING,
FIELD_TYPE_ENUM, FIELD_TYPE_SET:
begin
if Source = nil then
Dest^ := #0
else
Move(Source^, Dest^, Result);
end;
Else
Result := 0;
Raise EMySQLError.Create( 'Write field data - Unknown type field' );
end;
end;
My guess for now is it's memory related problem.
I am stacked. Anyone could help?
I also need TDataset documentation which list availlable descendant function and how to use it, or how to descendant from TDataset. anyone have them? I am lack of this kind of doumentation.
GetFieldData cannot have UpdateCursorPos and Resync calls. Otherwise you may get unpredicatable errors.
FD := oRecordset.FieldDef(I) ... FD := oRecordset.FieldDef(I); - looks strange. Second assigment is not needed.
finally ... end with local variables reset is not needed.
I have no idea what returns MySQLDataSize. For example, MySQLDataSize may return size in Delphi data type representation units, or may return length of data returned by MySQL. But depending on that MySQLWriteFieldData may be correct or may be not.
I dont know how DirectMySQL works. If it uses raw TCP/IP to talk to MySQL, then the problem may be there. For example, it incorrectly handles a sequence of packets.
And finally - what are the errors you are getting ? What is your Delphi version ? What is your MySQL client and server versions ?
And so on ....
IOW, that will be really hard to say, what is wrong. To do so, I for example, will need to get all sources, sit at Delphi IDE debugger and analyze many details of what is going on - sorry, no time :)
It's solved now by adding #0 at the end of the line... Thanks so much to all who replied to my problem.