I have a database that I need to query over and over as fast as possible. My queries execute pretty quickly, but there seems to be some additional lag.
I have a feeling that this lag is due to the fact that I am initiating and de-initiating a connection the connection each time. Is there a way to avoid this?
I am not using libmysql (at least, not directly). I am using the "mysql50" package in Lazarus/FreePascal (similar to delphi), which in turn uses libmysql ( I think ).
I would really appreciate if someone took a look at my code and pointed out (or maybe even fixed ) some inefficiencies.
The purpose of this library is to pass along a query sent from MQL4 (a propitiatory C-like language for the financial exchange market), and return a single row from my MYSQL database (to which it connects through a pipe).
{$CALLING STDCALL}
library D1Query;
{$mode objfpc}{$H+}
uses
cmem,
Windows,
SysUtils,
profs_win32exceptiontrap,
mysql50;
var
sock: PMYSQL;
qmysql: st_mysql;
type
VArray = array[0..100] of Double;
PArray = ^VArray;
procedure InitSQL; stdcall;
begin
mysql_init(PMySQL(#qmysql));
sock :=
mysql_real_connect(PMysql(#qmysql), '.', 'root', 'password', 'data', 3306, 'mysql', CLIENT_MULTI_STATEMENTS);
if sock = nil then
begin
OutputDebugString(PChar(' Couldn''t connect to MySQL.'));
OutputDebugString(PChar(mysql_error(#qmysql)));
halt(1);
end;
end;
procedure DeInitSQL; stdcall;
begin
mysql_close(sock);
end;
function SQL_Query(QRY: PChar; output: PArray): integer; stdcall;
var
rowbuf: MYSQL_ROW;
recbuf: PMYSQL_RES;
i: integer;
nfields: LongWord;
begin
InitSQL();
if (mysql_query(sock, QRY) < 0) then
begin
OutputDebugString(PChar(' Query failed '));
OutputDebugString(PChar(' ' + mysql_error(sock)));
end;
recbuf := mysql_store_result(sock);
nfields := mysql_num_fields(recbuf);
rowbuf := mysql_fetch_row(recbuf);
if (rowbuf <> nil) then
begin
for i:=0 to nfields-1 do
output^[i] := StrToFloatDef(rowbuf[i], -666);
end;
mysql_free_result(recbuf);
DeInitSQL();
Result := i;
end;
exports
SQL_Query,
InitSQL,
DeInitSQL;
begin
end.
You could use Initialization and Finalization blocks to handle setting up and tearing down the SQL connection. That way you remove the overhead of connection setup from each query that you execute. You can find more info on Initialization and Finalization here.
From the link:
The initialization block is used to initialize certain variables or execute code that is necessary for the correct functioning of the unit. The initialization parts of the units are executed in the order that the compiler loaded the units when compiling a program. They are executed before the first statement of the program is executed.
The finalization part of the units are executed in the reverse order of the initialization execution. They are used for instance to clean up any resources allocated in the initialization part of the unit, or during the lifetime of the program. The finalization part is always executed in the case of a normal program termination: whether it is because the final end is reached in the program code or because a Halt instruction was executed somewhere.
Related
I'll say how I reproduce the problem on lazarus.
I have a form and a datamodule using zeos to enstablish a connection with a local oracle db.
The problem born when I put some code to interlocute with the db.
Here is an example:
OracleMng.ZQuery1.SQL.Clear;
That is exactly the line going in error.
Here is the full code of the form:
unit form1;
{$mode objfpc}{$H+}
interface
uses
Classes, SysUtils, Forms, Controls, Graphics, Dialogs, DBGrids, StdCtrls,
datamodule2;
type
{ TLogin }
TLogin = class(TForm)
Button1: TButton;
DBGrid1: TDBGrid;
procedure Button1Click(Sender: TObject);
private
public
end;
var
Login: TLogin;
implementation
{$R *.lfm}
{ TLogin }
procedure TLogin.Button1Click(Sender: TObject);
begin
OracleMng.ZQuery1.SQL.Clear;
end;
end.
Here is the code of the datamodule:
unit datamodule2;
{$mode objfpc}{$H+}
interface
uses
Classes, SysUtils, DB, ZConnection, ZDataset, ZSqlMonitor;
type
{ TOracleMng }
TOracleMng = class(TDataModule)
DataSource1: TDataSource;
ZConnection1: TZConnection;
ZQuery1: TZQuery;
private
public
end;
var
OracleMng: TOracleMng;
implementation
{$R *.lfm}
{ TOracleMng }
end.
I'm trying
if (OracleMng <> Nil) and (OracleMng.Zquery1 <> Nil) then OracleMng.ZQuery1.SQL.add('select * from help');
if (OracleMng <> Nil) and (OracleMng.Zquery1 <> Nil) then OracleMng.ZQuery1.ExecSQL;
dbgrid1.refresh;
I have no more errors but the DBGrid1 is not filled.
This is my project lpr file:
program project1;
{$mode objfpc}{$H+}
uses
{$IFDEF UNIX}{$IFDEF UseCThreads}
cthreads,
{$ENDIF}{$ENDIF}
Interfaces, // this includes the LCL widgetset
Forms, zcomponent, datamodule2, form1
{ you can add units after this };
{$R *.res}
begin
RequireDerivedFormResource:=True;
Application.Scaled:=True;
Application.Initialize;
Application.CreateForm(TLogin, Login);
Application.Run;
end.
The fact that the change I suggested in my comment, namely
if (OracleMng <> Nil) and (OracleMng.Zquery1 <> Nil) then
OracleMng.ZQuery1.SQL.Clear
evidently stopped you getting the SIGSEGV error suggests that your DataModule and
form are being created in the wrong order, i.e. form first. Check this out by going to
Project | View Source in the IDE. If you see something like
program MyProgram;
[...]
begin
Application.Initialize;
Application.MainFormOnTaskbar := True;
Application.CreateForm(TForm1, Form1);
Application.CreateForm(TDataModule1, DataModule1);
Application.Run;
end.
they are in the wrong order, so swap the two CreateForm lines
Application.CreateForm(TDataModule1, DataModule1);
Application.CreateForm(TForm1, Form1);
With that change, you should no longer need the
if (OracleMng <> Nil) and (OracleMng.Zquery1 <> Nil) then`
Next thing: You seem to be confused about when to use
ZQuery1.ExecSQL
and
ZQuery1.Open
Open is intended for when the SQL statement you are using produces a result set, that is
a collection of records which can be viewed in a TDBGrid. The most usual way to do this
is to use a SELECT statement as in
ZQuery1.SQL.Text := 'select * from MyTable';
ZQuery1.Open;
ExecQuery is intended for use where your SQL statement performs some operation on the database
which does not involve SELECTing records. The most common SQL statements which need ExecSQL are
UPDATE
INSERT
DELETE
though there are others, for example statements which execute stored procedures on the SQL Server
(note that some stored procedures return result sets and so need Open, rather than ExecSQL).
Note that ExecSQL will clear out any records which are in the dataset (ZQuery1) so after
you need to do Open again using a suitable SQL statement
var
S : String;
begin
S := 'update MyTable set number = number +1 where id = 5';
ZQuery.SQL.Text := S;
ZQuery1.ExecSQL; // no records shown in DBGrid1 from here
S := 'select * from MyTable';
ZQuery.SQL.Text := S;
ZQuery1.Open; // records shown in DBGrid1 again
end;
Note that I do
S := 'select * from MyTable';
ZQuery.SQL.Text := S;
instead of
ZQuery1.SQL.Clear;
ZQuery1.SQL.Add('select * from myTable');
The reason for this is that it's much easier to see the whole SQL statement in the debugger by
inspecting the variable S than inspecting the ZQuery1.SQL.Text property and much easier to
see any syntax errors.
You should always Close a dataset that you've Opened once you have finished working with it as it ensures what the data on disk is up to date. if the last SQL operation was ExecSQL, you don't need to close the dataset.
If you set the query's Text property the way I do, with ZQuery1.SQL.Text, you don't need to uses Clear. In any case, it is only equivalent to doing ZQuery1.SQL.Text := '' and it does not affect the state of the dataset - it only does anything when you call ExecSQL or Open.
In the below script, an AfterInstall procedure (SetupOperation) executes after the installation of the lone file installed. If it fails, I want to abort setup, which I do by calling WizardForm.Close. The problem is, if the CurPageChanged procedure is merely declared, the the exception is never properly rethrown in SetupOperation. If I comment it out, everything works properly. There's got to be something simple I'm missing.
[Setup]
DisableDirPage=yes
DisableProgramGroupPage=yes
AppID=someuniqueid
DefaultDirName={pf}\MyCompany\ExampleApp
SetupLogging=yes
AppName=ExampleApp
AppVersion=1.2.3.4
AppVerName=ExampleApp 1.2.3.4
AppPublisher=MyCompany
OutputBaseFilename=MyCompany.ExampleApp.1234
[Files]
Source: "d:\temp\test.txt"; DestDir: "{commonappdata}\MyCompany\ExampleApp"; AfterInstall: SetupOperation
[Code]
var SharedProgressPage: TOutputProgressWizardPage;
// Called by Inno to initialize the user interface
procedure InitializeWizard();
begin
Log('InitializeWizard called');
SharedProgressPage := CreateOutputProgressPage('Installing', 'Please wait while Setup installs the ExampleApp on your computer.');
SharedProgressPage.SetProgress(0,1);
SharedProgressPage.ProgressBar.Style := npbstMarquee;
end;
// Called when the current wizard page changes
procedure CurPageChanged(CurPageID: Integer);
begin
end;
procedure SetupOperation();
begin
Log('SetupOperation called');
try
SharedProgressPage.SetText('', '');
SharedProgressPage.Show;
try
SharedProgressPage.SetText('Performing operation...', '');
RaiseException('Some weird error');
except
Log('Caught an exception; re-raising');
RaiseException(GetExceptionMessage());
finally
Log('Closing progress page');
SharedProgressPage.Hide;
end;
except
Log('An error occurred setting performing the operation: ' + GetExceptionMessage());
WizardForm.Close();
end;
end;
This might have something to do with the usage of a TOutputProgressWizardPage, but I don't see how. It seems to get shown/hidden properly.
I'm currently designing a Datasnap rest server with Delphi. But I have serious memory leaks.
For example, my method station
procedure TV1.station(ID: integer);
begin
GetInvocationMetadata().ResponseContent := Manager.xxxxxxAPI.GetObjectStation(ID);
GetInvocationMetadata().CloseSession := true;
end;
Which call this function :
function TSmmAPI.GetObjectStation( const ID: integer ) : string;
(...)
jsonObject := TJSONObject.Create;
stationSelected := xxxxxManager.WorkShops.GetStation( CNCHandle );
with StatesDB.QueryGetCurrentState( stationSelected.Handle ) do begin
if RecordCount <> 0 then begin
ConvertFileToPcom(stationSelected.Ini.FileName, Pcom);
jsonObject.AddPair( TJSONPair.Create('ID', inttostr(ID)));
jsonObject.AddPair( TJSONPair.Create('Name', FieldByName(sbStaStationField).AsString));
jsonObject.AddPair( TJSONPair.Create('Workshop', stationSelected.Shop.Name));
jsonObject.AddPair( TJSONPair.Create('Group', Pcom.Others.GroupName));
jsonObject.AddPair( TJSONPair.Create('CurrentRef', FieldByName(sbStaRefNameField).AsString));
jsonObject.AddPair( TJSONPair.Create('CurrentState', FieldByName(sbStaStateField).AsString));
jsonObject.AddPair( TJSONPair.Create('Job', FieldByName(sbStaOPNameField).AsString));
jsonObject.AddPair( TJSONPair.Create('Order',FieldByName(sbStaOFNameField).AsString));
//(...), I have 12 addpair.
Disconnect;
end;
Destroy;
end;// with StatesDB.QueryGetCurrentState
result := jsonobject.toString;
jsonObject.FreeInstance;
end;
You can see, I use the resultContent instead of result from a function because I don't want result: in my json response.
So with the report from ReportMemoryLeaksOnShutdown, I see that all my jsonObject and each jsonpair are not destroy !!!
Result leak memory report, 5501 request from my client application
LifeCycle from the server class : Session
I use DSRESTWebDispatcher, set in Session Cycle and Timout at 60000.
Someone have an explanation? Did I forget to do something?
You should call jsonObject.Free instead of jsonObject.FreeInstance
You should never call FreeInstance directly to release the object. It is part of internal allocation/deallocation mechanism. In Delphi destructors call FreeInstance automatically to deallocate object instance memory.
Proper ways to release object instances in Delphi are:
TObject.Free - calls object instance destructor if instance is not nil
TObject.DisposeOf - introduced with Delphi ARC mobile compilers and on dektop compilers it calls TObject.Free.
FreeAndNil(var Obj) - procedure that calls Free on passed object instance and nils that reference
I`m using Delphi XE6 and UniDAC and MySQL
I have some TUniQuery components in my DM and I want to Refresh theme repeatedly, so I put some Timers in my main form and in each timer I create a thread and pass a query to it for refreshing data :
for Example :
TUpdateThread = class(TThread)
private
FQuery : TUniQuery;
FResultHandle : THandle;
public
constructor Create(var Query : TUniQuery; ResultHandle : THandle);
protected
procedure Execute; override;
end;
constructor TUpdateThread.Create(var Query: TUniQuery; ResultHandle : THandle);
begin
inherited Create;
Suspend;
FQuery := Query;
FResultHandle := ResultHandle;
FreeOnTerminate := True;
Resume;
end;
procedure TUpdateThread.Execute;
var
Msg : String;
B : Boolean;
begin
try
B := True;
try
FQuery.Refresh;
except
on E:Exception do
begin
B := False;
Msg := 'Error : ' + #13 + E.Message;
SendMessage(FResultHandle, MSG_UPDATERESULT, 2, Integer(Msg));
end;
end;
finally
if B = True then
SendMessage(FResultHandle, MSG_UPDATERESULT, 1, 1);
Terminate;
end;
end;
Sometimes it`s done successfully but many times I got some errors such as AVs or "Net Pack Header ... " error
or sometimes I have problem in my Grids ( Ehlib DBGrid ) such as error in drawing rows or ... ( specially when I use DisableControls and EnableControls )
All of Queries have same connection , I think each Thread should have his own connection, because of all timers intervals are same , I suggest sometimes refreshing queries interrupts each others
In fact, my database is in a VPS server and there is some client applications , I want to have Live-Tables in Clients and update theme repeatedly
What is the best way to achieve that ?
how I should update my Tables without application hangs !
there is some components as TThreadTimer ( or ... ) , is theme useful for this situation ?!
thanks ...
The first issue is here :
constructor TUpdateThread.Create(var Query: TUniQuery; ResultHandle : THandle);
begin
inherited Create; // Create with no arguments
Suspend; // means CreateSuspended = false
FQuery := Query;
FResultHandle := ResultHandle;
FreeOnTerminate := True;
Resume;
end;
Here you create the thread with the default constructor (CreateSuspended = false) where the thread begins running immediately. You call suspend (which is deprecated and should not be used) immediately, but this is still a race condition since your thread may or may not start trying to Refresh your query before you've assigned it. To create the thread in a suspended state use the overload constructor of
inherited Create(true);
Resume is also deprecated. Instead you should use Start;.
Further, you're passing in a TUniQuery to this thread's constructor. We can assume, I imagine, that this query has affinity to the main thread - this is to say that it is (perhaps) a visual component on a form, has databindings to visual components, or is otherwise interacted with by the user or user interface.
The answer, if so, is that you simply cannot do this - a thread cannot modify an object with affinity to another thread. Your interface may be in the middle of retrieving records from the query when the background thread, for example, is simultaneously destroying them in preparation to refresh the query contents. Naturally this will cause all sorts of problems.
The simple solution is to use a regular timer and refresh synchronously on the main thread. If this takes too long then you need to consider a different strategy altogether. We don't really have sufficient information to suggest much further. If network access and I/O is the bottleneck then you might consider asynchronously refreshing to a separate query object owned by the thread, then synchronously assign it to your view components.
Does anyone have a tip or example of how to use R functions in Delphi? I have used R and Delphi in an integrated way through MySQL I send input from Delphi to MySQL, run the function / on R script that connects to MySQL (package RMySQL) and it returns the output to MySQL, then use the Delphi . But this process is slow all depending on the size of the script R. Does anyone have an example or a tip to speed up the process?
This website had an example but all links are down. The code below shows a small example of how to use present R and Delphi.
unit Unit1;
interface
uses
Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Vcl.StdCtrls, Datasnap.Provider,
Data.DB, Datasnap.DBClient;
type
TForm1 = class(TForm)
Button1: TButton;
Edit1: TEdit;
Edit2: TEdit;
Edit3: TEdit;
Label1: TLabel;
Label2: TLabel;
Label3: TLabel;
procedure Button1Click(Sender: TObject);
private
{ Private declarations }
public
{ Public declarations }
end;
var
Form1: TForm1;
implementation
{$R *.dfm}
uses Unit2;
function StartRAndWait (CommandLine : string) : Boolean;
var
Proc_info: TProcessInformation;
Startinfo: TStartupInfo;
ExitCode: longword;
CreateOK : Boolean;
begin
Result := False;
{ Initialize the structures }
FillChar(proc_info, sizeof (TProcessInformation), #0);
FillChar(startinfo, sizeof (TStartupInfo), #0);
Startinfo.cb := sizeof (TStartupInfo);
Startinfo.dwFlags := STARTF_USESHOWWINDOW or STARTF_USESTDHANDLES;
Startinfo.wShowWindow := SW_HIDE;
{ Attempt to create the process. If successful wait for it to end}
CreateOK := CreateProcess(Nil, PChar('C:\Program Files\R\R-3.0.2\bin\x64\R.exe ' + CommandLine), nil,
nil,False, CREATE_NEW_PROCESS_GROUP+NORMAL_PRIORITY_CLASS, nil,
nil, StartInfo, proc_info);
if (CreateOK) then begin
WaitForSingleObject (proc_info.hProcess, INFINITE);
GetExitCodeProcess(proc_info.hProcess, ExitCode);
Result := True
end;
CloseHandle(proc_info.hThread);
CloseHandle(proc_info.hProcess);
end;
procedure TForm1.Button1Click(Sender: TObject);
var
Command: STRING;
begin
DataModule.ClientDataSet.Open;
DataModule.ClientDataSet.Insert;
DataModule.ClientDataSeta.AsFloat:= strtofloat(Edit1.Text);
DataModule.ClientDataSetb.AsFloat:= strtofloat(Edit2.Text);
DataModule.ClientDataSet.Post;
DataModule.ClientDataSet.ApplyUpdates(0);
DataModule.ClientDataSet.Close;
Screen.Cursor := crHourGlass;
try
Command := 'CMD BATCH script.R outputconsole.txt';
StartRAndWait(Command);
finally
Screen.Cursor := crDefault
end;
DataModule.ClientDataSet.Open;
DataModule.ClientDataSet.Last;
Edit3.Text:= DataModule.ClientDataSetresult.AsString;
DataModule.ClientDataSet.Close;
end;
end.
It seems to me that this is not really a Delphi issue at all. You've provided Delphi code but it's really the R code that is important here. Your Delphi program simply starts a new R process, and waits for it to terminate. Yes there will be a little overhead involved with spinning up the new process, but my guess is that the bulk of the time is spent executing the R code.
So, if you want to speed up the entire calculation, the first thing to do is to speed up that part of the calculation which is taking the longest. Which appears to be the part executing under R.
It is possible to embed R in other processes. This could allow you to avoid waiting for new R processes to start up each time you want to execute your R code. However, embedding R is not the easiest thing to do. There are good R packages that make it easy to embed R in C++. I'm thinking of Dirk Eddelbuettel's Rcpp suite, specifically Rinside. You could use that, or at least draw inspiration from it. However, I strongly suspect that embedding won't get around the root issue which is simply that your R code is taking longer to run than you would like.
I think you could do that through "ShellExecute" and there is no need to TProcessinformation and TStartupInfo.