Why mongoDB takes less time for Select than Fetch time? - mysql

I have collection with 10mill rows without any index.In this case system should read whole table ?
When i use eplain statement then it shows
db.employees.find({hundreds2:{$lt:1}},{}).explain();
"nscannedObjects" : 10000000,
"n" : 105
millis" : 6027
It works fine.
But i am using java to process query . Here is code
whereQuery = new BasicDBObject();
whereQuery.put("hundreds2",new BasicDBObject("$lt", rangeQuery));
timer.start();
setupMongoDBReadQuery(arrForRange[posOfArr]);
cursor = coll.find(whereQuery);
timer.stop();
this.selectTime= timer.elapsed();
timer.start();
while (cursor.hasNext())
{
numberOfRows++;
cursor.next();
}
timer.stop();
this.fetchTime= timer.elapsed();
this.totalOfSelAndFetch=this.selectTime+this.fetchTime;
But after test result .I got this information
selTime=2 fetchTime=6350 numRows105 TotalTime6352
selTime=0 fetchTime=6290 numRows471 TotalTime6290
selTime=0 fetchTime=6365 numRows922 TotalTime6365
Why fetch time is more than select .As per my knowledge ,while loop is just printing data . Why it taking so much time to print and how mongoDB select number of rows with 0 or 2 millSec?
Same experiment i did in MySQL with similiar code and results are
selTime=6302 fetchTime=1 numRows105 TotalTime6303
selTime=6318 fetchTime=1 numRows471 TotalTime6319
selTime=6387 fetchTime=2 numRows922 TotalTime6389

MongoDB uses lazy evaluation with cursors. That means in many cases when you start a MongoDB query which returns a cursor, the query doesn't get executed yet.
The actual selection happens when you start requesting data from the cursor.
The main reason is that this allows you to call methods like sort(by), limit(n) or skip(n) on the cursor which can often be processed much more efficiently on the database before selecting any data.
So what you measure with the "fetch time" is actually also part of the selection.
When you want to force the query to execute without fetching any data yet, you could call explain() on the cursor. The database can't measure the execution time without actually performing the query. However, in actual real-world use, I would recommend you to not do this and use cursors the way they were intended.

Related

FDQuery Interface to Progress Bar - DELPHI

I am trying to make an interface for my insert Query, this moves a huge number of data.
DataModule2.FDQueryInsertExceed.Close;
DataModule2.FDQueryInsertExceed.Sql.Clear;
DataModule2.FDQueryInsertExceed.Sql.Add('INSERT IGNORE INTO tblLogs');
DataModule2.FDQueryInsertExceed.Sql.Add('SELECT * FROM tbllogs WHERE Datetimelog <
date_sub(NOW(), INTERVAL 1 month);');
DataModule2.FDQueryInsertExceed.ExecSQL;
Dont know where to put the progress bar, and also if you can post here a link with full manual of FD Components.
FireDAC includes the TFDEventAlerter, which allows applications to receive certain notifications
from the SQL backend, including some progress notifications. Unfortunately, it does not support
MySQL so you are going to have to devise a way of updating your ProgressBar yourself.
The main problem with doing it yourself is that after you call DataModule2.FDQueryInsertExceed.ExecSQL,
it will not return until ExecSQL has completed execution, so you can't get any opportunity
to update your ProgressBar until then and by the time that you do, the SQL backend has finished
working.
So, realistically, the best you can do is to snip your db inserted up into a series of batches and
update the ProgressBar after each one. Before you start the batches, you can do a SELECT COUNT(*)
which should tell you at least approximately how many rows will be involved overall, so that
you can calculate the percentage complete after each batch. I say "approximately"
in case new inserts are required while your ExecSQL is executing,
The simplest place to start is by determining how many of your inserts can be executed in
a few hundred milliseconds, parameterize your SQL to match that and use a TTimer to start
each batch. So your TTimers's OnTimeer wou look something like
TForm1.OnTimer1.OnTimer(Sender : TObject);
var
S : String;
begin
S := 'INSERT IGNORE INTO tblLogs SELECT * FROM tbllogs WHERE Datetimelog ... //
// complete as necessary to add a starting DateTime and an ending one for the batch
DataModule2.FDQueryInsertExceed.Sql.Text := S;
DataModule2.FDQueryInsertExceed.ExecSQL;
// Update ProgressBar here
end;
An enhancement (but a step change in difficulty) would be to call DataModule2.FDQueryInsertExceed.ExecSQL
in a background thread and, once it has executed, execute code in the main, VCL thread, not in the background thread, to update
the ProgressBar. However, exactly how to do that is way beyond the scope of this answer.

Is it good to filter data in controller or use sql query in model?

What is the best approach for searching?What will be difference if i filter the all data in controller and get result, and use where query in model and get result ?Please suggest your opinion.
It depends on the complexity of your query..
You can try to mesure the time of processing by putting flags in your code between each steps in order to measure the time of processing.
Then you make one test of speed processing like:
print time_flag 1
var results_sql_processing = *complex query*
print time flag 2
var raw_results_script_processing = *dump query*
print time flag 3
var results_script_processing = *processing the data*
print time flag 4
and make sure results_script_processing == results_sql_processing.
You can also set different datasets size (limit 100, 500, 1000) and see how the difference evolves between both solutions
Also, i would recommend to take a look at the query builders you might find in many frameworks (I use laravel's query builder).
They are usually a very good compromise when the query isn't too complex (no data aggregation, complex concat,....), you can still use joins, union and many filters on them.
But in case you want to get a super complex pivot table for instance, just build a strong sql query and then fire it in your code!

how to use async.series in middle of for loop

I am trying to find a way to ensure my database query is ran as part of the for loop shown below. I don't know if async.series is the best way, or if fibers might work? Rather than paste loads of code, in pseudo code it looks like this:
for length of array
newArray = array.split
databaseQuery(select **** from *** where x = newArray[0] and y = newArray[1]
my problem is as node is asynchronous, it simply repeats the for loop split while the query is running and then gets the result of the query, meaning I only get the last result returned.
Is there a way to ensure the database query is executed each time in the for loop? I've used nested queries, and callbacks in the past but I can't seem to figure out the best way to call the query each time for the loop

Show MessageBox if no records for report

I already asked a Question Here.
My this question is just a step ahead to same problem.
I have a delphi code which works perfectly for calling report.
But, now I wanted to show a MessageBox before opening rpt file.
I tried to query it separately for record count and then decide for MessageBox. But, this solution is having a worst case where a aprticular report's query itself takes 3 mins to execute and then again querying while opening rpt it takes 30 sec to load (in second query it takes less time may be because of some data may present in buffer/temp place,etc.).
qPODPhy.close;
qPODPhy.SQL.clear;
qPODPhy.SQL.text :='select * from ViewName';
qPODPhy.Open;
If qPODPhy.RecordCount < 1 Then
MessageBOx('No data To Display...');
Else
Begin
crRep.Somproperties := Initialization
.
.
.
CrRep.SQLQuery := qPODPhy.SQL.text;
crRep.action := 1
End
My Question is :
How can I show a MessageBox if no record is going to present for particular view's output.
OR
Is there a method where I can open the dataset of .rpt file in delphi code and just check the count of records and make the decision? In short, is there some property of crystalreport component which can do this?
You can do a select count(*) separately, that is much faster.
Or maybe select only one record: SELECT TOP 1 ....
And, as RBA suggested, you can try putting that SELECT COUNT in a stored procedure for even more speed.
Just experiments with these methods to see if/when you've gained enough speed.
Are you pushing the data ? You can probably use ReadRecords method of the ReportDocument and check Rows.Count property. If the report is retrieving the data the you can use the NoData event of ReportDocument.

How IQueryables are dealt with in ASP.NET MVC Views?

I have some tables in a MySQL database to represent records from a sensor. One of the features of the system I'm developing is to display this records from the database to the web user, so I used ADO.NET Entity Data Model to create an ORM, used Linq to SQL to get the data from the database, and stored them in a ViewModel I designed, so I can display it using MVCContrib Grid Helper:
public IQueryable<TrendSignalRecord> GetTrends()
{
var dataContext = new SmgerEntities();
var trendSignalRecords = from e in dataContext.TrendSignalRecords
select e;
return trendSignalRecords;
}
public IQueryable<TrendRecordViewModel> GetTrendsProjected()
{
var projectedTrendRecords = from t in GetTrends()
select new TrendRecordViewModel
{
TrendID = t.ID,
TrendName = t.TrendSignalSetting.Name,
GeneratingUnitID = t.TrendSignalSetting.TrendSetting.GeneratingUnit_ID,
//{...}
Unit = t.TrendSignalSetting.Unit
};
return projectedTrendRecords;
}
I call the GetTrendsProjectedMethod and then I use Linq to SQL to select only the records I want. It is working fine in my developing scenario, but when I test it in a real scenario, where the number of records is way greater (something around a million records), it stops working.
I put some debug messages to test it, and everything works fine, but when it reaches the return View() statement, it simply stops, throwing me a MySQLException: Timeout expired. That let me wondering if the data I sent to the page is retrieved by the page itself (it only search for the displayed items in the database when the page itself needs it, or something like that).
All of my other pages use the same set of tools: MVCContrib Grid Helper, ADO.NET, Linq to SQL, MySQL, and everything else works alright.
You absolutely should paginate your data set before executing your query if you have millions of records. This could be done using the .Skip and .Take extension methods. And those should be called before running any query against your database.
Trying to fetch millions of records from a database without pagination would very likely cause a timeout at best.
Well, assuming information in this blog is correct, .AsPagination method requires you to sort your data by a particular column. It's possible that trying to do an OrderBy on a table with millions of records in it is just a time consuming operation and times out.