I have some tables in a MySQL database to represent records from a sensor. One of the features of the system I'm developing is to display this records from the database to the web user, so I used ADO.NET Entity Data Model to create an ORM, used Linq to SQL to get the data from the database, and stored them in a ViewModel I designed, so I can display it using MVCContrib Grid Helper:
public IQueryable<TrendSignalRecord> GetTrends()
{
var dataContext = new SmgerEntities();
var trendSignalRecords = from e in dataContext.TrendSignalRecords
select e;
return trendSignalRecords;
}
public IQueryable<TrendRecordViewModel> GetTrendsProjected()
{
var projectedTrendRecords = from t in GetTrends()
select new TrendRecordViewModel
{
TrendID = t.ID,
TrendName = t.TrendSignalSetting.Name,
GeneratingUnitID = t.TrendSignalSetting.TrendSetting.GeneratingUnit_ID,
//{...}
Unit = t.TrendSignalSetting.Unit
};
return projectedTrendRecords;
}
I call the GetTrendsProjectedMethod and then I use Linq to SQL to select only the records I want. It is working fine in my developing scenario, but when I test it in a real scenario, where the number of records is way greater (something around a million records), it stops working.
I put some debug messages to test it, and everything works fine, but when it reaches the return View() statement, it simply stops, throwing me a MySQLException: Timeout expired. That let me wondering if the data I sent to the page is retrieved by the page itself (it only search for the displayed items in the database when the page itself needs it, or something like that).
All of my other pages use the same set of tools: MVCContrib Grid Helper, ADO.NET, Linq to SQL, MySQL, and everything else works alright.
You absolutely should paginate your data set before executing your query if you have millions of records. This could be done using the .Skip and .Take extension methods. And those should be called before running any query against your database.
Trying to fetch millions of records from a database without pagination would very likely cause a timeout at best.
Well, assuming information in this blog is correct, .AsPagination method requires you to sort your data by a particular column. It's possible that trying to do an OrderBy on a table with millions of records in it is just a time consuming operation and times out.
Related
my code:
DB::transaction(function () {
$foo = Foo::whereId(1)->lockForUpdate()->with('bars')->first();
// dump foo & bars
// update foo's columns
// update bars' columns
});
and I run this code at the same time twice, first time it can update correctly, but the second time when I query the foo, foo's columns is correct, but the bars are still old(in database it's correct), why is it and how to solve it?
Since you are using lockForUpdate() and want to use the new data after you update, you need to re-hydrate the model using refresh()
The refresh method will re-hydrate the existing model using fresh data from the database. In addition, all of its loaded relationships will be refreshed as well:
https://laravel.com/docs/9.x/eloquent#refreshing-models
There is a great answer by NoOorZ24 # Laravel lockforupdate (Pessimistic Locking) explaining how the lock for transaction works to further clarify.
I have just migrated my asp.net core web app's database from Sql Server to MySQL. It went very smooth but i am noticing a loss in performance. When i change my database back to the SQL Server, the page loads in less then one second. When im using the MySQL database, it takes +- 15 seconds to load the page. They both have the same data. It is not just on this page; I just picked this one out as an example.
I use Entity Framework and I query for all the Sales in my database (+- 100 records). I use IQueryable and then return it to the View. When I am debugging I can tell the queries go pretty fast, but when I return my View, this is where it starts to slow down.
public IActionResult SalesOverview()
{
var sales = _repo.QueryAsNoTracking().
.Select(result => new
{
Id = result.Id,
ClientId= result.Client.Id,
ClientNamr = result.Client.Name,
DeliveryDate = result.DeliveryDate,
NumberOfDetails = result.NumberOfDetails.Count
});
return View(sales);
}
Did I miss out on anything during the migration? I'm pretty sure it is not my code because it runs fine with the SQL Server
Kind regards,
Brian
I have collection with 10mill rows without any index.In this case system should read whole table ?
When i use eplain statement then it shows
db.employees.find({hundreds2:{$lt:1}},{}).explain();
"nscannedObjects" : 10000000,
"n" : 105
millis" : 6027
It works fine.
But i am using java to process query . Here is code
whereQuery = new BasicDBObject();
whereQuery.put("hundreds2",new BasicDBObject("$lt", rangeQuery));
timer.start();
setupMongoDBReadQuery(arrForRange[posOfArr]);
cursor = coll.find(whereQuery);
timer.stop();
this.selectTime= timer.elapsed();
timer.start();
while (cursor.hasNext())
{
numberOfRows++;
cursor.next();
}
timer.stop();
this.fetchTime= timer.elapsed();
this.totalOfSelAndFetch=this.selectTime+this.fetchTime;
But after test result .I got this information
selTime=2 fetchTime=6350 numRows105 TotalTime6352
selTime=0 fetchTime=6290 numRows471 TotalTime6290
selTime=0 fetchTime=6365 numRows922 TotalTime6365
Why fetch time is more than select .As per my knowledge ,while loop is just printing data . Why it taking so much time to print and how mongoDB select number of rows with 0 or 2 millSec?
Same experiment i did in MySQL with similiar code and results are
selTime=6302 fetchTime=1 numRows105 TotalTime6303
selTime=6318 fetchTime=1 numRows471 TotalTime6319
selTime=6387 fetchTime=2 numRows922 TotalTime6389
MongoDB uses lazy evaluation with cursors. That means in many cases when you start a MongoDB query which returns a cursor, the query doesn't get executed yet.
The actual selection happens when you start requesting data from the cursor.
The main reason is that this allows you to call methods like sort(by), limit(n) or skip(n) on the cursor which can often be processed much more efficiently on the database before selecting any data.
So what you measure with the "fetch time" is actually also part of the selection.
When you want to force the query to execute without fetching any data yet, you could call explain() on the cursor. The database can't measure the execution time without actually performing the query. However, in actual real-world use, I would recommend you to not do this and use cursors the way they were intended.
I am getting an IQueryable from my database and then I am getting another IQueryable from that first one -that is, I am filtering the first one.
My question is -does this affect performance? How many times will the code call the database? Thank you.
Code:
DataContext _dc = new DataContext();
IQueryable offers =
(from o in _dc.Offers
select o);
IQueryable filtered =
(from o in offers
select new { ... } );
return View(filtered);
The code you have given will never call the database since you're never using the results of the query in any code.
IQueryable collections aren't filled until you iterate through them...and you're not iterating through anything in that code sample (ah, the beauty of lazy initialization).
That also means that each of those statements will be executed as its own query against the database which results in no performance cost over doing two completely independent queries.
SO is not a replacement for developer tools. There are many good free tools able to tell you exactly what this code translates into and how it works. Use Reflector on this method and look at what code is generated and reason for yourself what is going on from there.
This code is a no-go
var errors = (from error in db.ELMAH_Error
select new
{
error.Application,
error.Host,
error.Type,
error.Source,
error.Message,
error.User,
error.StatusCode,
error.TimeUtc
}).ToList();
return View(errors);
as it results in a 'requires a model of type IEnumerable' error. The following code of course works fine, but selects all the columns, some of which I'm simply not interested in:
var errors = (from error in db.ELMAH_Error
select error).ToList();
return View(errors);
I'm brand spanking new to MVC2 + L2E, so maybe I'm just not thinking in the right mindset yet, but this seems counter-intuitive. Is there an easy way to select a limited number of columns, or is this just part of using an ORM?
This isn't an issue with L2E / L2S, but rather with MVC strongly-typed views.
I'm betting it's not just "requires a model of type IEnumerable", but "a model of type IEnumerable<Something>". If you have a strongly-typed view bound to the type of your errors table, then trying to pass it the anonymous type created by your query (via select new {...}) will cause a type mismatch.
You could either create a new type which contains only the rows that you want, and use that as your view page Model, or you could just do the full select and then limit the columns in your ASP code.