OrmLite (ServiceStack): Only use temporary db-connections (use 'using'?) - mysql

For the last 10+ years or so, I have always opened a connection the database (mysql) and kept it open, until the application closed. All queries was executed on the connection.
Now, when I see examples on Servicestack webpage, i always see the using-block being used, like:
using (var db = dbFactory.Open())
{
if (db.CreateTableIfNotExists<Poco>())
{
db.Insert(new Poco { Id = 1, Name = "Seed Data"});
}
var result = db.SingleById<Poco>(1);
result.PrintDump(); //= {Id: 1, Name:Seed Data}
}
In my current test-project, I got OrmLite to work in my normal way (one db-connection, no using-statements), so I basically had a class-wide _db, like this:
_dbFactory = new OrmLiteConnectionFactory($"Uid={dbAccount.Username};Password={dbAccount.Password};Server={dbAccount.Address};Port={dbAccount.Port};Database={dbAccount.Database}", MySqlDialect.Provider);
_db = _dbFactory.Open(); // var kept in memory, and used for all queries
It worked in the beginning, but now I suddenly got the Exception:
There is already an open DataReader associated with this Connection which must be closed first
Some code might run a SELECT here and there, and if I understand it correctly, if a SELECT and an INSERT would occur at the same time, this error appears?
If so, is it best-practice to always open a new connection for every single query (say, inside a using-statement)? Isnt that a big overhead, to do what for every query?

Having 1 DB Connection is not ThreadSafe so holding on to the connection is only an option if there’s at most 1 thread accessing the DB connection.
Most ADO.NET providers enable connection pooling by default so it’s more efficient to close the connection when you’re done with it as the connection gets returned back to the pool which reduces the number of active connections in use.

Related

Issue with concurrent requests in CakePHP 2.0

Thanks in advance for attempting to asssist me with this issue.
I'm using CakePHP 2 (2.10.22).
I have a system which creates applications. Each application that gets created has a unique application number. The MySQL database column that stores this application number is set to 'Not null' and 'Unique'. I'm using CakePHP to get the last used application number from the database to then build the next application number for the new application that needs to be created. The process that I have written works without any problem when a single request is received at a given point in time. The problem arises when two requests are received to create an application at the exact same time. The behaviour that I have observed is that the the request that gets picked up first gets the last application number - e.g. ABC001233 and assigns ABC001234 as the application number for the new application it needs to create. It successfully saves this application into the database. The second request which is running concurrently also gets ABC001233 as the last application number and tries to create a new application with ABC001234 as the application number. The MySQL database returns an error saying that the application number is not unique. I then put the second request to sleep for 2 seconds by which time the first application has successfully saved to the database. I then re-attempt the application creation process which first gets the last application number which should be ABC001234 but instead each database read keeps returning ABC001233 even though the first request has long been completed. Both requests have transactions in the controller. What I have noticed is that when I remove these transactions, the process works correctly where for the second request after the first attempt fails, the second attempt works correctly as the system correctly gets ABC001234 as the last application number and assigns ABC001235 as the new application number. I want to know what I need to be doing so as to ensure the process works correctly even with the transaction directives in the controller.
Please find below some basic information on how the code is structured -
Database
The last application number is ABC001233
Controller file
function create_application(){
$db_source->begin(); //The process works correctly if I remove this line.
$result = $Application->create_new();
if($result === true){
$db_source->commit();
)else{
$db_source->rollback();
}
}
Application model file
function get_new_application_number(){
$application_record = $this->find('first',[
'order'=>[
$this->name.'.application_number DESC'
],
'fields'=>[
$this->name.'.application_number'
]
]);
$old_application_number = $application_record[$this->name]['application_number'];
$new_application_number = $old_application_number+1;
return $new_application_number;
}
The above is where I feel the problem originates. For the first request that gets picked up, this find correctly finds that ABC001233 is the last application number and this function then returns ABC001234 as the next application number. For the second request, it also picks up ABC001233 as the last application number but will fail when it tries to save ABC001234 as the application number as the first request has already saved an application with that number. As a part of the second attempt for the second request (which occurs because of the do/while loop) this find is requested again, but instead of returning ABC001234 as the last application number (per the successfuly save of the first request), it keeps returning ABC001233 resulting in a failure to correctly save. If I remove the transaction from the controller, this then works correctly where it will return ABC001234 in the second attempt. I couldn't find any documentation as to why that is and what can be done about the same and is where I need some assistance. Thank you!
function create_new(){
$new_application_number = $this->get_new_application_number();
$save_attempts = 0;
do{
$save_exception = false;
try{
$result = $this->save([$this->name=>['application_number'=>$new_application_number]], [
'atomic'=>false
]);
}catch(Exception $e){
$save_exception = true;
sleep(2);
$new_application_number = $this->get_new_application_number();
}
}while($save_exception === true && $save_attempts++<5);
return !$save_exception;
}
You just have to lock the row with the previous number in a transaction using SELECT ... FOR UPDATE. It's much better than the whole table lock as said in the comments.
According to documentation https://book.cakephp.org/2/en/models/retrieving-your-data.html you just have to add 'lock' => true to get_new_application_number function:
function get_new_application_number(){
$application_record = $this->find('first',[
'order'=>[
$this->name.'.application_number DESC'
],
'fields'=>[
$this->name.'.application_number'
],
'lock'=>true
]);
$old_application_number = $application_record[$this->name]['application_number'];
$new_application_number = $old_application_number+1;
return $new_application_number;
}
How does it work:
The second transaction will wait on that request while the first transaction is ended.
P.S. According to documentation lock option was added in the 2.10.0 version of CakePHP.

IndexedDB: When to close a connection

I would like to know what the correct place to close a connection to the database is.
Let's say that I have the following piece of code:
function addItem(dbName, versionNumber, storeName, element, callback){
var requestOpenDB = indexedDB.open(dbName, versionNumber); //IDBRequest
requestOpenDB.onsuccess = function(event){
//console.log ("requestOpenDB.onsuccess ");
var db = event.target.result;
var trans = db.transaction(storeName, "readwrite");
var store = trans.objectStore(storeName);
var requestAdd = store.add(element);
requestAdd.onsuccess = function(event) {
callback("Success");
};
requestAdd.onerror = function(event) {
callback("Error");
};
};
requestOpenDB.onerror = function(event) {
console.log ("Error:" + event.srcElement.error.message);/* handle error */
callback("Error");
};
}
addItem basically adds a new element into the database. As per my understanding, when the requestAdd event is triggered that doesn't mean necessarily that the transaction has finished. Therefore I am wondering what the best place to call db.close() is. I was closing the connection inside of requestAdd.onsucess, but if an error happens and requestAdd.onerror is triggered instead, the connection might still be opened. I am thinking about adding trans.oncomplete just under request.onerror and close the db connection here which might be a better option. Any inputs will be more than welcome. Thank you.
You may wish to explicitly close a connection if you anticipate upgrading your database schema. Here's the scenario:
A user opens your site in one tab (tab #1), and leaves it open.
You push an update to your site, which includes code to upgrade the database schema, increasing the version number.
The same user opens a second tab to your site (tab #2) and it attempts to connect to the database.
If the connection is held open by tab #1, the connection/upgrade attempt by tab #2 will be blocked. Tab #1 will see a "versionchange" event (so it could close on demand); if it doesn't close its connection then tab #2 will see a "blocked" event.
If the connection is not held open by tab #1, then tab #2 will be able to connect and upgrade. If tab #1 then tries (based on user action, etc) to open the database (with an explicit version number) it will fail since it will be using an old version number (since it still has the old code).
You generally never need to close a connection. You are not creating memory leaks or anything like that. Leaving the connection open does not result in a material performance hit.
I would suggest not worrying about it.
Also, whether you add trans.oncomplete before or after request.onerror is not important. I understand how it can be confusing, but the order in which you bind the listeners is irrelevant (qualified: from within the same function scope).
You can call db.close() immediately after creating the transaction
var trans = db.transaction(storeName, "readwrite");
db.close();
and it will close the connection only after the transaction has completed.
https://developer.mozilla.org/en-US/docs/Web/API/IDBDatabase/close says
The connection is not actually closed until all transactions created using this connection are complete. No new transactions can be created for this connection once this method is called.
If you want to run multiple versions of your app and both access the same database, you might think it's possible to keep connections open to both. This is not possible. You must close the database on one before opening it on another. But one problem is that there is currently no way to know when the database actually closes.

How to manage many openAsync() calls in a big Flex application?

I used openAsync() function many times in my application to open SQLite connection with a success. But lately I added more code that also uses openAsync() and now I obtain this error:
Error: Error #3110: Operation cannot be performed while SQLStatement.executing is true.
at Error$/throwError()
at flash.data::SQLStatement/checkReady()
at flash.data::SQLStatement/execute()
at Function/com.lang.SQL:SQLErrorStack/deleteAllRecordsFromErrorStackTable/com.lang.SQL:connOpenHandler()[C:\work\Lang\trunk\actionscript\src\com\lang\SQL\SQLErrorStack.as:466]
It looks like the previous code didn't finish executing while another has started.
My question is: Why the execution of code in the second connection was rejected? I expected that some kind of a queue mechanism is used but it isn't. I looked everywhere for a solution how to cope with this problem but I failed. Can you help?
Can one opened DB connection solve the problem? What changes to my code should I apply then?
This is the code similar to this, that appears a few times in my application.
var SQLquery:String;
SQLquery = "DELETE FROM ErrorStackTable";
var sqlConn:SQLConnection = new SQLConnection();
sqlConn.addEventListener(SQLEvent.OPEN, connOpenHandler);
var dbFile:File = new File();
dbFile.nativePath = FlexGlobals.topLevelApplication.databaseFullPath_conf+"\\"+FlexGlobals.topLevelApplication.databaseName_conf;
sqlConn.openAsync(dbFile); // openDB
sqlSelect = new SQLStatement();
sqlSelect.sqlConnection = sqlConn;
sqlSelect.text = SQLquery;
function connOpenHandler(event:SQLEvent):void
{
sqlSelect.addEventListener(SQLEvent.RESULT, resultSQLHandler);
sqlSelect.addEventListener(SQLErrorEvent.ERROR, errorHandler);
sqlSelect.execute();
}
In Big Flex Applications Try To Avoid openAsync(db) calls because of the reusablity of the SQL code , if u have many sql statments to be executed then you should define more and more sql statments . and if you have dynamic result [Array] getting from web service (RPC ) then you will surely get an error although it is successful Execution and array insertion in the database will be fail .. Just Look at
the link Click Here You Will Get your answer
I just changed conn.openAsync(db); to conn.open(db); and it worked
Thanks

entity createdatabase ldf log file name change default how to

Visual Web Developer. Entity data sources model. I have it creating the new database fine. Example
creates SAMPLE1.MDF and SAMPLE1.LDF
When I run my app, it creates another SAMPLE1_LOG.lDF file.
When I run createdatabase, is there a place I can specify the _LOG.ldf for the log file? SQL 2008 r2.
It messes up when I run the DeleteDatabase functions... 2 log files...
How come it does not create the file SAMPLE1_Log.ldf to start with, if that is what it is looking for...
Thank you for your time,
Frank
// database or initial catalog produce same results...
// strip the .mdf off of newfile and see what happens?
// nope. this did not do anything... still not create the ldf file correctly!!!
// sample1.mdf, sample1.ldf... but when run, it creates sample1_log.LDF...
newfile = newfile.Substring(0, newfile.Length - 4);
String mfile = "Initial Catalog=" + newfile + ";data source=";
String connectionString = FT_EntityDataSource.ConnectionManager.GetConnectionString().Replace("data source=", mfile);
// String mexclude = #"attachdbfilename=" + "|" + "DataDirectory" + "|" + #"\" + newfile + ";";
// nope. must have attach to create the file in the app_data, otherwise if goes to documents & setting, etc sqlexpress.
// connectionString = connectionString.Replace(mexclude, "");
Labeldebug2.Text = connectionString;
using (FTMAIN_DataEntities1 context = new FTMAIN_DataEntities1(connectionString))
{
// try
// {
if (context.DatabaseExists())
{
Buttoncreatedb.Enabled = false;
box.Checked = true;
boxcreatedate.Text = DateTime.Now.ToString();
Session["zusermdf"] = Session["zusermdfsave"];
return;
// Make sure the database instance is closed.
// context.DeleteDatabase();
// i have entire diff section for deletedatabase.. not here.
}
// View the database creation script.
// Labeldebug.Text = Labeldebug.Text + " script ==> " + context.CreateDatabaseScript().ToString().Trim();
// Console.WriteLine(context.CreateDatabaseScript());
// Create the new database instance based on the storage (SSDL) section
// of the .edmx file.
context.CreateDatabaseScript();
context.CreateDatabase();
}
took out all the try, catch so i can see anything that might happen...
==========================================================================
Rough code while working out the kinks..
connection string it creates
metadata=res://*/FT_EDS1.csdl|res://*/FT_EDS1.ssdl|res://*/FT_EDS1.msl;provider=System.Data.SqlClient;provider connection string="Initial Catalog=data_bac100;data source=.\SQLEXPRESS;attachdbfilename=|DataDirectory|\data_bac100.mdf;integrated security=True;user instance=True;multipleactiveresultsets=True;App=EntityFramework"
in this example, the file to create is "data_bac100.mdf".
It creates the data_bac100.mdf and data_bac100.ldf
when I actually use this file and tables to run, it auto-creates data_bac100_log.LDF
1) was trying just not to create the ldf, so when the system runs, it just creates the single one off the bat...
2) the Initial Catalog, and/or Database keywords are ONLY added to the connection string to run the createdatabase().. the regular connection strings created in web config only have attachdbfilename stuff, and works fine.
I have 1 connection string for unlimited databases, with the main database in the web.config.. I use a initialize section based on the user roles, whether visitor, member, admin, anonymous, or not authenticated... which sets the database correctly with a expression builder, and function to parse the connection string with the correct values for the database to operate on. This all runs good.
The entity framework automatically generates the script. I have tried with and without the .mdf extensions, makes no difference... thought maybe there is a setup somewhere that holds naming conventions for ldf files...
Eventually all of this will be for naught when start trying to deploy where not using APP_Data folder anyways...
Here is an example of connection string created when running application
metadata=res://*/FT_EDS1.csdl|res://*/FT_EDS1.ssdl|res://*/FT_EDS1.msl;provider=System.Data.SqlClient;provider connection string="data source=.\SQLEXPRESS;attachdbfilename=|DataDirectory|\TDSLLC_Data.mdf;integrated security=True;user instance=True;multipleactiveresultsets=True;App=EntityFramework"
in this case, use the TDSLLCData.mdf file...
04/01/2012... followup...
Entity Framework
feature
Log files created by the ObjectContext.CreateDatabase method
change
When the CreateDatabase method is called either directly or by using Code First with the SqlClient provider and an AttachDBFilename value in the connection string, it creates a log file named filename_log.ldf instead of filename.ldf (where filename is the name of the file specified by the AttachDBFilename value).
impact.
This change improves debugging by providing a log file named according to SQL Server specifications. It should have no unexpected side effects.
http://msdn.microsoft.com/en-us/library/hh367887(v=vs.110).aspx
I am on a Windows XP with .net 4 (not .net 4.5)... will hunt some more.. but looks like a issue that cannot be changed.
4/1/2012, 4:30...
ok, more hunting and searching and some of the inconsistancies I have experienced with createdatabase and databaseexists... so .net 4.5 is supposed to add the _log.ldf, and not just .ldf files, so they must have addressed this for some reason....
found others with same issues, but different server....
MySQL has a connector for EF4, the current version is 6.3.5 and its main functionalities are working fine but it still has issues with a few methods, e.g.
•System.Data.Objects.ObjectContext.CreateDatabase()
•System.Data.Objects.ObjectContext.DatabaseExists()
which makes it difficult to fully use the model-first approach. It's possible by manually editing the MySQL script (available with the CreateDatabaseScript method). The MySQL team doesn't seem eager to solve those bugs, I'm not sure what the commitment level actually is from their part but it certainly is lower than it once was.
That being said, the same methods fail with SQL CE too (they are not implemented, and I don't see the MS team as likely to tackle that soon).
Ran out of space below... it just becomes a problem when create a database, and it does not create the _log.ldf file, but just the ldf file, then use the database, and it creates a _log.ldf file... now you have 2 ldf files.. one becomes invalid.. Then when done with the database, delete it, then try to create a new, and a ldf exists, it will not work....
it turns out this is just the way it is with EF4, and they changed with EF4.5 beta to create the _log.ldf file to match what is created when the database is used.
thanks for time.
I've never used this "mdf attachment" feature myself and I don't know much about it, but according to the xcopy deployment documentation, you should not create a log file yourself because it will be automatically created when you attach the mdf. The docs also mention naming and say that the new log filename ends in _log.ldf. In other words, this behaviour appears to be by design and you can't change it.
Perhaps a more important question is, why do you care what the log file is called? Does it actually cause any problems for your application? If so, you should give details of that problem and see if someone has a solution.

How IQueryables are dealt with in ASP.NET MVC Views?

I have some tables in a MySQL database to represent records from a sensor. One of the features of the system I'm developing is to display this records from the database to the web user, so I used ADO.NET Entity Data Model to create an ORM, used Linq to SQL to get the data from the database, and stored them in a ViewModel I designed, so I can display it using MVCContrib Grid Helper:
public IQueryable<TrendSignalRecord> GetTrends()
{
var dataContext = new SmgerEntities();
var trendSignalRecords = from e in dataContext.TrendSignalRecords
select e;
return trendSignalRecords;
}
public IQueryable<TrendRecordViewModel> GetTrendsProjected()
{
var projectedTrendRecords = from t in GetTrends()
select new TrendRecordViewModel
{
TrendID = t.ID,
TrendName = t.TrendSignalSetting.Name,
GeneratingUnitID = t.TrendSignalSetting.TrendSetting.GeneratingUnit_ID,
//{...}
Unit = t.TrendSignalSetting.Unit
};
return projectedTrendRecords;
}
I call the GetTrendsProjectedMethod and then I use Linq to SQL to select only the records I want. It is working fine in my developing scenario, but when I test it in a real scenario, where the number of records is way greater (something around a million records), it stops working.
I put some debug messages to test it, and everything works fine, but when it reaches the return View() statement, it simply stops, throwing me a MySQLException: Timeout expired. That let me wondering if the data I sent to the page is retrieved by the page itself (it only search for the displayed items in the database when the page itself needs it, or something like that).
All of my other pages use the same set of tools: MVCContrib Grid Helper, ADO.NET, Linq to SQL, MySQL, and everything else works alright.
You absolutely should paginate your data set before executing your query if you have millions of records. This could be done using the .Skip and .Take extension methods. And those should be called before running any query against your database.
Trying to fetch millions of records from a database without pagination would very likely cause a timeout at best.
Well, assuming information in this blog is correct, .AsPagination method requires you to sort your data by a particular column. It's possible that trying to do an OrderBy on a table with millions of records in it is just a time consuming operation and times out.