I am using NHibernate/Fluent NHibernate in an ASP.NET MVC app with a MySQL database. I am working on an operation that reads quite a bit of data (relative to how much is inserted), processes it, and ends up inserting (currently) about 50 records. I have one ISession per request which is created/destroyed in the begin/end request event handlers (exactly like like http://ayende.com/Blog/archive/2009/08/06/challenge-find-the-bug-fixes.aspx), and I am reading in data and adding new objects (as in section 16.3 at https://www.hibernate.org/hib_docs/nhibernate/html/example-parentchild.html), and finally calling Flush() on the session to actually run all the inserts.
Getting data out and lazy loading work fine, and when I call Flush exactly 2 new records are being inserted (I am manually checking the table to find this out), and then I get the following error:
NonUniqueObjectException: a different object with the same
identifier value was already
associated with the session: 0, of
entity: ...
I am new to NHibernate and while searching for a solution have tried explicitly setting the Id property's generator to both Native and Identity (it is a MySQL database and the Id column is an int with auto_increment on), and explicitly setting the unsaved value for the Id property to 0. I still get the error, however.
I have also tried calling Flush at different times (effectively once per INSERT) and I then get the same error, but for an identity value other than 0 and at seemingly random points in the process (sometimes I do not get it at all in this scenario, but sometimes I do at different points).
I am not sure where to go from here. Any help or insight would be greatly appreciated.
EDIT: See the answer below.
EDIT: I originally posted a different "answer" that did not actually solve the problem, but I want to document my findings here for anyone else who may come across it.
After several days of trying to figure out the problem and resolve this issue, and being extremely frustrated because the issue seemed to go away for awhile and then come back intermittently (causing me to think multiple times that a change I made fixed it, when in fact it did not), I believe I have tracked down the real issue.
A few times after I turned the log4net level for NHibernate up to DEBUG, the problem went away, but I was finally able to get the error with that log level. Included in the log were these lines:
Building an IDbCommand object for the SqlString: SELECT LAST_INSERT_ID()
...
NHibernate.Type.Int32Type: 15:10:36 [8] DEBUG NHibernate.Type.Int32Type: returning '0' as column: LAST_INSERT_ID()
NHibernate.Id.IdentifierGeneratorFactory: 15:10:36 [8] DEBUG NHibernate.Id.IdentifierGeneratorFactory:
Natively generated identity: 0
And looking up just a few lines I saw:
NHibernate.AdoNet.ConnectionManager: 15:10:36 [8] DEBUG NHibernate.AdoNet.ConnectionManager: aggressively releasing database connection
NHibernate.Connection.ConnectionProvider: 15:10:36 [8] DEBUG NHibernate.Connection.ConnectionProvider: Closing connection
It seems that while flushing the session and performing INSERTs, NHibernate was closing the connection between the INSERT statement and the "SELECT LAST_INSERT_ID()" to get the id that was generated by MySQL for the INSERT statement. Or rather, I should say it was sometimes closing the connection which is one reason I believe the problem was intermittent. I can't find the link now, but I believe I also read in all my searching that MySQL will sometimes return the correct value from LAST_INSERT_ID() even if the connection is closed and reopened, which is another reason I believe it was intermittent. Most of the time, though, LAST_INSERT_ID() will return 0 if the connection is closed and reopened after the INSERT.
It appears there are 2 ways to go about fixing this issue. First is a patch available here that looks like it will make it into NHibernate 2.1.1, or which you can use to make your own build of NHibernate, which forces the INSERT and SELECT LAST_INSERT_ID() to run together. Second, you can set the connection.release_mode to on_close as described in this blog post which prevents NHibernate from closing the connection until the ISession is explicitly closed.
I took the latter approach, which is done in FluentNHibernate like this:
Fluently.Configure()
...
.ExposeConfiguration(c => c.Properties.Add("connection.release_mode", "on_close"))
...
This also had the side effect of drastically speeding up my code. What was taking 20-30 seconds to run (when it just so happened to work before I made this change) is now running in 7-10 seconds, so it is doing the same work in ~1/3 the time.
Related
I am in the process of writing my first C client for MySQL 5.5 and have stumbled across the following page in the documentation. Nearly at the end, it states (bold emphasis mine, italic emphasis not mine):
An advantage of mysql_use_result() is [...]. Disadvantages are that
[...]. Furthermore, you must retrieve all the rows even if you
determine in mid-retrieval that you've found the information you were
looking for.
The last sentence is not clear to me.
1) What happens if I don't follow that line?
2) I think that there actually must be a way to prematurely end fetching rows if I decide that I have enough information (otherwise, this whole thing wouldn't make much sense in my eyes).
I understand that something bad could happen if I just stop fetching rows and then try to execute the next statement, but isn't there a function like mysql_finish_fetch() or something like that?
And what happens if I call mysql_free_result()? This should free the result even if I haven't fetched all rows yet, so it should be safe to call it in mid-retrieval and continue with whatever I'd like to do. Am I wrong here?
This sounds like an internal threading issue that MySQL exposes to the client. Chalk it up to the various MySQL gotchas. The short of it is that MySQL apparently has a finite number of "searchers" internally, and using mysql_use_result() apparently dedicates one of them to your API request. Further, MySQL apparently has no exposed API call to cancel such a request. The only option is to see the fetch through until the end.
The slightly longer version: internally, MySQL's cursors apparently have a single code path -- I imagine for performance in the common cases. That code path exits only when the cursor finds no more results. When you use the more common mysql_store_result(), MySQL has done this already before returning the result to the application. When you use mysql_use_result(), however, MySQL requires that you do "the dirty work" of iterating the rest of the result set so as to clear the cursor. Fun.
From the documentation:
mysql_use_result() initiates a result set retrieval but does not actually read the result set into the client like mysql_store_result() does. Instead, each row must be retrieved individually by making calls to mysql_fetch_row(). This reads the result of a query directly from the server without storing it in a temporary table or local buffer, which is somewhat faster and uses much less memory than mysql_store_result(). The client allocates memory only for the current row and a communication buffer that may grow up to max_allowed_packet bytes.
On the other hand, you should not use mysql_use_result() for locking reads if you are doing a lot of processing for each row on the client side, or if the output is sent to a screen on which the user may type a ^S (stop scroll). This ties up the server and prevent other threads from updating any tables from which the data is being fetched.
When using mysql_use_result(), you must execute mysql_fetch_row() until a NULL value is returned, otherwise, the unfetched rows are returned as part of the result set for your next query. The C API gives the error Commands out of sync; you can't run this command now if you forget to do this!
So, to actually answer your questions:
1) What happens if I don't follow that line?
The C API will return the error message: Commands out of sync; you can't run this command now
2) I think that there actually must be a way to prematurely end fetching rows if I decide that I have enough information (otherwise, this whole thing wouldn't make much sense in my eyes).
One would think, but no. You must iterate the result set completely.
I'm building an application that involves significant image processing. We're building it in a distributed manor, with an arbitrary number of rendering machines receiving requests to render images from a RabbitMQ exchange.
Because a request for an image might come in while a rendering is already in process, and because I don't want to have two render servers duplicating work, I created a column in the mySQL images table called is_rendering as a boolean.
When a render server receives a request for render it follows a series of steps that looks like this:
Select for update the image row
If is_rendering == true abort render request
Set is_rendering == true and commit the transaction
Render the image and store the thumbnail in a globally accessible store
Set is_rendering == false and return
It definitely works, but I'm worried that these frequent database updates seem a little bit silly. Furthermore I'm considering the edge case where a render server fails in the middle of a rendering and leaves is_rendering == true, preventing that image from ever being rendered. The solution I'm considering to this problem is to change the is_rendering column from a tinyint(1) to a datetime field, and storing the date of locking as the "true" value, and null as the "false" value. A periodic service health check could select all rows with a is_rendering value past a certain time period and release the lock in this case.
Is this a sane approach to this problem, or are there other more elegant approaches I should be considering?
Dear As I understand your problem, Your first approach is also correct if you are following below rule:
1) your table type is innoDB.
2) You are using transaction in your code. Because it will rollback if any breakup happend during updation.
In the end, your 2nd approach is also better. if you are not fulfilling my mentioned points
I've gone ahead and changed the implementation to use a DATETIME column.
I was really curious to here if this was a poor use of mySQL in general. From what I've researched, I could go with something like Hadoop's ZooKeeper: http://zookeeper.apache.org/doc/r3.1.2/recipes.html or something like Google's internal Chubby system. Since this is just the first iteration of the service I'm going to stick with mySQL.
From what I've read here, and what I've read online using mySQL as a way to produce a global lock isn't a terrible idea, and changing it to a DATETIME column, while a little obtuse, is the best way to implement an expiration policy to handle the odd edge case of a machine shutting down in the middle of processing a job.
Holding onto a transaction-level lock would be another approach, but it doesn't make sense when a single server is running many threads with a small connection pool, it would tie up the connection unnecessarily, although it has a built-in expiration when the client connection is lost.
I was originally using Nhibernate with MSSQL (2008) and everything worked great. I had a web app which bound a session per request and wrapped every unit of work within a transaction. The transactions were injected via AOP and a custom attribute, so everything with the required attribute would be wrapped in a transaction that either ended with a commit or a rollback.
Again just to clarify everything was working fine in MSSQL and using Nhibernate Profiler I can see that all the transactions are occurring as expected.
Now I have just added support for Mysql (5.1) (as one environment uses a Mysql db), I have setup all the tables as InnoDB and the table structures are all identical, so I am a bit baffled as to why I get the following error:
Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [SomeRandomEntity#1]Could not synchronize database state with session
Checking on Nhibernate profiler, it is beginning a transaction, firing an update call then bombing out with the error, whereas really it should be committing at that point.
If it helps the sql being fired in the update is (names changed):
UPDATE
some_table SET some_column_1 = 1230697028 /* ?p0 */,
some_column_2 = '2011-07-21T10:58:59.00' /* ?p1 */
WHERE some_column_3 = 1 /* ?p2 */
It is also worth noting that I am not using any Isolation Level on my transactions, I wasnt sure which one would be best for my setup and Mssql didnt complain when I didnt set it.
Has anyone seen any issues like this before, as it is a show stopper at the moment :(
=== Edit ===
Incase it helps here is my connection string:
<connectionStrings>
<add name="NHibernateConnection" connectionString="Server=localhost;Database=xxxxx;Uid=root;" providerName="System.Data.SqlClient"/>
</connectionStrings>
It seems the problem was down to a wonky column that was not inserting somewhere else, which was stopping the update. Although the error was not really clear as to what the underlying problem was.
I'm working on an hibernate Spring Mysql app, sometimes when i make a gethibernateTemplate()get(class,id) i can see a bunch of HQL in the logs and the application hangs, have to kill tomcat. This method reads trhough a 3,000 lines file, and there should be 18 files of these, i've been thinking i probably been looking at this wrong. I need you to help me check this at database level , but i don´t know hot to approach. Maybe my database can´t take so many hits so fast.
I´v looked in phpMyAdmin in the information bout executions time section, i see a red values in:
Innodb_buffer_pool_reads 165
Handler_read_rnd 40
Handler_read_rnd_next 713 k
Created_tmp_disk_tables 8
Opened_tables 30
Can i set the application some how to threat more gently the database ?
How can i check if this is the issue ?
Update
I put a
Thread.sleep(2000);
at the end of each cycle and it made the same numbers of calls (18), so i guess this wont be te reason ? can i discard this approach ?
This is a different view of this question
Hibernate hangs or throws lazy initialization no session or session was closed
trying some different
Update 2
Think it might be the buffer reader reading the file?? file is 44KB, tried this method:
http://code.hammerpig.com/how-to-read-really-large-files-in-java.html
class but did not work.
Update 1 -- do never use a Sleep or something slow within an transaction. A transaction has to be closed as fast as possible, because it can block other database operations (what exactly will be blocked depends on the isolation level)
I do not really understand how the database is related to the files in your usecase. But if the stuff works for the first file and become slow later on, then the problem can be the Hibernate Session (to many objects), in this case start an new Transaction/Hibernate Session for each file.
I rewrote the program so i load the information directly into the database using mysql query LOAD DATA INFILE. It works very fast. Then i updated rows changing some fields i need also with sql queries. I think there is simply too much information at the same time to manage trhough memory and abstractions.
A really weird (for me) problem is occurring lately. In an application that accepts user submitted data the following occurs at random:
Rows from the Database Table where the user submitted data is stored are disappearing.
Please note that there is NO DELETE, DROP, TRUNCATE or other SQL statement issued on the database table except from the INSERT statement.
Could this be a bug of Mysql? Did some research on mysql.com (forums, bugs, etc) and found 2 similar cases but without getting a solid answer (just suggestions).
Some info you might find useful:
Storage Engine: InnoDB
User Submitted Data sanitized and checked for SQL Injection attempts
Appreciate any suggestions, info.
regards,
Here's 3 possibilities:
The data never got to the database in the first place. Something happened elsewhere so the data disappeared. Maybe intermitten network issues, overloaded server, application bug.
A database transaction was not commited, and got rolled back. Maybe a bug in your application code, maybe some invalid data screwd things up, maybe a concurrency exception occured etc.
A bug in mysql.
I'd look at 1. and 2. first.
A table on which you only ever insert (and presumably select) and never update or delete should be really stable. Are you absolutely certain you're protecting thoroughly against SQL injection attacks? Because those could (of course) delete rows and such if successful.
You haven't mentioned which table engine you're using (there are several), but it's well worth running whatever diagnostic tools there are for it on the table in question. For instance, on a MyISAM table, run myisamchk. Or more generically (this works for several table types), use the CHECK TABLE statement.
Have you had issues with the underlying storage? It may be worth checking for those.
Activating binlog and periodically monitoring DELETE queries can help to identify the culprit.
One more case to fullfill the above. There could also be the case of client-side and server-side parts of application. Client-side initiated changes can be processed on the server side with additional code logics.
For example, in our case, local admin panel updated an order information with pay_date = NULL and php-website processed this table to clean-up overdue orders from this table. As php logics were developed by another programmer, it looked strange when orders update resulted in records to disappear after some time.
The same refers to crone operations, working on mysql database in a schedule.