We are encountering a lot of deadlocks, and while we found out that the problematic Foreign Key, we could not understand why exactly it happened.
I looked into the performance_schema tables to understand but I dont think I have sufficient knowledge. Here's what I thought would help me debug deadlock:
Look into the transation ID/Thread ID of the two conflicting transactions (available from output of Show Engine innodb status)
I want to see all the statements for the two transaction, after one has failed and one has succeeded. Is that even possible?
Once I have that info, I can get more clarity, and hopefully pinpoint why the deadlock happened
I was focusing on events_statements_history_long, but with the Thread ID I got in step 1, I got no rows in response withing a minute of deadlock.
Is this the correct approach? If not, where I'm going wrong? If yet, Is there relevant literature out there which can give more clarity?
Look at the output of Show Engine innodb status as you said.
The latest deadlock information section will show you the two statements that caused the latest deadlock in the system.
If you are encountering many different deadlocks, you may want to enable innodb_print_all_deadlocks to view them in the mysqld error log.
You will be able to see the statements and lock types on which table that resulted in a deadlock.
Related
I have a piece of (Perl) code, of which I have multiple instances running at the same time, all with a different - unique - value for a variable $dsID. Nearly all of them keep falling over when they try to execute the following (prepared) SQL statement:
DELETE FROM ssRates WHERE ssID IN (SELECT id FROM snapshots WHERE dsID=?)
returning the error:
Lock wait timeout exceeded; try restarting transaction
Which sounds clear enough, except for a few things.
I have autocommit enabled, and am not using (explicit) transactions.
I'm using InnoDB which is supposed to use row-level locking.
The argument passed as $dsID is unique to each code, so there should be no conflicting locks to get into deadlocks.
Actually, at present, there are no rows that match the inner SELECT clause (I have verified this).
Given these things, I cannot understand why I am getting lock problems -- no locks should be waiting on each other, and there is no scope for deadlocks! (Note, though, that the same script later on does insert into the ssRates table, so some instances of the code may be doing that).
Having googled around a little, this looks like it may be a "gap locking" phenomenon, but I'm not entirely sure why, and more to the point, I'm not sure what the right solution is. I have some possible workarounds, -- the obvious one being to split the process up: do the select clause, and then loop over results giving delete command. But really, I'd like to understand this otherwise I'm going to end up in this mess again!
So I have two questions for you friendly experts.
Is this a gap-locking thing?
If not - what is it? If yes -- why. I can't see how this condition matches the gap lock definition.
(NB, server is running MariaDB: 5.5.68-MariaDB; in case this is something fixed in newer versions).
Logs showing that from time to time this error is raised.
I'm reading the docs and it's very confusing because we're not locking any tables to do inserts and we have no transactions beyond individual SQL calls.
So - might this be happening because we're running out of the mySQL connection pool in Node? (We've set it to something like 250 simultaneous connections).
I'm trying to figure out how to replicate this but having no luck.
Every query not run within an explicit transaction runs in an implicit transaction that immediately commits when the query finishes or rolls back if an error occurs... so, yes, you're using transactions.
Deadlocks occur when at least two queries are in the process of acquiring locks, and each of them holds row-level locks that they happened to acquire in such an order that they each now need another lock that the other one holds -- so, they're "deadlocked." An infinite wait condition exists between the running queries. The server notices this.
The error is not so much a fault as it is the server saying, "I see what you did, there... and, you're welcome, I cleaned it up for you because otherwise, you would have waited forever."
What you aren't seeing is that there are two guilty parties -- two different queries that caused the problem -- but only one of them is punished. The query that has accomplished the least amount of work (admittedly, this concept is nebulous) will be killed with the deadlock error, and the other query happily proceeds along its path, having no idea that it was the lucky survivor.
This is why the deadlock error message ends with "try restarting transaction" -- which, if you aren't explicitly using transacrions, just means "run your query again."
See https://dev.mysql.com/doc/refman/5.6/en/innodb-deadlocks.html and examine the output of SHOW ENGINE INNODB STATUS;, which will show you the other query -- the one that helped cause the deadlock but that was not killed -- as well as the one that was.
My website is experiencing issues at checkout. I'm using Magento Enterprise 1.8 and my checkout module is Idev's Onestepcheckout.
The issue we are seeing is that the eav_entity_store table is taking an exceedingly long time (up to 51 seconds) to return an order number to Mage_Eav_Model_Entity_Type.
What I do know is that the query run to get this is a transaction run as 'FOR UPDATE' so the row being accessed is locked until the transaction completes. I've looked at other parts of the code as well as the PHP code throughout the transaction where the row is locked (we're using InnoDB so the lock should be getting released once the transaction is committed) and I'm just not seeing anything there (or in the slow query logs) that should be causing a lock wait anywhere near 51 seconds.
I have considered that requests may be getting stacked up and slowly creeping up in time as they wait, but I'm seeing the query time go from 6ms to 20k ms to 50k ms 1,2,3. It isn't an issue of 100-200 requests stacked up, as there are only a few dozen of these a day.
I'm aware that MySql uses parent locking, but there are no FK's related to this table whatsoever. There are two BTREE indexes that at one point were FK's but have since been Altered (that happened years ago). For those who are un-Magento savy, the eav_entity_store table has less than 50 rows and is only 5 columns wide (4 smallint and a varchar). I seriously doubt tablesize or improper indexing is the culprit. In the spirit of TLDR, however, I will say that the two BTREE indexes are the two columns by which we select from this table.
One possibility is that I may need to replace the two indexes with a compound index, as the ONLY reads to this table are coming from a query that reads (FROM [Column with Index A] AND [Column with Index B]). I simply don't know if row-level locking would prevent this query from accessing another row in the table with the indexes currently on the table.
At this point, I've become convinced that the underlying issue is strictly DB related, but any Magento or MySql advice regarding this would be greatly appreciated. Anybody still actually reading this can hopefully appreciate that I have exhausted a number of options already and am seriously stumped here. Any info that you think may help is welcome. Thanks.
Edit The exact error we are seeing is:
Error message: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction
Issue solved. Wasn't a problem with MySql. For some reason, generation of Invoice Numbers was taking an obscene amount of time. Company doesn't use Invoices from Magento. Turned them off. Problem solved. No full RCA done on what specifically the problem with invoice generation was.
My code is a bit of a mess, I'm not sure where the problem is, but I'm getting deadlocks without using any transactions or table locking. Any information about this would help.
I've looked up deadlocks and it seems the only way to cause them is by using transactions.
Error Number: 1213
Deadlock found when trying to get lock; try restarting transaction
UPDATE `x__cf_request` SET `contact_success` = 1, `se_engine_id` = 0, `is_fresh` = 1 WHERE `id` = '28488'
Edit: Why downvotes? It's a valid question. If it's impossible just say why, so that other people can see when they run into this issue.
In InnoDB each statement is run in a transation; BEGIN and autocommit=0 are used for multi-statement transactions. Having said that, the deadlock happens between different transactions.
It seems you don't have index on the id field, or more than one record have the same id. If not, than you have an index-gap locking in place. To diagnose further, you need to provide the output of SHOW ENGINE InnoDB STATUS
In my application, I have two queries that occur from time to time (from different processes), that cause a deadlock.
Query #1
UPDATE tblA, tblB SET tblA.varcharfield=tblB.varcharfield WHERE tblA.varcharfield IS NULL AND [a few other conditions];
Query #2
INSERT INTO tmp_tbl SELECT * FROM tblA WHERE [various conditions];
Both of these queries take a significant time, as these tables have millions of rows. When query #2 is running, it seems that tblA is locked in mode S. It seems that query #1 requires an X lock. Since this is incompatible with an S lock, query #1 waits for up to 30 seconds, at which point I get a deadlock:
Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
Based on what I've read in the documentation, I think I have a couple options:
Set an index on tblA.varcharfield. Unfortunately, I think that this would require a very large index to store the field of varchar(512). (See edit below... this didn't work.)
Disable locking with SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
. I don't understand the implications of this, and am worried about corrupt data. I don't use explicit transactions in my application currently, but I might at some point in the future.
Split my time-consuming queries into small pieces so that they can queue and run in MySQL without reaching the 30-second timeout. This wouldn't really fix the heart of the issue, and I am concerned that when my database servers get busy that the problem will occur again.
Simply retrying queries over and over again... not an option I am hoping for.
How should I proceed? Are there alternate methods I should consider?
EDIT: I have tried setting an index on varcharfield, but the table is still locking. I suspect that the locking happens when the UPDATE portion is actually executing. Are there other suggestions to get around this problem?
A. If we assume that indexing varcharField takes a lot of disk space and adding new column will not hit you hard I can suggest the following approach:
create new field with datatype "tinyint"
index it.
this field will store 0 if varcharField is null and 1 - otherwise.
rewrite the first query to do update relying on new field. In this case it will not cause entire table locking.
Hope it helps.
You can index only part of the varchar column, it will still work, and will require less space. Just specify index size:
CREATE INDEX someindex ON sometable (varcharcolumn(32))
I was able to solve the issue by adding explicit LOCK TABLE statements around both queries. This turned out to be a better solution, since each query affects so many records, and that both of these are background processes. They now wait on each other.
http://dev.mysql.com/doc/refman/5.0/en/lock-tables.html
While this is an okay solution for me, it obviously isn't the answer for everyone. Locking with WRITE means that you cannot READ. Only a READ lock will allow others to READ.