I have a table in MySQL 5 (InnoDB) that is used as a daemon Processing Queue, thus it is being accessed very often. It is typical to have around 250 000 records inserted per day. When I select records to be processed, they are read using a FOR UPDATE query to eliminate race conditions (everything is Transaction Based).
Now I am developing a "queue archive" and I have stumbled into a serious dead-lock problem. I need to delete "executed" records from the table as they are being processed (live), yet the table dead-locks every once in a while if I do so (two-three times per day at).
I though of moving towards delayed deletion (once per day at low load times) but this will not eliminate the problem only make it less obvious.
Is there a common-practice in dealing with high-load tables in MySQL?
InnoDB locks all rows it examines, not only those requested.
See this question for more details.
You need to create an index that would exactly match your search condition to get rid of unnecessary locks, and make sure it is used.
Unfortunately, DML queries in MySQL do not accept hints.
Related
For MySQL;
What's the difference between a DROP PARTITION vs a DELETE WHERE query?
Which to use when?
My Scenario:
is the simple matter of deleting data older than a month from a few of my tables, at the end of every month. Tables fill at the slow rate of around 5 entries every second.
Pros / Cons
PARTITIONing with InnoDB requires me to disable my FOREIGN KEYs. So, DELETEing seems better for me. What, if any, advantages would PARTITIONing give me? Is it only the query's execution speed, a.k.a. performance? My deletion query would run only once every month so I don't see a problem with execution time.
For what it's worth, dropping a partition is a data definition language statement. It happens quickly.
DELETE is a data manipulation statement. If you use InnoDB (you probably do) it's transactional. What does that mean?
When you issue the statement, for example,
DELETE FROM tbl WHERE start_date < CURDATE() - INTERVAL 1 MONTH
it means that other programs accessing your database will see either all the rows you're deleting, before your DELETE transaction, or none of them. The operation is called atomic or indivisible -- it appears to happen all at once.
If you delete many rows, this can put a big burden on your database server. It has to accumulate a transaction log containing all the deleted rows, then commit the transaction log all at once, possibly locking out other access.
Your question says you must delete about 13 megarows each month. If you do that with just one DELETE statement, you will put a heavy burden on your database. To reduce the burden when deleting unwanted historical rows, do the DELETE operation in chunks. How? You have a couple of choices.
#Akina suggested this: Do the deletion often enough that you don't delete a large number of rows at once, or
Do the deletion in batches of 1000 rows with a statement like this:
DELETE FROM tbl
WHERE start_date < CURDATE() - INTERVAL 1 MONTH
LIMIT 1000;
and repeat the statement until it deletes no rows.
"5 entries every second" = about 400K/day or 13M/month
DELETING 3M rows in a single statement:
Very slow for that many rows. (Not bad for under 1K rows)
Blocks most activity on the table
Builds a very big list of things for potential "rollback" (in case of power failure)
Scheduled DELETE
Why wait for the end of the month? Do up to 1000 every minute; that will keep up with much less overhead. Be sure to have a suitable index, else it won't work efficiently.
Rather than a monthly task, have a separate task that is continually running, deleting up to 200 rows, then moving on to the next table; eventually repeating. (If it is not keeping up, increase the "LIMIT 200"; if it is too invasive, add a SLEEP in the loop.)
Do not use cron or EVENT to schedule the delete. If, for whatever reason, a Delete run fails to finish before the next Delete, the job could become a train wreck. OTOH, a continually-running job needs a "keep-alive" task to restart it if it dies for any unforeseen reason.
DROP PARTITION
Because of how PARTITIONs are implemented as separate 'tables', DROP PARTITION is similar to DROP TABLE.
Very fast, regardless of the number of rows in the partition. (Well, the OS may show a slight sluggishness for huge files.)
Easy to do if using PARTITION BY RANGE(..).
I recommend that the number of partitions be between 20 and 50; adjust the deletion frequency accordingly. (1-month retention --> daily partitions; 3-month retention --> weekly partitions; 1-year retention --> monthly or weekly; etc.)
When partitioning a table, rethink all the indexes. You may be able to improve a few queries by making use of partition pruning. (But don't expect much.)
More info: Partition
PARTITIONing conflicts with FOREIGN KEYS and some UNIQUE keys. This puts a burden on the programmer to worry about (or ignore) the loss of those constraints.
Here's my blog on other big deletions techniques
I have a mysql table that keep gaining new records every 5 seconds.
The questions are
can I run query on this set of data that may takes more than 5 seconds?
if SELECT statement takes more than 5s, will it affect the scheduled INSERT statement?
what happen when INSERT statement invoked while SELECT is still running, will SELECT get the newly inserted records?
I'll go over your questions and some of the comments you added later.
can I run query on this set of data that may takes more than 5 seconds?
Can you? Yes. Should you? It depends. In a MySQL configuration I set up, any query taking longer than 3 seconds was considered slow and logged accordingly. In addition, you need to keep in mind the frequency of the queries you intend to run.
For example, if you try to run a 10 second query every 3 seconds, you can probably see how things won't end well. If you run a 10 second query every few hours or so, then it becomes more tolerable for the system.
That being said, slow queries can often benefit from optimizations, such as not scanning the entire table (i.e. search using primary keys), and using the explain keyword to get the database's query planner to tell you how it intends to work on that internally (e.g. is it using PKs, FKs, indices, or is it scanning all table rows?, etc).
if SELECT statement takes more than 5s, will it affect the scheduled INSERT statement?
"Affect" in what way? If you mean "prevent insert from actually inserting until the select has completed", that depends on the storage engine. For example, MyISAM and InnoDB are different, and that includes locking policies. For example, MyISAM tends to lock entire tables while InnoDB tends to lock specific rows. InnoDB is also ACID-compliant, which means it can provide certain integrity guarantees. You should read the docs on this for more details.
what happen when INSERT statement invoked while SELECT is still running, will SELECT get the newly inserted records?
Part of "what happens" is determined by how the specific storage engine behaves. Regardless of what happens, the database is designed to answer application queries in a way that's consistent.
As an example, if the select statement were to lock an entire table, then the insert statement would have to wait until the select has completed and the lock has been released, meaning that the app would see the results prior to the insert's update.
I understand that locking database can prevent messing up the SELECT statement.
It can also put a potentially unacceptable performance bottleneck, especially if, as you say, the system is inserting lots of rows every 5 seconds, and depending on the frequency with which you're running your queries, and how efficiently they've been built, etc.
what is the good practice to do when I need the data for calculations while those data will be updated within short period?
My recommendation is to simply accept the fact that the calculations are based on a snapshot of the data at the specific point in time the calculation was requested and to let the database do its job of ensuring the consistency and integrity of said data. When the app requests data, it should trust that the database has done its best to provide the most up-to-date piece of consistent information (i.e. not providing a row where some columns have been updated, but others yet haven't).
With new rows coming in at the frequency you mentioned, reasonable users will understand that the results they're seeing are based on data available at the time of request.
All of your questions are related to locking of table.
Your all questions depend on the way database is configured.
Read : http://www.mysqltutorial.org/mysql-table-locking/
Perform Select Statement While insert statement working
If you want to perform a select statement during insert SQL is performing, you should check by open new connection and close connection every time. i.e If I want to insert lots of records, and want to know that last record has inserted by selecting query. I must have to open connection and close connection in for loop or while loop.
# send a request to store data
insert statement working // take a long time
# select statement in while loop.
while true:
cnx.open()
select statement
cnx.close
//break while loop if you get the result
My website is experiencing issues at checkout. I'm using Magento Enterprise 1.8 and my checkout module is Idev's Onestepcheckout.
The issue we are seeing is that the eav_entity_store table is taking an exceedingly long time (up to 51 seconds) to return an order number to Mage_Eav_Model_Entity_Type.
What I do know is that the query run to get this is a transaction run as 'FOR UPDATE' so the row being accessed is locked until the transaction completes. I've looked at other parts of the code as well as the PHP code throughout the transaction where the row is locked (we're using InnoDB so the lock should be getting released once the transaction is committed) and I'm just not seeing anything there (or in the slow query logs) that should be causing a lock wait anywhere near 51 seconds.
I have considered that requests may be getting stacked up and slowly creeping up in time as they wait, but I'm seeing the query time go from 6ms to 20k ms to 50k ms 1,2,3. It isn't an issue of 100-200 requests stacked up, as there are only a few dozen of these a day.
I'm aware that MySql uses parent locking, but there are no FK's related to this table whatsoever. There are two BTREE indexes that at one point were FK's but have since been Altered (that happened years ago). For those who are un-Magento savy, the eav_entity_store table has less than 50 rows and is only 5 columns wide (4 smallint and a varchar). I seriously doubt tablesize or improper indexing is the culprit. In the spirit of TLDR, however, I will say that the two BTREE indexes are the two columns by which we select from this table.
One possibility is that I may need to replace the two indexes with a compound index, as the ONLY reads to this table are coming from a query that reads (FROM [Column with Index A] AND [Column with Index B]). I simply don't know if row-level locking would prevent this query from accessing another row in the table with the indexes currently on the table.
At this point, I've become convinced that the underlying issue is strictly DB related, but any Magento or MySql advice regarding this would be greatly appreciated. Anybody still actually reading this can hopefully appreciate that I have exhausted a number of options already and am seriously stumped here. Any info that you think may help is welcome. Thanks.
Edit The exact error we are seeing is:
Error message: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction
Issue solved. Wasn't a problem with MySql. For some reason, generation of Invoice Numbers was taking an obscene amount of time. Company doesn't use Invoices from Magento. Turned them off. Problem solved. No full RCA done on what specifically the problem with invoice generation was.
I currently am looking for a solution to a basic problem I have: the deletion of old records.
To explain the situation, I have a table, which I'll call table1, with a reduced number of records. Usually it stays empty, as it is used to relay messages. These messages are read within two seconds of being added to the database, and deleted so that they aren't read again.
However, if one of the clients supposed to receive the messages from table1 goes offline, several messages can become pending. Sometimes hundreds. Sometimes thousands, or even hundreds of thousands, if not more.
Not only does this hurt the client's performance, which will have to process a huge amount of messages, it also hurts the database's which is kept in memory and should keep a minimal amount of records.
Considering the clients check for new messages every second, what would be the best way to delete old records? I've thought about adding timestamps, but won't that hurt the performance: the fact that it has to calculate timestamps when inserting? I've tried it out, and all those queries ended up in the slow queries log.
What would the best solution be? I've thought about something like checking if the table was altered in the past 5 seconds, and if not, we can be safe that all messages that should be relayed have been relayed already, and it can be wiped. But how can this be done?
I've thought about events running every couple of minutes, but I'm not sure how to implement something that would have no (or meaningless) impact on the select/insert/delete queries.
PS: This situation arrives when I noticed that some clients were offline, and there were 8 million messages pending.
EDIT :
I had forgotten to mention that the storage engine is MEMORY, and therefore all records are kept in RAM. That's the main reason I want to get rid of these records: because millions of records which shouldn't even be there, being kept in RAM, has an impact on system resources.
Here is an extract from the error log:
# Query_time: 0.000283 Lock_time: 0.000070 Rows_sent: 0 Rows_examined: 96
SET timestamp=1387199997;
DELETE FROM messages WHERE clientid='100';
[...]
# Query_time: 0.000178 Lock_time: 0.000054 Rows_sent: 0 Rows_examined: 96
SET timestamp=1387199998;
DELETE FROM messages WHERE clientid='14';
So I guess they do have a quite small delay, but is it in any way meaningful in MySQL? I mean, in "real life", 0.0003 could be completely ignored due to its insignificance, can the same be said about MySQL and connections with approximately 10ms ping?
Your question is interesting, but hasn't a lot of detail, so I can only give general points of view.
Firstly - there exist already a number of message queuing solutions which may do what you need out of the box. They hide the underlying implementation of data storage, clean-up etc. and allow you to focus on the application logic. RabbitMQ is a popular open source option.
Secondly, unless you are working with constrained hardware, 100s of thousands of records in a MySQL table is not a performance problem in most cases, nor is generating a time stamp on insert. So, I would recommend building a solution that's obvious and straightforward (and therefore less error prone) - add a timestamp column to your message table, and find a way of removing messages older than 5 minutes. You could add this to the logic which cleans up the records after delivery. As long as your queries are hitting indexed columns, I don't think you have to worry about hundreds of thousands of records.
I would put some energy into creating a performance test suite that allows you to experiment with solutions and see which is really faster. That can be tricky, especially if you want to test scenarios with multiple clients, but you will learn a lot more about the performance characteristics of the app by working through those scenarios.
EDIT:
You can have one column in your table automatically set a timestamp value - I've always found this to be extremely fast. As in - it's never been a problem on very large tables (tens of millions of rows).
I've not got much experience with the memory storage engine - but the MySQL documentation suggests that data modification actions (like insert or update or delete) can be slow due to locking time - that's borne out by your statistics, where the locking time is roughly 30% of the total.
I've had a similar problem.
A couple of questions: First, how long should undelivered messages dwell in the system? Forever? A day? Ten seconds?
Second, what is the consequence of erroneously deleting an undelivered message? Does it cause the collapse of the global banking system? Does it cause a hospital patient not to receive a needed injection? Or does a subsequent message simply cover for the missing one?
The best situation is short dwell time and low error consequence. If the error consequence is high, none of this is wise.
Setting up the solution took several steps for me.
First, write some code to fetch the max id from the messages table.
SELECT MAX(message_id) AS max_message_id FROM message
Then, an hour later, or ten seconds, or a day, or whatever, delete all the messages with id numbers less than the recorded one from the previous run.
DELETE FROM message WHERE message_id <= ?max_message_id
If all is functioning correctly, there won't be anything to delete. But if you have a bunch of stale messages for a client that's gone walkabout, pow, they're gone.
Finally, before putting this into production, wait for a quiet moment in your system, and, just once, issue the command
TRUNCATE TABLE message
to clear out any old rubbish in the table.
You can do this with an event (a stored job in the MySQL database) by creating a little one-row, one-column table to store the max_message_id.
EDIT
You can also alter your table to add a message_time column, in such a way that it gets set automatically whenever you insert a row. Issue these three statements at a time when your system is quiet and you can afford to trash all extant messages.
TRUNCATE TABLE message;
ALTER TABLE message ADD COLUMN message_time TIMESTAMP
NOT NULL
DEFAULT CURRENT_TIMESTAMP;
ALTER TABLE message ADD INDEX message_time (message_time);
Then you can just use a single statement to clean out the old records, like so.
DELETE FROM message WHERE message_time <= NOW() - INTERVAL 1 HOUR
(or whatever interval is appropriate). You should definitely alter an empty or almost-empty table because it takes time to alter lots of rows.
This is a good solution because there's a chance that you don't have to alter your message-processing client code at all. (Of course, if you did SELECT * anywhere, you probably will have to alter it. Pro-tip: never use SELECT * in application code.)
We recently switched our tables to use InnoDB (from MyISAM) specifically so we could take advantage of the ability to make updates to our database while still allowing SELECT queries to occur (i.e. by not locking the entire table for each INSERT)
We have a cycle that runs weekly and INSERTS approximately 100 million rows using "INSERT INTO ... ON DUPLICATE KEY UPDATE ..."
We are fairly pleased with the current update performance of around 2000 insert/updates per second.
However, while this process is running, we have observed that regular queries take very long.
For example, this took about 5 minutes to execute:
SELECT itemid FROM items WHERE itemid = 950768
(When the INSERTs are not happening, the above query takes several milliseconds.)
Is there any way to force SELECT queries to take a higher priority? Otherwise, are there any parameters that I could change in the MySQL configuration that would improve the performance?
We would ideally perform these updates when traffic is low, but anything more than a couple seconds per SELECT query would seem to defeat the purpose of being able to simultaneously update and read from the database. I am looking for any suggestions.
We are using Amazon's RDS as our MySQL server.
Thanks!
I imagine you have already solved this nearly a year later :) but I thought I would chime in. According to MySQL's documentation on internal locking (as opposed to explicit, user-initiated locking):
Table updates are given higher priority than table retrievals. Therefore, when a lock is released, the lock is made available to the requests in the write lock queue and then to the requests in the read lock queue. This ensures that updates to a table are not “starved” even if there is heavy SELECT activity for the table. However, if you have many updates for a table, SELECT statements wait until there are no more updates.
So it sounds like your SELECT is getting queued up until your inserts/updates finish (or at least there's a pause.) Information on altering that priority can be found on MySQL's Table Locking Issues page.