So after reading Performance in PDO / PHP / MySQL: transaction versus direct execution in regards to performance issues I was thinking about I did some research on locking tables in MySQL.
On http://dev.mysql.com/doc/refman/5.0/en/table-locking.html
Table locking enables many sessions to
read from a table at the same time,
but if a session wants to write to a
table, it must first get exclusive
access. During the update, all other
sessions that want to access this
particular table must wait until the
update is done.
This part struck me particularly because most of our queries will be updates rather than inserts. I was wondering if one created a table called foo on which all updates/inserts were carried out and then a view called foo_view (A copy of foo, or perhaps foo and a linkage of several other tables plus foo) on which all selects occurred, would this locking issue still occur?
That is, would SELECT queries on foo_view still have to wait for an update to finish on foo?
Another brief question my colleague asked. Does this affect caching? I.e. if the SELECT is cached will it hit the cache and return results, or will it wait for the lock to finish first?
Your view will experience the same locking as the underlying tables.
From the MySQL Reference page on locking:
MySQL grants table write locks as
follows:
If there are no locks on the table, put a write lock on it.
Otherwise, put the lock request in the write lock queue.
MySQL grants table read locks as
follows:
If there are no write locks on the table, put a read lock on it.
Otherwise, put the lock request in the read lock queue.
It's worth mentioning that this depends on the database engine you are using. MyISAM will follow the steps above and lock the entire table (even if it is split into multiple partitions) where an engine like InnoDB will do row level locking instead.
If you're not reaching the necessary performance benchmarks with MyISAM and you have shown your bottleneck is waiting on table locks via updates, I would suggest changing the storage engine of your table to InnoDB.
Related
Does MySQL InnoDB table wait for write locks even for query such as SELECT COUNT(*) FROM t?
My situation:
I've got table with 50000 rows with many updates (views count in every row). InnoDB should put a write lock on the updated row. But when I make a query with only COUNT(*) on this table, MySQL could answer this query even without waiting for write locks because no UPDATE will change the number of rows.
Thanks a lot!
No, MySql doesn't lock InnoDb tables for queries that only read data from tables.
This is only the case for old MyIsam tables, where all readers must wait until the writer is done and vice versa.
For InnoDb tables they implemented Multiversion concurrency control
In MySql terminology it is called Consistent Nonlocking Reads
In short - when the reader starts the query, the database makes a snapshot of the database at a point in time when the query was started, and the reader (the query) sees only changes made visible (commited) up to this point in time, but doesn't see changes made by later transactions. This allows readers to read data without locking and waiting for writers, but still keeping ACID
There are subtle differences depending on the transaction isolation level, you can find detailed description here: http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html
In short - in read uncommited, read commited and repeatable read modes, all SELECT statements that only read data (SELECTs without FOR UPDATE or LOCK IN SHARE MODE clasues) are performed in a nonlocking fashion.
In serializable mode all transacions are serialized and, depending on autocommit mode, SELECT can be blocked when conflicts with other transactions (when autocommit=true), or is automatically converted to SELECT ... LOCK IN SHARE MODE (when autocommit=false). All details are explained in the above links.
I have a table in my database (actually a few related tables) that get can be manipulated manually from various points through our interface but also automatically from two sources on a continuous basis. The periodic updates can contain huge amounts of data and can result in thousands of inserts/updates. In order to improve performance of the inserts/updates I have used "SET autocommit = 0" around the updates from these automated sources. This has resulted in the desired performance improvement, maybe even more than expected. However the problem now is that if the automated sources overlap or if a manual update is performed very often the database locks up and after a while throws an error:
Lock wait timeout exceeded; try restarting transaction
This may be thrown even in a single statement with autocommit on and no transaction but I guess that is reasonable as well if it conflicts with a transaction. I have read various suggestions, unfortunately there is no ideal solution. I guess my options are:
Try to order updates/inserts on the tables so that locks on all threads are requested in the same order and there is no deadlock. Unfortunately this is no possible, updates need to be applied in the order they are received.
Use LOCK TABLES to serialize transactions. This is theoretically possible but a) Apart from the two automated sources the tables are updated from many points in the system, including triggers, schedules, manually from various interfaces. It would be a nightmare to identify and maintain LOCK tables around all these places and no easy way to know that all have been identified, and b) LOCK TABLES has to lock all tables involved and the updates/inserts though not often but sometimes may need to update many tables as a result of the updates and again need to identify and maintain all the tables that might be updated so that they are included in the LOCK TABLES.
Use a semaphore table before each update in order to achieve the serialization of updates as with LOCK TABLES above but without actually having to use LOCK TABLES. This is an improvement but still has problem a) of LOCK TABLES above.
Any other suggestions? Could the improvement benefits of autocommit = 0 (transactions) be achieved some other way that does not involve locks? Could innodb be configured to actually not lock or lock much less on updates/inserts?
Last resort option may be to move to MyISAM tables. Would this actually achieve performance improvements with heavy inserts/update operations?
Thank you
You can achieve the benefits of autocommit = 0 while still not using long transactions.
a) You can commit the transaction every X statements, assuming that you don't need to rollback the entire transaction
b) instead of using autocommit = 0 you can use ALTER TABLE x DISABLE keys / ALTER TABLE x ENABLE keys before/after the import. This is the reason for the performance improvement of the operation - the non-unique indexes are not updated until the transaction finishes, and then are updated in bulk.
i am reading the book High Performance MySQL, it mentions:
performing one query per table uses table locks more efficiently: the queries
will lock the tables invididually and relatively briefly, instead of locking
them all for a longer time.
MyISAM places table-lock even when selecting something? can someone explain a little bit?
MyISAM has different kinds of locks. A SELECT operation places a READ LOCK on the table. There can be multiple active read locks at any given time, as long as there are no active WRITE LOCKS. Operations that modify the table, eg. INSERT, UPDATE, DELETE or ALTER TABLE place a WRITE LOCK on the table. Write lock can only be placed on a table when there are no active read locks; If there are active read locks, MyISAM queues the write lock to be activated as soon as all active read locks are expired.
Likewise when there's an active write lock, attempting to place a read lock on a table will queue the lock (and the associated query) until write locks have expired on the table.
Ultimately this all means that:
You can have any number of active read locks (also called shared locks)
You can only have one active write lock (also called an exclusive lock)
For more information see: http://dev.mysql.com/doc/refman/5.5/en/internal-locking.html
reko_t provided a good answer, I will try to elaborate on it:
Yes.
You can have EITHER one writer or several readers
Except there is a special case, called concurrent inserts. This means that you can have one thread doing an insert, while one or more threads are doing select (read) queries.
there are a lot of caveats doing this:
it has to be "at the end" of the table - not in a "hole" in the middle
Only inserts can be done concurrently (no updates, deletes)
There is still contention on the single MyISAM key buffer. There is a single key buffer, protected by a single mutex, for the whole server. Everything which uses an index needs to take it (typically several times).
Essentially, MyISAM has poor concurrency. You can try to fake it, but it's bad whichever way you look at it. MySQL / Oracle has made no attempts to improve it recently (looking at the source code, I'm not surprised - they'd only introduce bugs).
If you have a workload with lots of "big" SELECTs which retrieve lots of rows, or are hard in some way, they may often overlap, this may seem ok. But a single row update or delete will block the whole lot of them.
In MySQL:
Every one minute I empty the table and fill it with a new data. Now I want that users should not read data during the fill process, before or after is ok.
How do I achieve this?
Is transaction the way?
Assuming you use a transactional engine (Usually Innodb), clear and refill the table in the same transaction.
Be sure that your readers use READ_COMMITTED or higher transaction isolation level (the default is REPEATABLE READ which is higher).
That way readers will continue to be able to read the old contents of the table during the update.
There are a few things to be careful of:
If the table is so big that it exhausts the rollback area - this is possible if you update the whole of (say) a 1M row table. Of course this is tunable but there are limits
If the transaction fails part way through and gets rolled back - rolling back big transactions is VERY inefficient in InnoDB (it is optimised for commits, not rollbacks)
Be careful of deadlocks and lock wait timeouts, which are more likely if you use big transactions.
You can LOCK your table for the duration of your operation:
http://dev.mysql.com/doc/refman/5.1/en/lock-tables.html
A table lock protects only against
inappropriate reads or writes by other
sessions. The session holding the
lock, even a read lock, can perform
table-level operations such as DROP
TABLE. Truncate operations are not
transaction-safe, so an error occurs
if the session attempts one during an
active transaction or while holding a
table lock.
I don't know enough about the internal row-versioning mechanisms of MySql (or indeed, if there is one), but other databases (Oracle, Postgresql, and more recently, Sql Server) have invested a lot of effort into allowing writers to not block readers, in so far as readers have access to the version of the rows that existed immediately before the update/write process started. Once the update is committed, that version of the row becomes the one made availabe to all readers, thereby avoiding a bottleneck that the above behaviour in MySql will introduce.
This policy ensures that table locking
is deadlock free. There are, however,
other things you need to be aware of
about this policy: If you are using a
LOW_PRIORITY WRITE lock for a table,
it means only that MySQL waits for
this particular lock until there are
no other sessions that want a READ
lock. When the session has gotten the
WRITE lock and is waiting to get the
lock for the next table in the lock
table list, all other sessions wait
for the WRITE lock to be released. If
this becomes a serious problem with
your application, you should consider
converting some of your tables to
transaction-safe tables.
You can load your data into a shadow table as slowly as you like, then instantly swap the shadow and actual with RENAME TABLE:
truncate table shadow; # make sure it is clean to start with
insert into shadow .....; # lots of inserts etc against shadow table
rename table active to temp, shadow to active, temp to shadow;
truncate table shadow; # throw away the old active data
The rename statement is atomic. An intermediate name "temp" is used to help swap the names of temp and active.
This should work with all storage engines.
Rename table - MySQL Manual
I need to use a table for a queuing system. The table will be constantly be updated.
For example, multiple users on my website, will add their files for process, and I heard that when updates occur simultaneously from multiple users, the table becomes non responsive or something like that.
so do I need locking tables in this situation ? how do i apply a lock to a mysql table ?
By constantly be updated do you mean it will be appended to 10,000 times per second? Even for middle-range servers, that still presents 10,000 opportunities per second for the table to be shared by other users.
It's only necessary to lock the table when several dependent operations need to occur as a unit. In most cases, it is sufficient to include the series of operations in a database transaction. If the "constant updates" are merely insert sometable values ( ...), then it will be easy to guarantee transaction consistency.
so do I need locking tables in this situation ?
All tables can be locked, there isn't a special type of table. That said, deal with the issue when you run into deadlocks. Isolation levels are a closely related topic as well.
how do i apply a lock to a mysql table ?
There's the LOCK TABLES syntax - this article covers how & why you'd want to lock tables.
When you do several updates at once, you can get into a deadlock situation. Engines such as InnoDB will detect this and fail one of your transactions (You can retry, but only the whole transaction).
You can avoid this by table locking, but it reduces concurrency.
Engines such as MyISAM (which does not support MVCC or transactions anyway) use table locking implicitly anyway. Such table locks exist only for the duration of a query; they're automatically released soonish after the table isn't needed any more (not necessarily as soon as possible, but quite soon)
I recommend you do not use LOCK TABLE unless you feel you really need to; if you get a deadlock, retry the transaction.