I have a MyISAM table that has to be cleaned once in a while (a total of ~5M rows out of 12M rows are being deleted).
Afterwards, I have to optimize table, and I know that OPTIMIZE TABLE goes faster if I drop indexes first.
The problem is,
ALTER TABLE t1 DISABLE KEYS;
--> here
OPTIMIZE TABLE t1;
--> or here
ALTER TABLE t1 ENABLE KEYS;
MySQL may decide to serve some other queries, it leads to multiple slow non-indexed table scans, delaying further steps.
So how do I lock the table for other threads?
So how do I lock the table for other threads?
You can use LOCK TABLES:
LOCK TABLES t1 WRITE;
But, as noted:
If you use ALTER TABLE on a locked table, it may become unlocked. For example, if you attempt a second ALTER TABLE operation, the result may be an error Table 'tbl_name' was not locked with LOCK TABLES. To handle this, lock the table again prior to the second alteration. See also Section C.5.7.1, “Problems with ALTER TABLE”.
MyISAM lacks transactional support, so this is impossible to do reliably (there are a lot of DDL-level operations that will unlock silently). That's why you should schedule maintenance like this at night so you don't catch the extra load in busy hours.
You have an XY problem though - you're asking about a self-devised solution to a problem you're having, instead of asking for a proper solution to the problem.
Alternative options to solve the underlying problem are:
Doing the cleanup far more frequently - in smaller batches the load shouldn't be really noticeable
Doing the OPTIMIZE step only at night and the cleanups frequently during the day.
Since you consider downtime acceptable apparently (as you suggest locking yourself) you could even consider renaming the table temporarily during the duration - it's a prehistoric way of assuring nobody else touches the table, but it'll work.
If you describe your actual problem in more detail we might be able to give better solutions.
Related
We have a very huge Mysql table which is MyISAM. Whenever we run optimize table command, the table is locked and performance is getting impacted. The table is not read only and hence creating temporary tables and swapping them may not work out. We are not able to partition the table also.
Is there any other way/tool to achieve optimize table functionality without degrading the performance. Any suggestion would be of great help.
Thanks in advance.
http://dev.mysql.com/doc/refman/5.5/en/optimize-table.html
For InnoDB tables, OPTIMIZE TABLE is mapped to ALTER TABLE, which
rebuilds the table (...)
Therefore, I would not expect any improvement in switching to InnoDB, as Quassnoi probably suggests.
By definition, OPTIMIZE TABLE needs some exclusive access to the table, hence the degraded performances during OPTIMIZE'ation
Nevertheless, there could be some steps to take to reduce the time taken by OPTIMIZE, depending on how your table is "huge" :
if your table has many fields, your table might need to be normalized. Conversely, you might want to de-normalize your table by spreading your columns into several "narrower" tables, and establish one-to-one relations.
if your table has many records, implement a "manual" partitionning in your application code. A simple step would be to create an "archive" table that holds rarely updated records. This way you only need to optimize a smaller set of records (the non-archive table).
optimize table command lock the table,it decrease the performance.
you download percona tool kit command to optimize table.
this command not lock the table during optimize table.
use below link :
https://www.percona.com/doc/percona-toolkit/2.1/pt-online-schema-change.html
This is a two part question. When I perform a SELECT on a table, am I locking that table from any usage while the SELECT is running?
If so, what is a way to not lock the table while performing a SELECT statement? I'm using MySQL 4.1.20 MyISAM.
update there is a similar question here Any way to select without causing locking in MySQL? but the answer doesn't work with MyISAM
Yes, with MyISAM tables, the select locks the table for inserts/updates. However several selects can run concurrently (i.e. it applies a read lock). If there's no free space in the middle of the table, inserts will add data to the end of the (internal) storage, and those inserts can still be run concurrently with the selects though.
More info here. Note that MyISAM and InnoDB works very differently in this regard.
There's table locking and row locking. I recommend you read up on database engines MyISAM with Table Locking and InnoDB Row Locking
Think about it: are you changing any table cells when you do your SELECT? If not, there's no need to lock the table and MySQL doesn't. Table locking has to happen at UPDATE time, not at SELECT time.
We have a huge database and inserting a new column is taking too long. Anyway to speed up things?
Unfortunately, there's probably not much you can do. When inserting a new column, MySQL makes a copy of the table and inserts the new data there. You may find it faster to do
CREATE TABLE new_table LIKE old_table;
ALTER TABLE new_table ADD COLUMN (column definition);
INSERT INTO new_table(old columns) SELECT * FROM old_table;
RENAME table old_table TO tmp, new_table TO old_table;
DROP TABLE tmp;
This hasn't been my experience, but I've heard others have had success. You could also try disabling indices on new_table before the insert and re-enabling later. Note that in this case, you need to be careful not to lose any data which may be inserted into old_table during the transition.
Alternatively, if your concern is impacting users during the change, check out pt-online-schema-change which makes clever use of triggers to execute ALTER TABLE statements while keeping the table being modified available. (Note that this won't speed up the process however.)
There are four main things that you can do to make this faster:
If using innodb_file_per_table the original table may be highly fragmented in the filesystem, so you can try defragmenting it first.
Make the buffer pool as big as sensible, so more of the data, particularly the secondary indexes, fits in it.
Make innodb_io_capacity high enough, perhaps higher than usual, so that insert buffer merging and flushing of modified pages will happen more quickly. Requires MySQL 5.1 with InnoDB plugin or 5.5 and later.
MySQL 5.1 with InnoDB plugin and MySQL 5.5 and later support fast alter table. One of the things that makes a lot faster is adding or rebuilding indexes that are both not unique and not in a foreign key. So you can do this:
A. ALTER TABLE ADD your column, DROP your non-unique indexes that aren't in FKs.
B. ALTER TABLE ADD back your non-unique, non-FK indexes.
This should provide these benefits:
a. Less use of the buffer pool during step A because the buffer pool will only need to hold some of the indexes, the ones that are unique or in FKs. Indexes are randomly updated during this step so performance becomes much worse if they don't fully fit in the buffer pool. So more chance of your rebuild staying fast.
b. The fast alter table rebuilds the index by sorting the entries then building the index. This is faster and also produces an index with a higher page fill factor, so it'll be smaller and faster to start with.
The main disadvantage is that this is in two steps and after the first one you won't have some indexes that may be required for good performance. If that is a problem you can try the copy to a new table approach, using just the unique and FK indexes at first for the new table, then adding the non-unique ones later.
It's only in MySQL 5.6 but the feature request in http://bugs.mysql.com/bug.php?id=59214 increases the speed with which insert buffer changes are flushed to disk and limits how much space it can take in the buffer pool. This can be a performance limit for big jobs. the insert buffer is used to cache changes to secondary index pages.
We know that this is still frustratingly slow sometimes and that a true online alter table is very highly desirable
This is my personal opinion. For an official Oracle view, contact an Oracle public relations person.
James Day, MySQL Senior Principal Support Engineer, Oracle
usually new line insert means that there are many indexes.. so I would suggest reconsidering indexing.
Michael's solution may speed things up a bit, but perhaps you should have a look at the database and try to break the big table into smaller ones. Take a look at this: link. Normalizing your database tables may save you loads of time in the future.
I have a MyISAM table with 125M records. I added 25M more records to it via:
ALTER TABLE x DISABLE KEYS;
INSERT INTO x SELECT * FROM y;
ALTER TABLE x ENABLE KEYS;
Currently ALTER TABLE x ENABLE KEYS is in the "Repair with Keycache" state. How fast is this repair operation? Is it at least as fast as the case if I didn't disable the index and let rows be added with indexes updated on the fly or is it slower?
If I kill the query now, DROP all the indexes and then re-create them again to force repair by sort (my buffer sizes are large enough) would I risk losing any data?
If you kill the query while the ALTER TABLE operation is in progress, you risk losing any data that was added to the table after the ALTER TABLE operation began. Additionally, dropping and re-creating the indexes on the table would also cause you to lose any data that was added to the table after the indexes were dropped.
In general, it is a good idea to avoid making any changes to the table while the ALTER TABLE operation is in progress. If you are concerned about the speed of the operation, it may be better to let it complete rather than trying to interrupt it.
In terms of the speed of the operation, whether or not you disable the keys before inserting new rows into the table will not affect the speed of the ALTER TABLE operation. The ALTER TABLE operation will run at roughly the same speed regardless of whether the keys were disabled or not. The speed of the operation will depend on several factors, including the size of the table and the speed of the underlying hardware.
If you want to force the ALTER TABLE operation to use the "Repair by sort" method, you can do so by setting the myisam_repair_threads system variable to a value greater than 1 before running the ALTER TABLE operation. This will cause the operation to use the "Repair by sort" method, which may be faster in some cases. However, keep in mind that this will also use more system resources, so it may not be appropriate for all situations.
It is always a good idea to back up your data before making any changes to a table, in case something goes wrong. This way, you can restore the data from the backup if needed.
I need to use a table for a queuing system. The table will be constantly be updated.
For example, multiple users on my website, will add their files for process, and I heard that when updates occur simultaneously from multiple users, the table becomes non responsive or something like that.
so do I need locking tables in this situation ? how do i apply a lock to a mysql table ?
By constantly be updated do you mean it will be appended to 10,000 times per second? Even for middle-range servers, that still presents 10,000 opportunities per second for the table to be shared by other users.
It's only necessary to lock the table when several dependent operations need to occur as a unit. In most cases, it is sufficient to include the series of operations in a database transaction. If the "constant updates" are merely insert sometable values ( ...), then it will be easy to guarantee transaction consistency.
so do I need locking tables in this situation ?
All tables can be locked, there isn't a special type of table. That said, deal with the issue when you run into deadlocks. Isolation levels are a closely related topic as well.
how do i apply a lock to a mysql table ?
There's the LOCK TABLES syntax - this article covers how & why you'd want to lock tables.
When you do several updates at once, you can get into a deadlock situation. Engines such as InnoDB will detect this and fail one of your transactions (You can retry, but only the whole transaction).
You can avoid this by table locking, but it reduces concurrency.
Engines such as MyISAM (which does not support MVCC or transactions anyway) use table locking implicitly anyway. Such table locks exist only for the duration of a query; they're automatically released soonish after the table isn't needed any more (not necessarily as soon as possible, but quite soon)
I recommend you do not use LOCK TABLE unless you feel you really need to; if you get a deadlock, retry the transaction.