How do I optimize mysql tables to not use locking? Can I alter table to 'turn off' locking all the time.
Situation:
I have app which use database of 15M records. Once weekly scripts doing some task (insert/update/delete) for 20 hours, and app servers that feed data to front end (web server), and that is fine, very small performance loss I see during that time.
Problem:
Once monthly I need to optimize table, since huge number of records is out there it take 1-2 hours to finish this task (starting optimize from mysql command line, or phpMyAdmin, same) and in that period mysql DOESN'T SERVE data to front end (I suppose it is about locking tables for optimize)
Question:
So how to optmize tables to avoid locking, since there is only reading of data (no insert or update) so I suppose 'unlocking' while optimize, in this case can't make any damage?
In case your table engine is InnoDB and MySQL version is > 5.6.17 - the lock won't happen. Actually there will be lock, but for VERY short period.
Prior to Mysql 5.6.17, OPTIMIZE TABLE does not use online DDL.
Consequently, concurrent DML (INSERT, UPDATE, DELETE) is not permitted
on a table while OPTIMIZE TABLE is running, and secondary indexes are
not created as efficiently.
As of MySQL 5.6.17, OPTIMIZE TABLE uses online DDL for regular and
partitioned InnoDB tables. The table rebuild, triggered by OPTIMIZE
TABLE and performed under the cover by ALTER TABLE ... FORCE, is
performed in place and only locks the table for a brief interval,
which reduces downtime for concurrent DML operations.
Optimize Tables Official Ref.
Just better prepare free space that is > than the space currently occupied by your table, because whole table copy can happen for index rebuild.
Related
I am currently trying to figure out why the site I am working on (Laravel 4.2 framework) is really slow at times, and I think it has to do with my database setup. I am not a pro at all so I would assume that where the problem is
My sessions table has roughly 2.2 million records in it, when I run show processlist;, all the queries that take the longest relate to that table.
Here is a picture for example:
Table structure
Surerly I am doing something wrong or it's not index properly? I'm not sure, not fantastic with databases.
We don't see the complete SQL being executed, so we can't recommend appropriate indexes. But if the only predicate on the DELETE statements is on the last_activity column i.e.
DELETE FROM `sessions` WHERE last_activity <= 'somevalue' ;
Then performance of the DELETE statement will likely be improved by adding an index with a leading column of somevalue, e.g.
CREATE INDEX sessions_IX1 ON sessions (last_activity);
Also, if this table is using MyISAM storage engine, then DML statements cannot execute concurrently; DML statements will block while waiting to obtain exclusive lock on the table. The InnoDB storage engine uses row level locking, so some DML operations can be concurrent. (InnoDB doesn't eliminate lock contention, but locks will be on rows and index blocks, rather than on the entire table.)
Also consider using a different storage mechanism (other than MySQL database) for storing and retrieving info for web server "sessions".
Also, is it necessary (is there some requirement) to persist 2.2 million "sessions" rows? Are we sure that all of those rows are actually needed? If some of that data is historical, and isn't specifically needed to support the current web server sessions, we might consider moving the historical data to another table.
I have a MySQL InnoDB database running on the Google App-Engine.
One of the tables has the current date and a user_id as primary key stored with some additional data.
The table had around 7 million rows and I deleted 6 million of them with a DELETE query. Since that any query using this table is much slower than before.
Any ideas what could cause this behavior or how to solve this?
Thanks in advance!
After such a massive delete on innodb you would better to use OPTIMISE table statement
Use OPTIMIZE TABLE in these cases, depending on the type of table:
After doing substantial insert, update, or delete operations on an InnoDB table that has its own .ibd file because it was created with
the innodb_file_per_table option enabled. The table and indexes are
reorganized, and disk space can be reclaimed for use by the operating
system.
After doing substantial insert, update, or delete operations on columns that are part of a FULLTEXT index in an InnoDB table. Set the
configuration option innodb_optimize_fulltext_only=1 first. To keep
the index maintenance period to a reasonable time, set the
innodb_ft_num_word_optimize option to specify how many words to
update in the search index, and run a sequence of OPTIMIZE TABLE
statements until the search index is fully updated.
Prior to optimize, check the table's state using ANALYSE TABLE, and it's indexes using SHOW INDEX. These instructions will provide you with information regarding the "flaws" that OPTIMIZE can fix.
All this is easy to do in phpmyadmin.
I have a large mysql database that receives large volumes of queries, each query takes around 5-10 seconds to perform.
Queries involve checking records, updating records and adding records.
I'm experiencing some significant bottle necks in the query executions, which I believe is due to incoming queries having to 'queue' whilst current queries are using records that these incoming queries need to access.
Is there a way, besides completely reformatting my database structure and SQL queries, to enable simultaneous use of database records by queries?
An INSERT, UPDATE, or DELETE operation locks the relevant tables - myISAM - or rows -InnoDB - until the operation completes. Be sure your query of this type are fastly commited .. and also chechck for you transacation isolating the part with relevant looking ..
For MySQL internal locking see: https://dev.mysql.com/doc/refman/5.5/en/internal-locking.html
Also remeber that in mysql there are differente storage engine with different features eg:
The MyISAM storage engine supports concurrent inserts to reduce
contention between readers and writers for a given table: If a MyISAM
table has no holes in the data file (deleted rows in the middle), an
INSERT statement can be executed to add rows to the end of the table
at the same time that SELECT statements are reading rows from the
table.
https://dev.mysql.com/doc/refman/5.7/en/concurrent-inserts.html
eventually take a look at https://dev.mysql.com/doc/refman/5.7/en/optimization.html
We have a large MySQL 5.5 database in which many rows are inserted daily and never deleted or updated. There are also users querying the live database. Tables are MyISAM.
But it is effectively impossible to run ANALYZE TABLES because it takes way too long. And so the query optimizer will often pick the wrong index. (15 hours, and sometimes crashes the tables.)
We want to try switching to all InnoDB. Will we need to run ANALYZE TABLES or not?
The MySQL docs say:
The cardinality (the number of different key values) in every index of a table
is calculated when a table is opened, at SHOW TABLE STATUS and ANALYZE TABLE and
on other circumstances (like when the table has changed too much).
But that begs the question: when is a table opened? If that means accessed during a connection then we need do nothing special. But I do not think that that is the case for InnoDB.
So what is the best approach? Run ANALYZE TABLE periodically? Perhaps with an increased dive count?
Or will it all happen automatically?
The query users use apps to get the data, so each run is a separate connection. They generally do NOT expect the rows to be up-to-date within just minutes.
Im using fulltext for searches on my site, but there are a lot of updates being done to the table also, which contains 700,000 records. these updates are usually done via cron, by a script that runs every 10 minutes and commits changes from a temporary table where I simply INSERT all the changes for speed purposes, since updating the live table directly is a lot slower.
The table uses MyISAM, so when there are bursts of update queries being done, it overwhelms the server, and it starts lagging, since it locks the entire table for updates.
Are there any storage engines that both have row-level locking, and support FULLTEXT?
With MySQL 5.6 it's now possible to have fulltext indexes on tables with InnoDB engine.
Read more about it here.