I seem to be having slow inserts, updates and deletes on all tables on a specific database with MySQL. Not a lot of data in those tables (from 2k to 20k). Small number of columns (5-10), indexes (two of them), and no duplicate index issue. I'm running MySQL 5.0.45 with MyISAM.
I run the following query and it takes about 5-7 seconds:
UPDATE accounts SET updated_at = '2010-10-09 11:22:53' WHERE id = 8;
Selects seem to come back right away.
Explain gives me the following:
+----+-------------+----------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+-------+---------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | accounts | index | NULL | PRIMARY | 4 | NULL | 1841 | Using index |
+----+-------------+----------+-------+---------------+---------+---------+------+------+-------------+
The profiler doesn't show any significant data for anything other than a seemingly high number of context switches:
+----------------------+----------+-------------------+---------------------+
| Status | Duration | Context_voluntary | Context_involuntary |
+----------------------+----------+-------------------+---------------------+
| (initialization) | 0.000057 | 0 | 0 |
| checking permissions | 0.000008 | 0 | 0 |
| Opening tables | 0.000013 | 0 | 0 |
| System lock | 0.000005 | 0 | 0 |
| Table lock | 0.000005 | 0 | 0 |
| init | 0.000061 | 0 | 0 |
| Updating | 0.000101 | 0 | 0 |
| end | 7.957233 | 7951 | 2 |
| query end | 0.000008 | 0 | 0 |
| freeing items | 0.000011 | 0 | 0 |
| closing tables | 0.000007 | 1 | 0 |
| logging slow query | 0.000002 | 0 | 0 |
+----------------------+----------+-------------------+---------------------+
This might also help:
+----------------------+----------+-----------------------+---------------+-------------+
| Status | Duration | Source_function | Source_file | Source_line |
+----------------------+----------+-----------------------+---------------+-------------+
| (initialization) | 0.000057 | check_access | sql_parse.cc | 5306 |
| checking permissions | 0.000008 | open_tables | sql_base.cc | 2629 |
| Opening tables | 0.000013 | mysql_lock_tables | lock.cc | 153 |
| System lock | 0.000005 | mysql_lock_tables | lock.cc | 162 |
| Table lock | 0.000005 | mysql_update | sql_update.cc | 167 |
| init | 0.000061 | mysql_update | sql_update.cc | 429 |
| Updating | 0.000101 | mysql_update | sql_update.cc | 560 |
| end | 7.957233 | mysql_execute_command | sql_parse.cc | 5122 |
| query end | 0.000008 | mysql_parse | sql_parse.cc | 6116 |
| freeing items | 0.000011 | dispatch_command | sql_parse.cc | 2146 |
| closing tables | 0.000007 | log_slow_statement | sql_parse.cc | 2204 |
| logging slow query | 0.000002 | dispatch_command | sql_parse.cc | 2169 |
+----------------------+----------+-----------------------+---------------+-------------+
Additional info:
It's running on a CentOS-5 VPS with 4 gigs of ram guaranteed. No index on the updated_at column and not triggers anywhere.
[New things that I tried]
Created a new table (using like)
running innodb and inserted all
records from one of the affected
tables. (same problem)
Backed up the database and restored it to a
different database within the same
server instance. (same problem)
Restored that same backup to my
local machine and I didn't have a
problem.
Tried another database
within the same mysql server
instance that has the problem
database and the other database (a
Wordpress DB) ran
updates/inserts/deletes just fine.
Restarted mysqld and restarted the entire server last night (same problem)
Updated MySQL to version 5.0.77 (same problem)
Deleted all indexes from one of the affected tables (same problem)
Any ideas what to look at next or what might be the issue? Seems to be more of a recent problem though I can't say when it started to show up exactly.
If you have variable length rows, you might need to run OPTIMIZE TABLE occasionally.
Finally found the answer. That database was somehow missing the MYD and MYI files and still running. Not sure how that's possible considering that the MYD file holds the data for MyISAM tables but that was causing the slow inserts/updates/deletes.
I ran an ALTER TABLE to set the engine to MyISAM (which it already was) and it recreated those files. Updates/inserts/deletes running fast again!
Related
MySQL gurus
We have readonly dataset, which contains around 4 million rows. This dataset is represented as 1 InnoDB table with a bunch of indices, like this:
CREATE TABLE IF NOT EXISTS dim_users (
ds varchar(10),
user_id varchar(32),
date_joined varchar(10),
payer boolean,
source varchar(12),
country varchar(2),
is1dayactive boolean,
is7dayactive boolean,
is28dayactive boolean,
istest boolean
) ENGINE = InnoDB;
ALTER TABLE dim_users ADD INDEX (ds);
ALTER TABLE dim_users ADD INDEX (user_id);
ALTER TABLE dim_users ADD INDEX (date_joined);
ALTER TABLE dim_users ADD INDEX (source);
ALTER TABLE dim_users ADD INDEX (country);
MySQL8 is running on Ubuntu 18.04 in servers.com provider on a cloud server with 8GB of RAM and swap turned off.
The database has default config, except
[mysqld]
innodb-buffer-pool-size=6G
read_only=1
The problem is that aggregation queries like select count(*) from dim_users or select country,count(*) from dim_users group by 1 order by 1 desc; take too much time (up to 12 seconds) comparing to default Postgres 11 setup on the same vhost.
Default psql11 performs the last query for less than 2 seconds.
We were trying to profile and analyze this request on MySQL:
Explain:
mysql> explain analyze select country,count(*) from dim_users group by 1 order by 1 desc;
+---------------+
| EXPLAIN
+---------------+
| -> Group aggregate: count(0) (actual time=241.091..14777.679 rows=11 loops=1)
| -> Index scan on dim_users using country (reverse) (cost=430171.90 rows=4230924) (actual time=0.054..11940.166 rows=4228692 loops=1)
|
+---------------+
1 row in set (14.78 sec)
Profile:
mysql> SELECT SEQ,STATE,DURATION,SWAPS,SOURCE_FUNCTION,SOURCE_FILE,SOURCE_LINE FROM INFORMATION_SCHEMA.PROFILING WHERE QUERY_ID=1;
+-----+--------------------------------+-----------+-------+-------------------------+----------------------+-------------+
| SEQ | STATE | DURATION | SWAPS | SOURCE_FUNCTION | SOURCE_FILE | SOURCE_LINE |
+-----+--------------------------------+-----------+-------+-------------------------+----------------------+-------------+
| 2 | starting | 0.000115 | 0 | NULL | NULL | NULL |
| 3 | Executing hook on transaction | 0.000008 | 0 | launch_hook_trans_begin | rpl_handler.cc | 1119 |
| 4 | starting | 0.000013 | 0 | launch_hook_trans_begin | rpl_handler.cc | 1121 |
| 5 | checking permissions | 0.000010 | 0 | check_access | sql_authorization.cc | 2176 |
| 6 | Opening tables | 0.000050 | 0 | open_tables | sql_base.cc | 5591 |
| 7 | init | 0.000012 | 0 | execute | sql_select.cc | 677 |
| 8 | System lock | 0.000020 | 0 | mysql_lock_tables | lock.cc | 331 |
| 9 | optimizing | 0.000007 | 0 | optimize | sql_optimizer.cc | 282 |
| 10 | statistics | 0.000046 | 0 | optimize | sql_optimizer.cc | 502 |
| 11 | preparing | 0.000035 | 0 | optimize | sql_optimizer.cc | 583 |
| 12 | executing | 13.552128 | 0 | ExecuteIteratorQuery | sql_union.cc | 1409 |
| 13 | end | 0.000037 | 0 | execute | sql_select.cc | 730 |
| 14 | query end | 0.000008 | 0 | mysql_execute_command | sql_parse.cc | 4606 |
| 15 | waiting for handler commit | 0.000018 | 0 | ha_commit_trans | handler.cc | 1589 |
| 16 | closing tables | 0.000015 | 0 | mysql_execute_command | sql_parse.cc | 4657 |
| 17 | freeing items | 0.000039 | 0 | mysql_parse | sql_parse.cc | 5330 |
| 18 | cleaning up | 0.000020 | 0 | dispatch_command | sql_parse.cc | 2184 |
+-----+--------------------------------+-----------+-------+-------------------------+----------------------+-------------+
As you can see, group by query takes ages to complete.
So, my questions are the following.
I've never been using MySQL before, my main DB has always been Postgresql for years and I don't know whether it's normal behaviour for this DB or not - it's very slow when we want to execute group by queries.
Otherwise, if I'm missing something very important here, eg DB or OS configuration, could you please give me a pieces of advice on how to properly setup MySQL for read only purposes.
Thank you!
I just updated my MySQL version to 5.7. A SELECT query that has four INNER-JOINS and that previously took around 3 seconds to execute is now taking so long that I can't even keep track of it. A bit of profiling shows that the 'Send Data' part is taking too long. Can someone tell me what is going wrong? Here's some data. Note that the query is still running at this point in time:
+----------------------+-----------+
| Status | Duration |
+----------------------+-----------+
| starting | 0.001911 |
| checking permissions | 0.000013 |
| checking permissions | 0.000003 |
| checking permissions | 0.000003 |
| checking permissions | 0.000006 |
| Opening tables | 0.000030 |
| init | 0.000406 |
| System lock | 0.000018 |
| optimizing | 0.000019 |
| statistics | 0.000509 |
| preparing | 0.000052 |
| executing | 0.000004 |
| Sending data | 31.881794 |
| end | 0.000021 |
| query end | 0.003540 |
| closing tables | 0.000032 |
| freeing items | 0.000214 |
| cleaning up | 0.000028 |
+----------------------+-----------+
Here's the output of EXPLAIN:
+----+-------------+--------------------+------------+------+---------------+------------+---------+-------+---------+----------+----------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------+------------+------+---------------+------------+---------+-------+---------+----------+----------------------------------------------------+
| 1 | SIMPLE | movie_data_primary | NULL | ref | cinestopId | cinestopId | 26 | const | 1 | 100.00 | NULL |
| 1 | SIMPLE | mg | NULL | ALL | NULL | NULL | NULL | NULL | 387498 | 10.00 | Using where; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | crw | NULL | ALL | NULL | NULL | NULL | NULL | 1383452 | 10.00 | Using where; Using join buffer (Block Nested Loop) |
| 1 | SIMPLE | cst | NULL | ALL | NULL | NULL | NULL | NULL | 2184556 | 10.00 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+--------------------+------------+------+---------------+------------+---------+-------+---------+----------+----------------------------------------------------+
Looks like indexing problem when you upgrade the msssql version-
Documentation says-
If you perform a binary upgrade without dumping and reloading tables,
you cannot upgrade directly from MySQL 4.1 to 5.1 or higher. This
occurs due to an incompatible change in the MyISAM table index
formatin MySQL 5.0. Upgrade from MySQL 4.1 to 5.0 and repair all
MyISAM tables. Then upgrade from MySQL 5.0 to 5.1 and check and
repair your tables.Modifications to the handling of character sets or
collations might change the character sort order, which causes the
ordering of entries in any index that uses an affected character
set or collation to be incorrect. Such changes result in several
possible problems: Comparison results that differ from previous
results
Inability to find some index values due to misordered index entries
Misordered ORDER BY results
Tables that CHECK TABLE reports as being in need of repair
Check for the links-
1)checking-table-incompatibilities
2)check-table
3)rebuilding-tables
I have MyISAM tables with a big number of rows - 10M to 500M. The tables are used to store time series data infrequently and I would like to optimize it for SELECT, which I am doing throung 2 indexes: the epoch and a classifier column (which contain a few thousand categories).
My issue is that the first SELECT I do for a particular category is quite long (10 to 50 sec) while the subsequent ones are really fast, even when using SQL_NO_CACHE. Such a query will typically return between 100,000 and 1M elements.
Profiling shows that MySQL spends a lot of time "Sending data". This would indicate that a lot of the time spent is done doing disk I/O. But I do not really understand where the bottleneck is:
Is the bottleneck in the BTREE read? The tree has only a few thousands nodes and then less than 1M points in the selected node. I cannot believe it would take 30 seconds to do that on a modern machine, even with old school hard drives.
Is it in reading the rows in the table? That is again less than a million rows with an average length of ~40bytes.
Something else I am not accounting for?
Here are the query results:
mysql> SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63;
+---------------+
| COUNT(`Time`) |
+---------------+
| 450619 |
+---------------+
1 row in set (28.67 sec)
mysql> SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63;
+---------------+
| COUNT(`Time`) |
+---------------+
| 450619 |
+---------------+
1 row in set (2.20 sec)
mysql> SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63;
+---------------+
| COUNT(`Time`) |
+---------------+
| 450619 |
+---------------+
1 row in set (0.88 sec)
mysql> SHOW PROFILES;
+----------+-------------+-----------------------------------------------------------------------------------+
| Query_ID | Duration | Query |
+----------+-------------+-----------------------------------------------------------------------------------+
| 1 | 28.66720725 | SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63 |
| 2 | 2.19872350 | SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63 |
| 3 | 0.87811475 | SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63 |
+----------+-------------+-----------------------------------------------------------------------------------+
3 rows in set (0.00 sec)
mysql> SHOW PROFILE FOR QUERY 1;
+----------------------+-----------+
| Status | Duration |
+----------------------+-----------+
| starting | 0.000113 |
| checking permissions | 0.000010 |
| Opening tables | 0.000027 |
| System lock | 0.000017 |
| init | 0.000030 |
| optimizing | 0.000018 |
| statistics | 0.055731 |
| preparing | 0.000024 |
| executing | 0.000008 |
| Sending data | 28.611161 |
| end | 0.000019 |
| query end | 0.000005 |
| closing tables | 0.000014 |
| freeing items | 0.000021 |
| logging slow query | 0.000003 |
| logging slow query | 0.000004 |
| cleaning up | 0.000005 |
+----------------------+-----------+
17 rows in set (0.00 sec)
mysql> SHOW PROFILE FOR QUERY 2;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| starting | 0.000105 |
| checking permissions | 0.000011 |
| Opening tables | 0.000036 |
| System lock | 0.000015 |
| init | 0.000028 |
| optimizing | 0.000019 |
| statistics | 0.032255 |
| preparing | 0.000024 |
| executing | 0.000007 |
| Sending data | 2.166140 |
| end | 0.000020 |
| query end | 0.000004 |
| closing tables | 0.000014 |
| freeing items | 0.000025 |
| logging slow query | 0.000003 |
| cleaning up | 0.000018 |
+----------------------+----------+
16 rows in set (0.00 sec)
mysql> SHOW PROFILE FOR QUERY 3;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| starting | 0.000071 |
| checking permissions | 0.000009 |
| Opening tables | 0.000018 |
| System lock | 0.000012 |
| init | 0.000021 |
| optimizing | 0.000014 |
| statistics | 0.000059 |
| preparing | 0.000020 |
| executing | 0.000007 |
| Sending data | 0.877795 |
| end | 0.000021 |
| query end | 0.000004 |
| closing tables | 0.000015 |
| freeing items | 0.000029 |
| logging slow query | 0.000015 |
| cleaning up | 0.000006 |
+----------------------+----------+
16 rows in set (0.00 sec)
The particular table I am querying contains 107,407,213 rows with a data length of 4,237,427,600 bytes and an index length of 4,255,541,248 bytes. I optimized it yesterday and there where no added data since then.
If the query is I/O bound, I can always switch to a SSD and I also have the option to store the time index as an integer instead of a double. But so far I do not understand where my bottleneck is and I would like to avoid major changes before knowing more.
SQL_NO_CACHE means that mysql should not use the query cache.
The disk/buffer cache is still used, which is why the first query takes longer time, and the subseqent queries are faster.
I am running a simple query on a simple table on the same machine as the server running 5.5. It is taking 22sec to return ~7000 rows from a 20 million row table. Upon profiling most of the time is taken up by multiple "Waiting for query cache lock". What is "Waiting for query cache lock" and why is this query taking so long? Is it something with the way I set up the server?
Here is the profile (note the time for the operation is actually from the row below as stated here):
mysql> show profile for query 4;
+--------------------------------+----------+
| Status | Duration |
+--------------------------------+----------+
| starting | 0.000015 |
| Waiting for query cache lock | 0.000003 |
| checking query cache for query | 0.000045 |
| checking permissions | 0.000006 |
| Opening tables | 0.000027 |
| System lock | 0.000007 |
| Waiting for query cache lock | 0.000032 |
| init | 0.000018 |
| optimizing | 0.000008 |
| statistics | 0.033109 |
| preparing | 0.000019 |
| executing | 0.000002 |
| Sending data | 4.575480 |
| Waiting for query cache lock | 0.000005 |
| Sending data | 5.527728 |
| Waiting for query cache lock | 0.000005 |
| Sending data | 5.743041 |
| Waiting for query cache lock | 0.000004 |
| Sending data | 6.191706 |
| end | 0.000007 |
| query end | 0.000005 |
| closing tables | 0.000028 |
| freeing items | 0.000008 |
| Waiting for query cache lock | 0.000002 |
| freeing items | 0.000182 |
| Waiting for query cache lock | 0.000002 |
| freeing items | 0.000002 |
| storing result in query cache | 0.000004 |
| logging slow query | 0.000001 |
| logging slow query | 0.000002 |
| cleaning up | 0.000003 |
+--------------------------------+----------+
Here is the table:
mysql> SHOW CREATE TABLE prvol;
"Table","Create Table"
"prvol","CREATE TABLE `prvol` (
`ticker` varchar(10) DEFAULT NULL,
`date` date DEFAULT NULL,
`close` float unsigned DEFAULT NULL,
KEY `Index 1` (`date`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1"
Here is the query:
mysql> select close from prvol where date = '20100203';
EDIT: After running with SQL_NO_CACHE, all the time is now in the execution. Could this just be normal for a table this size on a 2.4GHz, 3GB ram machine?
+----------------------+-----------+
| Status | Duration |
+----------------------+-----------+
| starting | 0.000052 |
| checking permissions | 0.000007 |
| Opening tables | 0.000027 |
| System lock | 0.000008 |
| init | 0.000019 |
| optimizing | 0.000008 |
| statistics | 0.034766 |
| preparing | 0.000011 |
| executing | 0.000002 |
| Sending data | 22.071324 |
| end | 0.000012 |
| query end | 0.000005 |
| closing tables | 0.000020 |
| freeing items | 0.000170 |
| logging slow query | 0.000001 |
| logging slow query | 0.000003 |
| cleaning up | 0.000004 |
+----------------------+-----------+
EDIT: Include results of explain.
mysql> explain extended select cp from prvol where date = '20100208';
+----+-------------+-------+------+---------------+---------+---------+-------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref |rows | filtered | Extra |
+----+-------------+-------+------+---------------+---------+---------+-------+------+----------+-------------+
| 1 | SIMPLE | prvol | ref | Index 1 | Index 1 | 4 | const |6868 | 100.00 | Using where |
+----+-------------+-------+------+---------------+---------+---------+-------+------+----------+-------------+
1 row in set, 1 warning (0.08 sec)
I solved my slow query problem. To summarize the problem, it was taking 22sec to query 7000 rows from a 20mln row, 1.7GB indexed table. The problem was that the cache was too small and the query had to go to disk for every query. I would think the disk access would be faster than what I was seeing because I was going off an indexed column so the amount of data read off disk should have been small. But I'm guessing there is a lot of overhead with accessing the InnoDB storage on disk.
Once I set innodb_buffer_pool_size=1024M in the my.ini file, the initial query would take a long time, but all subsequent queries would finish in under a second.
Unfortunately, the profiling didn't really help.
This is a known problem with MySQL. It's really well described here:
https://web.archive.org/web/20160129162137/http://www.psce.com/blog/kb/how-query-cache-can-cause-performance-problems/
Query cache can help you a lot but at the same time it can become a bottleneck.
Is there an explanation of these statuses anywhere?
http://dev.mysql.com/tech-resources/articles/using-new-query-profiler.html
my specific question is in regards to this query:
select count(*)
from 135_5m.record_updates u, 135_5m.records r
where r.record_id = u.record_id and
(u.date_updated > null or null is null) and
u.date_updated <= '2011-01-03';
which returns a single number - 4053904. So why would the majority of the time be spent in "Sending data"? Is it just poorly named? Surely "Sending data" must be doing more than just sending data?
+--------------------------------+-----------+-------+
| Status | Duration | Swaps |
+--------------------------------+-----------+-------+
| starting | 0.000224 | 0 |
| checking query cache for query | 0.000188 | 0 |
| checking permissions | 0.000012 | 0 |
| checking permissions | 0.000017 | 0 |
| Opening tables | 0.000036 | 0 |
| System lock | 0.000015 | 0 |
| Table lock | 0.000067 | 0 |
| init | 0.000105 | 0 |
| optimizing | 0.000052 | 0 |
| statistics | 0.000254 | 0 |
| preparing | 0.000061 | 0 |
| executing | 0.000017 | 0 |
| Sending data | 32.079549 | 0 |
| end | 0.000036 | 0 |
| query end | 0.000012 | 0 |
| freeing items | 0.000089 | 0 |
| storing result in query cache | 0.000022 | 0 |
| logging slow query | 0.000008 | 0 |
| logging slow query | 0.000008 | 0 |
| cleaning up | 0.000011 | 0 |
+--------------------------------+-----------+-------+
http://dev.mysql.com/doc/refman/5.0/en/general-thread-states.html
Executing means the thread has started execution, Sending data apparently covers both the processing of the rows and sending the count back to the client.
before sending data to client, mysql need to read data, the phase of read data may take the majrity time of "sending data"