I have a table that with about 1 billion rows that looks like this:
CREATE TABLE `ghcnddata` (
`date` date NOT NULL ,
`TMIN` float(6,2) NULL DEFAULT NULL ,
`TMAX` float(6,2) NULL DEFAULT NULL ,
`PRCP` float(6,2) NULL DEFAULT NULL ,
`SNOW` float(6,2) NULL DEFAULT NULL ,
`SNWD` float(6,2) NULL DEFAULT NULL ,
`station` varchar(30),
PRIMARY KEY (`station`, `date`),
INDEX `date` (`date`) USING BTREE ,
INDEX `station` (`station`) USING BTREE
) ENGINE=InnoDB
All of the queries I run have a line that looks like this:
WHERE `station` = "ABSUXNNSDIA3"
and a line that looks like this:
AND `date` BETWEEN "1990-01-01" AND "2010-01-01"
There are about 30,000 unique values for the station field, and no queries refer to more than 1 station. Ideally I would like to simulate having 33,333 different tables; one per station (1 billion/30,000 = 33,333).
Initially I thought I could accomplish this by setting a HASH index on station, but apparently that is only for MEMORY tables. Then I thought I PARTITION BY KEY (station) PARTITIONS 33333, but it seems that this is far too many partitions.
What should I do in this scenario? I can't really experiment because the table is so large that any modifications take a very long time.
There is no master/slave or replication or clustering or anything fancy like that.
You don't necessarily need one partition per station. The point of HASH or KEY partitioning is that you define a fixed number of partitions, and multiple values are mapped into that partition.
mysql> alter table ghcnddata partition by key(station) partitions 31;
I choose a prime number for the number of partitions just out of habit, because it helps distribute data over the partitions more evenly if the data follows a pattern (like only odd values).
mysql> insert into ghcnddata (station, date) values ('abc', now());
mysql> insert into ghcnddata (station, date) values ('def', now());
mysql> insert into ghcnddata (station, date) values ('ghi', now());
mysql> insert into ghcnddata (station, date) values ('jkl', now());
mysql> insert into ghcnddata (station, date) values ('mno', now());
mysql> insert into ghcnddata (station, date) values ('qrs', now());
mysql> insert into ghcnddata (station, date) values ('tuv', now());
mysql> insert into ghcnddata (station, date) values ('wxyz', now());
When I run a query with EXPLAIN PARTITIONS it tells me which partition(s) it must read.
mysql> explain partitions select * from ghcnddata where station='tuv';
+----+-------------+-----------+------------+------+-----------------+---------+---------+-------+------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+------------+------+-----------------+---------+---------+-------+------+-------------+
| 1 | SIMPLE | ghcnddata | p21 | ref | PRIMARY,station | PRIMARY | 122 | const | 1 | Using where |
+----+-------------+-----------+------------+------+-----------------+---------+---------+-------+------+-------------+
We can see in this case that only partition 21 was read when I reference station 'tuv'.
Note that partitioning is not a panacea. It only helps to reduce the work of the query if you search for a constant value (not a variable, or a join condition, etc.) in the same column that you defined as the partitioning key.
The rows I just inserted should be roughly evenly distributed, but not perfectly evenly distributed. And there's no guarantee it's one station value per partition.
mysql> select table_name, partition_name, table_rows
from information_schema.partitions where table_name='ghcnddata';
+------------+----------------+------------+
| table_name | partition_name | table_rows |
+------------+----------------+------------+
| ghcnddata | p0 | 1 |
| ghcnddata | p1 | 2 |
| ghcnddata | p2 | 0 |
| ghcnddata | p3 | 0 |
| ghcnddata | p4 | 0 |
| ghcnddata | p5 | 0 |
| ghcnddata | p6 | 0 |
| ghcnddata | p7 | 0 |
| ghcnddata | p8 | 0 |
| ghcnddata | p9 | 0 |
| ghcnddata | p10 | 0 |
| ghcnddata | p11 | 0 |
| ghcnddata | p12 | 0 |
| ghcnddata | p13 | 0 |
| ghcnddata | p14 | 0 |
| ghcnddata | p15 | 0 |
| ghcnddata | p16 | 0 |
| ghcnddata | p17 | 0 |
| ghcnddata | p18 | 0 |
| ghcnddata | p19 | 0 |
| ghcnddata | p20 | 0 |
| ghcnddata | p21 | 2 |
| ghcnddata | p22 | 1 |
| ghcnddata | p23 | 1 |
| ghcnddata | p24 | 1 |
| ghcnddata | p25 | 0 |
| ghcnddata | p26 | 0 |
| ghcnddata | p27 | 0 |
| ghcnddata | p28 | 0 |
| ghcnddata | p29 | 0 |
| ghcnddata | p30 | 0 |
+------------+----------------+------------+
P.S.: Your table's index on station is redundant, because that's the leftmost column of your primary key already.
Related
I have a database which I make a lot of queries to like this:
mysql> explain SELECT time, user, x, y, z, type, action
FROM co_block
WHERE time >= 1642497664
AND wid = 4
AND x <= -7650
AND x >= -7651;
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
| 1 | SIMPLE | co_block | NULL | range | wid | wid | 10 | NULL | 3940334 | 33.33 | Using index condition |
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
This query returns 101 rows in 0.7 seconds.
Almost all of them return very quickly, but occasionally I run into one that takes an eternity, like this almost-identical query which differs only in the X range that is searched.
mysql> explain SELECT time, user, x, y, z, type, action FROM co_block WHERE time >= 1642497664 AND wid = 4 AND x <= -7651 AND x >= -7652;
+----+-------------+----------+------------+------+---------------+------+---------+------+-----------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+------+---------------+------+---------+------+-----------+----------+-------------+
| 1 | SIMPLE | co_block | NULL | ALL | wid | NULL | NULL | NULL | 116322801 | 3.59 | Using where |
+----+-------------+----------+------------+------+---------------+------+---------+------+-----------+----------+-------------+
This query returns 85 rows after a painful 68 seconds.
If I use FORCE INDEX to force the use of the index, the query returns in only 1.2 seconds, so it obviously can be used and to great benefit, and indeed MySQL seems to actually use it in 99.99% of the queries of this sort that I make. So why is it choosing not to use the index in this case?
The "explain" of the FORCE INDEX queries, in case they're insightful:
mysql> explain SELECT time, user, x, y, z, type, action FROM co_block FORCE INDEX (`wid`) WHERE time >= 1642497664 AND wid = 4 AND x <= -7650 AND x >= -7651;
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
| 1 | SIMPLE | co_block | NULL | range | wid | wid | 10 | NULL | 3940334 | 33.33 | Using index condition |
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
1 row in set, 1 warning (0.00 sec)
mysql> explain SELECT time, user, x, y, z, type, action FROM co_block FORCE INDEX (`wid`) WHERE time >= 1642497664 AND wid = 4 AND x <= -7651 AND x >= -7652;
+----+-------------+----------+------------+-------+---------------+------+---------+------+----------+----------+-----------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+---------------+------+---------+------+----------+----------+-----------------------+
| 1 | SIMPLE | co_block | NULL | range | wid | wid | 10 | NULL | 12543806 | 33.33 | Using index condition |
+----+-------------+----------+------------+-------+---------------+------+---------+------+----------+----------+-----------------------+
1 row in set, 1 warning (0.00 sec)
Table definition:
CREATE TABLE `co_block` (
`rowid` bigint NOT NULL AUTO_INCREMENT,
`time` int DEFAULT NULL,
`user` int DEFAULT NULL,
`wid` int DEFAULT NULL,
`x` int DEFAULT NULL,
`y` int DEFAULT NULL,
`z` int DEFAULT NULL,
`type` int DEFAULT NULL,
`data` int DEFAULT NULL,
`meta` blob,
`blockdata` blob,
`action` int DEFAULT NULL,
`rolled_back` tinyint(1) DEFAULT NULL,
PRIMARY KEY (`rowid`),
KEY `wid` (`wid`,`x`,`z`,`time`),
KEY `user` (`user`,`time`),
KEY `type` (`type`,`time`)
) ENGINE=InnoDB AUTO_INCREMENT=252511481 DEFAULT CHARSET=utf8mb3
Index info:
mysql> show index from co_block;
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression |
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| co_block | 0 | PRIMARY | 1 | rowid | A | 116288328 | NULL | NULL | | BTREE | | | YES | NULL |
| co_block | 1 | wid | 1 | wid | A | 886 | NULL | NULL | YES | BTREE | | | YES | NULL |
| co_block | 1 | wid | 2 | x | A | 135141 | NULL | NULL | YES | BTREE | | | YES | NULL |
| co_block | 1 | wid | 3 | z | A | 22393332 | NULL | NULL | YES | BTREE | | | YES | NULL |
| co_block | 1 | wid | 4 | time | A | 71268960 | NULL | NULL | YES | BTREE | | | YES | NULL |
| co_block | 1 | user | 1 | user | A | 49756 | NULL | NULL | YES | BTREE | | | YES | NULL |
| co_block | 1 | user | 2 | time | A | 13558248 | NULL | NULL | YES | BTREE | | | YES | NULL |
| co_block | 1 | type | 1 | type | A | 33126 | NULL | NULL | YES | BTREE | | | YES | NULL |
| co_block | 1 | type | 2 | time | A | 28455380 | NULL | NULL | YES | BTREE | | | YES | NULL |
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
Likely irrelevant notes:
The actual queries I'm making are grabbing 16x16 chunks by specifying both an X range and a Z range, but in trying to debug the problem myself, the Z range specified doesn't seem to be relevant to the problem, so I removed it from the queries.
In one of the many "why isn't MySQL using my index" questions I read, the problem was that the condition specified by the WHERE clause included almost all of the records. This made me wonder if the problem might be the order of the "x >= -7652" and "x <= -7651" conditions, as the first would include almost all of the records in the database, and only the second actually eliminates most of them. So I reversed the order of these two conditions, but it made no difference. I also tried using BETWEEN to specify the range but that too had no effect.
Though perhaps only due to insufficient effort to find an example case, it seems to be required that there is both an "x >=" and an "x <=" condition.
The time constrain is not required but I included it above because otherwise there is much more data returned by the queries. Here are the "explain" queries with the time constraint removed:
mysql> explain SELECT time, user, x, y, z, type, action FROM co_block WHERE wid = 4 AND x <= -7651 AND x >= -7652;
+----+-------------+----------+------------+------+---------------+------+---------+------+-----------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+------+---------------+------+---------+------+-----------+----------+-------------+
| 1 | SIMPLE | co_block | NULL | ALL | wid | NULL | NULL | NULL | 116325588 | 10.78 | Using where |
+----+-------------+----------+------------+------+---------------+------+---------+------+-----------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
mysql> explain SELECT time, user, x, y, z, type, action FROM co_block WHERE wid = 4 AND x <= -7650 AND x >= -7651;
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
| 1 | SIMPLE | co_block | NULL | range | wid | wid | 10 | NULL | 3940334 | 100.00 | Using index condition |
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+-----------------------+
1 row in set, 1 warning (0.00 sec)
Query results requested by nnichols:
mysql> SELECT COUNT(*) FROM co_block FORCE INDEX (wid) WHERE wid = 4 AND x <= -7650 AND x >= -7651;
+----------+
| COUNT(*) |
+----------+
| 2958623 |
+----------+
1 row in set (0.86 sec)
mysql> explain SELECT COUNT(*) FROM co_block FORCE INDEX (wid) WHERE wid = 4 AND x <= -7650 AND x >= -7651;
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+
| 1 | SIMPLE | co_block | NULL | range | wid | wid | 10 | NULL | 3940334 | 100.00 | Using where; Using index |
+----+-------------+----------+------------+-------+---------------+------+---------+------+---------+----------+--------------------------+
1 row in set, 1 warning (0.00 sec)
mysql> SELECT COUNT(*) FROM co_block FORCE INDEX (wid) WHERE wid = 4 AND x <= -7651 AND x >= -7652;
+----------+
| COUNT(*) |
+----------+
| 8982424 |
+----------+
1 row in set (2.38 sec)
mysql> explain SELECT COUNT(*) FROM co_block FORCE INDEX (wid) WHERE wid = 4 AND x <= -7651 AND x >= -7652;
+----+-------------+----------+------------+-------+---------------+------+---------+------+----------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+---------------+------+---------+------+----------+----------+--------------------------+
| 1 | SIMPLE | co_block | NULL | range | wid | wid | 10 | NULL | 12543806 | 100.00 | Using where; Using index |
+----+-------------+----------+------------+-------+---------------+------+---------+------+----------+----------+--------------------------+
1 row in set, 1 warning (0.00 sec)
Add these two indexes and see if things improve:
INDEX(wid, x, time),
INDEX(wid, time, x)
Also, run this once:
ANALYZE TABLE co_block;
How big is the table compared to innodb_buffer_pool_size? If quite big, then think about whether some of the INTs (4 bytes each) can be shrunk to smaller datatypes.
This might help, too:
ORDER BY x, time
More
innodb_buffer_pool_size should be about 70% of available RAM, not the old default of 128M. That serves as a cache for the data and/or indexes. So, when running several related queries, the first may be slow, and the rest may be faster. Run a query twice to make sure the timing is not biased by the caching.
As already discussed in the comments and mentioned by #RickJames, you really should update the datatypes to better fit the data. I would suggest starting with -
ALTER TABLE `co_block`
CHANGE COLUMN `time` `time` INT UNSIGNED NOT NULL, /* 5 => 4 bytes per index entry */
CHANGE COLUMN `wid` `wid` TINYINT NOT NULL, /* 5 => 1 byte */
CHANGE COLUMN `x` `x` SMALLINT NOT NULL, /* 5 => 2 bytes */
CHANGE COLUMN `z` `z` SMALLINT NOT NULL; /* 5 => 2 bytes */
This will reduce your wid index from 20 bytes => 9 bytes per row; less than half the size. Assuming similar changes can be made to all the integer fields, your space saving on both the data and index will be considerable. That reduction in size will also lead to improved performance.
If you are concerned about possible impact on the plugin then do some digging online or speak to the developer. Do you need to retain all the data? Should you implement a purge routine to remove data older than x months?
NULLness in InnoDB
I built a very small table and populated it with 50M rows -
CREATE TABLE `test_null_index` (
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`tiny` TINYINT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB;
Running in ultra-paranoid mode, I ran an OPTIMIZE and ANALYZE before grabbing the stats after each table change - no index, with index allowing nulls and with index not null. I then re-ran with just 5M rows. Here are the results -
TABLE_NAME
TABLE_ROWS
AVG_ROW_LENGTH
DATA_LENGTH
INDEX_LENGTH
DATA_FREE
test_null_index
49926120
30
1509933056
0
2097152
test_null_index
49926120
30
1509933056
664780800
5242880
test_null_index
49929637
28
1447018496
606044160
3145728
test_null_index
4992899
29
145358848
0
5242880
test_null_index
4992899
29
145358848
61407232
5242880
test_null_index
4992975
27
139067392
55115776
2097152
TABLE_ROWS
INDEX_LEN_NULL
INDEX_LEN_NOT_NULL
INDEX_LEN_DIFF
AVG_DIFF_PER_ROW
50000000
664780800
606044160
58736640
1.1747328
5000000
61407232
55115776
6291456
1.2582912
Looks like NULLness taking an extra byte in InnoDB is a thing.
From https://stackoverflow.com/a/51181742/3284469
If the table has no PRIMARY KEY or suitable UNIQUE index, InnoDB
internally generates a hidden clustered index named GEN_CLUST_INDEX on
a synthetic column containing row ID values. The rows are ordered by
the ID that InnoDB assigns to the rows in such a table. The row ID is
a 6-byte field that increases monotonically as new rows are inserted.
Thus, the rows ordered by the row ID are physically in insertion
order.
My mysql version is:
$ mysql --version
mysql Ver 8.0.11 for Linux on x86_64 (MySQL Community Server - GPL)
I followed the commands there to verify the internal index is created, but the last command doesn't show any index has been created. Why is that? Thanks.
Note that I changed the last command a little bit, because the original command gives me Unknown table 'INNODB_INDEX_STATS' in information_schema error.
# Create the table
create table test.check_table (id int, description varchar(10)) ENGINE = INNODB;
# Verify that there is no primary or unique column
desc test.check_table;
+-------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| id | int(11) | YES | | NULL | |
| description | varchar(10) | YES | | NULL | |
+-------------+-------------+------+-----+---------+-------+
# Insert some values
insert into test.check_table values(1, 'value-1');
insert into test.check_table values(2, 'value-2');
insert into test.check_table values(null, 'value-3');
insert into test.check_table values(4, null);
insert into test.check_table values(1, 'value-1');
# Verify table
select * from test.check_table;
+------+-------------+
| id | description |
+------+-------------+
| 1 | value-1 |
| 2 | value-2 |
| NULL | value-3 |
| 4 | NULL |
| 1 | value-1 |
+------+-------------+
# Verify that the GEN_CLUST_INDEX index is auto-created.
select * from INFORMATION_SCHEMA.INNODB_INDEX_STATS where TABLE_SCHEMA='test' and TABLE_NAME = 'check_table';
ERROR 1109 (42S02): Unknown table 'INNODB_INDEX_STATS' in information_schema
SELECT DISTINCT TABLE_NAME, COLUMN_NAME , INDEX_NAME FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = 'test' AND TABLE_NAME='check_table';
Empty set (0.00 sec)
In all versions of MySQL that I've been able to find, the INNODB_INDEX_STATS table is located in the mysql database, not INFORMATION_SCHEMA. This appears to be an error in the post you're referencing.
mysql> select * from mysql.innodb_index_stats where table_name = 'check_table';
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
| database_name | table_name | index_name | last_update | stat_name | stat_value | sample_size | stat_description |
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 11:34:01 | n_diff_pfx01 | 5 | 1 | DB_ROW_ID |
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 11:34:01 | n_leaf_pages | 1 | NULL | Number of leaf pages in the index |
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 11:34:01 | size | 1 | NULL | Number of pages in the index |
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
This index isn't a "real" index from the perspective of SQL (it doesn't appear in the output of DESCRIBE, and can't be modified or dropped), so it isn't shown in INFORMATION_SCHEMA.STATISTICS.
For version 8.0.11, the table innodb_index_stats is located in mysql schema in lieu of INFORMATION_SCHEMA. Following the commands, the last query gives result as below:
mysql> select VERSION();
+-----------+
| VERSION() |
+-----------+
| 8.0.11 |
+-----------+
mysql> select * from mysql.innodb_index_stats where database_name='test' and table_name = 'check_table';
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
| database_name | table_name | index_name | last_update | stat_name | stat_value | sample_size | stat_description |
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 18:57:45 | n_diff_pfx01 | 5 | 1 | DB_ROW_ID |
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 18:57:45 | n_leaf_pages | 1 | NULL | Number of leaf pages in the index |
| test | check_table | GEN_CLUST_INDEX | 2018-07-10 18:57:45 | size | 1 | NULL | Number of pages in the index |
+---------------+-------------+-----------------+---------------------+--------------+------------+-------------+-----------------------------------+
Also, the post referenced in the question creates a second table with a primary key specified. The index verification for that query gives:
mysql> create table test.check_table_2 (id int, description varchar(10), PRIMARY KEY(id)) ENGINE = INNODB;
mysql> desc check_table_2;
+-------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| description | varchar(10) | YES | | NULL | |
+-------------+-------------+------+-----+---------+-------+
mysql> select * from mysql.innodb_index_stats where database_name='test' and table_name = 'check_table_2';
+---------------+---------------+------------+---------------------+--------------+------------+-------------+-----------------------------------+
| database_name | table_name | index_name | last_update | stat_name | stat_value | sample_size | stat_description |
+---------------+---------------+------------+---------------------+--------------+------------+-------------+-----------------------------------+
| test | check_table_2 | PRIMARY | 2018-07-10 19:00:39 | n_diff_pfx01 | 0 | 1 | id |
| test | check_table_2 | PRIMARY | 2018-07-10 19:00:39 | n_leaf_pages | 1 | NULL | Number of leaf pages in the index |
| test | check_table_2 | PRIMARY | 2018-07-10 19:00:39 | size | 1 | NULL | Number of pages in the index |
+---------------+---------------+------------+---------------------+--------------+------------+-------------+-----------------------------------+
My Magento(Vesion 1.8.0.0) website was running without any problem when I was using MySQL5.5 database for years. After moving to mariadb-server-10.2 recently(exported data using sqldump and then imported on new mariadb server), many times PHP child process stays in waiting states until max_execution_time expires and finally php-fpm stops responding.
On investigation I found that all the php child processes are waiting on database queries. I checked the processlist on mariadb when I encountered the problem recently and output is below. Any idea how I could fix this? Thanks in advance.
MariaDB [(none)]> show processlist;
+--------+-------------+-------------------------------+---------+---------+------+------------------------------+---------------------------------------------------------------------- --------------------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-------------+-------------------------------+---------+---------+------+------------------------------+---------------------------------------------------------------------- --------------------------------+----------+
| 1 | system user | | NULL | Daemon | NULL | | NULL | 0.000 |
| 2 | system user | | NULL | Daemon | NULL | | NULL | 0.000 |
| 4 | system user | | NULL | Daemon | NULL | | NULL | 0.000 |
| 3 | system user | | NULL | Daemon | NULL | | NULL | 0.000 |
| 5 | system user | | NULL | Daemon | NULL | InnoDB shutdown handler | NULL | 0.000 |
| 109822 | catalog | mymagentositexxxxxx.com:56405 | catalog | Query | 985 | update | INSERT INTO `cataloginventory_stock_status` (`product_id`,`website_id `,`stock_id`,`qty`,`stock_statu | 0.000 |
| 109850 | catalog | mymagentositexxxxxx.com:56436 | catalog | Query | 985 | update | INSERT INTO `cataloginventory_stock_status` (`product_id`,`website_id `,`stock_id`,`qty`,`stock_statu | 0.000 |
| 109859 | catalog | mymagentositexxxxxx.com:56448 | catalog | Query | 985 | update | INSERT INTO `cataloginventory_stock_status` (`product_id`,`website_id `,`stock_id`,`qty`,`stock_statu | 0.000 |
| 109931 | catalog | mymagentositexxxxxx.com:56554 | catalog | Query | 986 | Waiting for table level lock | DELETE FROM `catalogsearch_fulltext` WHERE (store_id=1) AND (product_ id IN ('26515')) | 0.000 |
| 109966 | catalog | mymagentositexxxxxx.com:56606 | catalog | Query | 984 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33218 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 109993 | catalog | mymagentositexxxxxx.com:56645 | catalog | Query | 934 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 109997 | catalog | mymagentositexxxxxx.com:56650 | catalog | Query | 987 | Sending data | INSERT INTO `catalogsearch_result` SELECT 33215 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110040 | catalog | mymagentositexxxxxx.com:56709 | catalog | Query | 983 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33215 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110218 | catalog | mymagentositexxxxxx.com:56925 | catalog | Query | 940 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 6656 AS `query_id`, `s`.`pr oduct_id`, 0 AS `relevance` FRO | 0.000 |
| 110248 | catalog | mymagentositexxxxxx.com:56956 | catalog | Query | 932 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33220 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110380 | catalog | mymagentositexxxxxx.com:57095 | catalog | Query | 906 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33214 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110392 | catalog | mymagentositexxxxxx.com:57107 | catalog | Query | 905 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 33215 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110810 | catalog | mymagentositexxxxxx.com:57554 | catalog | Query | 825 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 6656 AS `query_id`, `s`.`pr oduct_id`, 0 AS `relevance` FRO | 0.000 |
| 110865 | catalog | mymagentositexxxxxx.com:57616 | catalog | Query | 808 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 21581 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 110968 | catalog | mymagentositexxxxxx.com:57782 | catalog | Query | 48 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 110969 | catalog | mymagentositexxxxxx.com:57783 | catalog | Query | 48 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 110989 | catalog | mymagentositexxxxxx.com:57803 | catalog | Query | 39 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 110999 | catalog | mymagentositexxxxxx.com:57813 | catalog | Query | 34 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 111012 | catalog | mymagentositexxxxxx.com:57867 | catalog | Query | 32 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 111067 | catalog | mymagentositexxxxxx.com:57959 | catalog | Query | 18 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 111077 | catalog | mymagentositexxxxxx.com:57974 | catalog | Query | 15 | update | INSERT INTO `sales_flat_quote_item` (`quote_id`, `created_at`, `updat ed_at`, `product_id`, `store_id | 0.000 |
| 111078 | root | localhost | NULL | Query | 0 | init | show processlist | 0.000 |
| 111117 | catalog | mymagentositexxxxxx.com:58015 | catalog | Query | 10 | Waiting for table level lock | INSERT INTO `catalogsearch_result` SELECT 13226 AS `query_id`, `s`.`p roduct_id`, 0 AS `relevance` FR | 0.000 |
| 111147 | moodle | mymagentositexxxxxx.com:58046 | moodle | Sleep | 0 | | NULL | 0.000 |
+--------+-------------+-------------------------------+---------+---------+------+------------------------------+---------------------------------------------------------------------- --------------------------------+----------+
29 rows in set (0.00 sec)
#update2
'Show create table output' of table's in waiting state is as below
MariaDB [catalog]> SHOW CREATE TABLE catalogsearch_fulltext;
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| catalogsearch_fulltext | CREATE TABLE `catalogsearch_fulltext` (
`product_id` int(10) unsigned NOT NULL COMMENT 'Product ID',
`store_id` smallint(5) unsigned NOT NULL COMMENT 'Store ID',
`data_index` longtext DEFAULT NULL COMMENT 'Data index',
`fulltext_id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Entity ID',
PRIMARY KEY (`fulltext_id`),
UNIQUE KEY `UNQ_CATALOGSEARCH_FULLTEXT_PRODUCT_ID_STORE_ID` (`product_id`,`store_id`),
FULLTEXT KEY `FTI_CATALOGSEARCH_FULLTEXT_DATA_INDEX` (`data_index`)
) ENGINE=MyISAM AUTO_INCREMENT=912741 DEFAULT CHARSET=utf8 COMMENT='Catalog search result table' |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
MariaDB [catalog]> SHOW CREATE TABLE catalogsearch_result;
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| catalogsearch_result | CREATE TABLE `catalogsearch_result` (
`query_id` int(10) unsigned NOT NULL COMMENT 'Query ID',
`product_id` int(10) unsigned NOT NULL COMMENT 'Product ID',
`relevance` decimal(20,4) NOT NULL DEFAULT 0.0000 COMMENT 'Relevance',
PRIMARY KEY (`query_id`,`product_id`),
KEY `IDX_CATALOGSEARCH_RESULT_QUERY_ID` (`query_id`),
KEY `IDX_CATALOGSEARCH_RESULT_PRODUCT_ID` (`product_id`),
CONSTRAINT `FK_CATALOGSEARCH_RESULT_QUERY_ID_CATALOGSEARCH_QUERY_QUERY_ID` FOREIGN KEY (`query_id`) REFERENCES `catalogsearch_query` (`query_id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `FK_CATSRCH_RESULT_PRD_ID_CAT_PRD_ENTT_ENTT_ID` FOREIGN KEY (`product_id`) REFERENCES `catalog_product_entity` (`entity_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Catalog search result table' |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
The process with the most time (#109997) is touching that MyISAM table? After doing SHOW FULL PROCESSLIST, I see
INSERT
INTO `catalogsearch_result`
SELECT 11474 AS `query_id`, `s`.`product_id`, 0 AS `relevance`
FROM `catalogsearch_fulltext` AS `s`
INNER JOIN `catalog_product_entity` AS `e` ON e.entity_id = s.product_id
WHERE (s.store_id = 1)
AND ((`s`.`data_index` LIKE '%Informaatika%'
OR `s`.`data_index` LIKE '%8.c%')
) ON DUPLICATE KEY
UPDATE `relevance` = VALUES(`relevance`)
That is a slow SELECT that only recently got started. Since we are talking about MyISAM, and there writes going on, the queries will happen one at a time (mostly).
It is slow because of both the OR and the leading wildcard in the LIKE test. Do you have any way to avoid both of those issues? With them there, the SELECT must scan the entire table, which apparently takes minutes. (How big is the table?)
Changing timeouts may get you past one instance, but then simply keep you in a never-ending sequence of queries waiting for other queries to finish.
The best solution is to get away from MyISAM. More tips on that: http://mysql.rjweb.org/doc.php/myisam2innodb
(Even after converting, there could be other issues.)
I have created a mysql table and hash partitioned it as below.
mysql> CREATE TABLE employees (
id INT NOT NULL,
fname VARCHAR(30),
lname VARCHAR(30),
hired DATE NOT NULL DEFAULT '1970-01-01',
separated DATE NOT NULL DEFAULT '9999-12-31',
job_code INT,
store_id INT,
PRIMARY KEY(id)
)
PARTITION BY HASH(id)
PARTITIONS 10;
After I created table successfully, I inserted value 1(into store_id) into the table shown below
mysql>INSERT INTO employees (store_id) values (1);
Now I don't understand where will this value of 1 go into? Into which partition (p0,p1,p2......p10) store_id value 1 go? I thought it would go into p0. but it did not. see below I checked it like this
mysql>SELECT TABLE_NAME, PARTITION_NAME, TABLE_ROWS, AVG_ROW_LENGTH,DATA_LENGTH FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME LIKE 'employees';
it has shown the value went into p1.see below
mysql>
+------------+----------------+------------+----------------+-------------+
| TABLE_NAME | PARTITION_NAME | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH |
+------------+----------------+------------+----------------+-------------+
| employees | p0 | 0 | 0 | 16384 |
| employees | p1 | 1 | 16384 | 16384 |
| employees | p2 | 0 | 0 | 16384 |
| employees | p3 | 0 | 0 | 16384 |
| employees | p4 | 0 | 0 | 16384 |
| employees | p5 | 0 | 0 | 16384 |
| employees | p6 | 0 | 0 | 16384 |
| employees | p7 | 0 | 0 | 16384 |
| employees | p8 | 0 | 0 | 16384 |
| employees | p9 | 0 | 0 | 16384 |
+------------+----------------+------------+----------------+-------------+
I don'tknow why it got inserted into p1.tested it again.. I inserted value 2 this time...
mysql> INSERT INTO employees (store_id) values (2);
It has got entered into p2.
+------------+----------------+------------+----------------+-------------+
| TABLE_NAME | PARTITION_NAME | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH |
+------------+----------------+------------+----------------+-------------+
| employees | p0 | 0 | 0 | 16384 |
| employees | p1 | 1 | 16384 | 16384 |
| employees | p2 | 1 | 16384 | 16384 |
| employees | p3 | 0 | 0 | 16384 |
| employees | p4 | 0 | 0 | 16384 |
| employees | p5 | 0 | 0 | 16384 |
| employees | p6 | 0 | 0 | 16384 |
| employees | p7 | 0 | 0 | 16384 |
| employees | p8 | 0 | 0 | 16384 |
| employees | p9 | 0 | 0 | 16384 |
+------------+----------------+------------+----------------+-------------+
why values are getting inserted into different partitions? Is there any rule that hash partition follow? Interestingly it left p0 and started getting inserted into p1? Explain?
If this explanation holds true for your MySQL version the partition number is found this way: MOD([Your input],[Number of partitions]).
In your case the first row probably has id = 1 and the calculation will be MOD(1,10) = 1. The row goes to partition 1 (id= 2 goes to partition 2).
It seems that mysql is only selecting data from the first partition and last partition when you use a date range.
| sales | CREATE TABLE `sales` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`quantity_sold` int(11) NOT NULL,
`prod_id` int(11) NOT NULL,
`store_id` int(11) NOT NULL,
`date` date NOT NULL,
KEY `prod_id` (`prod_id`),
KEY `date` (`date`),
KEY `store_id` (`store_id`),
KEY `id` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=577574322 DEFAULT CHARSET=utf8
/*!50100 PARTITION BY RANGE (to_days(date))
(PARTITION p0 VALUES LESS THAN (0) ENGINE = InnoDB,
PARTITION p201211 VALUES LESS THAN (735203) ENGINE = InnoDB,
PARTITION p201212 VALUES LESS THAN (735234) ENGINE = InnoDB,
PARTITION p201301 VALUES LESS THAN (735265) ENGINE = InnoDB,
PARTITION p201302 VALUES LESS THAN (735293) ENGINE = InnoDB,
PARTITION p201303 VALUES LESS THAN (735324) ENGINE = InnoDB,
PARTITION p201304 VALUES LESS THAN (735354) ENGINE = InnoDB,
PARTITION p201305 VALUES LESS THAN (735385) ENGINE = InnoDB,
PARTITION p201306 VALUES LESS THAN (735415) ENGINE = InnoDB,
PARTITION p201307 VALUES LESS THAN (735446) ENGINE = InnoDB,
PARTITION p201308 VALUES LESS THAN (735477) ENGINE = InnoDB,
PARTITION p201309 VALUES LESS THAN (735507) ENGINE = InnoDB,
PARTITION p201310 VALUES LESS THAN (735538) ENGINE = InnoDB,
PARTITION p201311 VALUES LESS THAN (735568) ENGINE = InnoDB,
PARTITION p201312 VALUES LESS THAN (735599) ENGINE = InnoDB,
PARTITION p201401 VALUES LESS THAN (735630) ENGINE = InnoDB,
PARTITION p201402 VALUES LESS THAN (735658) ENGINE = InnoDB,
PARTITION p201403 VALUES LESS THAN (735689) ENGINE = InnoDB,
PARTITION p201404 VALUES LESS THAN (735719) ENGINE = InnoDB,
PARTITION p201405 VALUES LESS THAN (735750) ENGINE = InnoDB,
PARTITION p201406 VALUES LESS THAN (735780) ENGINE = InnoDB,
PARTITION p201407 VALUES LESS THAN (735811) ENGINE = InnoDB,
PARTITION p201408 VALUES LESS THAN (735842) ENGINE = InnoDB,
PARTITION p201409 VALUES LESS THAN (735872) ENGINE = InnoDB,
PARTITION p201410 VALUES LESS THAN (735903) ENGINE = InnoDB,
PARTITION p201411 VALUES LESS THAN (735933) ENGINE = InnoDB,
PARTITION p201412 VALUES LESS THAN (735964) ENGINE = InnoDB,
PARTITION P201501 VALUES LESS THAN (735995) ENGINE = InnoDB,
PARTITION P201502 VALUES LESS THAN (736023) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */ |
Select the sales (should get data from all of the partitions) But only from the first and last?
`mysql> select * from sales where prod_id = 232744 and store_id = 300;
+-----------+---------------+---------+----------+------------+
| id | quantity_sold | prod_id | store_id | date |
+-----------+---------------+---------+----------+------------+
| 2309 | 1 | 232744 | 300 | 2012-11-26 |
| 2484 | 10 | 232744 | 300 | 2012-11-27 |
| 2837 | 7 | 232744 | 300 | 2012-11-29 |
| 3001 | 9 | 232744 | 300 | 2012-11-30 |
| 571930074 | 4 | 232744 | 300 | 2014-12-02 |
| 573051350 | 13 | 232744 | 300 | 2014-12-03 |
| 574181358 | 5 | 232744 | 300 | 2014-12-04 |
| 575322316 | 9 | 232744 | 300 | 2014-12-05 |
| 576455102 | 4 | 232744 | 300 | 2014-12-06 |
| 577545446 | 2 | 232744 | 300 | 2014-12-07 |
+-----------+---------------+---------+----------+------------+`
The explain partition show that it is scanning all of the partitions.
mysql> explain partitions select * from sales where prod_id = 232744 and store_id =300\G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: sales
partitions: p0,p201211,p201212,p201301,p201302,p201303,p201304,p201305,p201306,p201307,p201308,p201309,p201310,p201311,p201312,p201401,p201402,p201403,p201404,p201405,p201406,p201407,p201408,p201409,p201410,p201411,p201412,P201501,P201502,p1
type: index_merge
possible_keys: prod_id,store_id
key: prod_id,store_id
key_len: 4,4
ref: NULL
rows: 20
Extra: Using intersect(prod_id,store_id); Using where
1 row in set (0.00 sec)
If I manually select the partition we can see there is data there, which should be in the data above.
`mysql> select * from sales PARTITION (p201410) where prod_id = 232744 and store_id = 300;
+-----------+---------------+---------+----------+------------+
| id | quantity_sold | prod_id | store_id | date |
+-----------+---------------+---------+----------+------------+
| 509534154 | 2 | 232744 | 300 | 2014-10-01 |
| 510606312 | 10 | 232744 | 300 | 2014-10-02 |
| 511682398 | 4 | 232744 | 300 | 2014-10-03 |
| 512752933 | 2 | 232744 | 300 | 2014-10-04 |
| 514812731 | 3 | 232744 | 300 | 2014-10-06 |
| 515862308 | 6 | 232744 | 300 | 2014-10-07 |
| 516922728 | 5 | 232744 | 300 | 2014-10-08 |
| 517990349 | 19 | 232744 | 300 | 2014-10-09 |
| 519066761 | 17 | 232744 | 300 | 2014-10-10 |
| 520136175 | 3 | 232744 | 300 | 2014-10-11 |
| 522185901 | 1 | 232744 | 300 | 2014-10-14 |
| 523238559 | 3 | 232744 | 300 | 2014-10-15 |
| 524294166 | 7 | 232744 | 300 | 2014-10-16 |
| 525354982 | 3 | 232744 | 300 | 2014-10-17 |
| 526412605 | 1 | 232744 | 300 | 2014-10-18 |
| 527444329 | 1 | 232744 | 300 | 2014-10-19 |
| 528452608 | 1 | 232744 | 300 | 2014-10-20 |
| 529488414 | 2 | 232744 | 300 | 2014-10-21 |
| 530541002 | 3 | 232744 | 300 | 2014-10-22 |
| 531603714 | 4 | 232744 | 300 | 2014-10-23 |
| 532672667 | 6 | 232744 | 300 | 2014-10-24 |
| 534793524 | 1 | 232744 | 300 | 2014-10-26 |
| 535819138 | 1 | 232744 | 300 | 2014-10-27 |
| 537957232 | 1 | 232744 | 300 | 2014-10-29 |
| 539037254 | 1 | 232744 | 300 | 2014-10-30 |
| 540125545 | 2 | 232744 | 300 | 2014-10-31 |
+-----------+---------------+---------+----------+------------+
26 rows in set (0.03 sec)`
If you were to do a select * from sales where prod_id = 232744; it will select all of the data.It seems to be just when you add a store_id condition in there that it doesn't select the correct data.
I'm stumped. I've tried:
Restarting mysql
I'm about to try a OPTIMIZE TABLE (I have to move the databases because of space constraints)
Seems to me there is something wrong with the keys? corrupt table?
Thanks!