I have a simple table (created by django) - engine InnoDB:
+-------------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| correlation | double | NO | | NULL | |
| gene1_id | int(10) unsigned | NO | MUL | NULL | |
| gene2_id | int(10) unsigned | NO | MUL | NULL | |
+-------------+------------------+------+-----+---------+----------------+
The table has more than 411 million rows.
(The target table will have around 461M rows, 21471*21470 rows)
My main query looks like this, there might be up to 10 genes specified at most.
SELECT gene1_id, AVG(correlation) AS avg FROM genescorrelation
WHERE gene2_id IN (176829, 176519, 176230)
GROUP BY gene1_id ORDER BY NULL
This query is very slow, it takes almost 2 mins to run:
21471 rows in set (1 min 11.03 sec)
Indexes (cardinality looks strange - too small?):
Non_unique| Key_name | Seq_in_index | Column_name | Collation | Cardinality |
0 | PRIMARY | 1 | id | A | 411512194 |
1 | c_gene1_id_6b1d81605661118_fk_genes_gene_entrez | 1 | gene1_id | A | 18 |
1 | c_gene2_id_2d0044eaa6fd8c0f_fk_genes_gene_entrez | 1 | gene2_id | A | 18 |
I just run select count(*) on that table and it took 22 mins:
select count(*) from predictions_genescorrelation;
+-----------+
| count(*) |
+-----------+
| 411512002 |
+-----------+
1 row in set (22 min 45.05 sec)
What could be wrong?
I suspect that mysql configuration is not set up right.
During the import of data I experienced problem with space, so that might also affected the database, although I ran check table later - it took 2hours and stated OK.
Additionally - the cardinality of the indexes look strange. I have set up smaller database locally and there values are totally different (254945589,56528,17).
Should I redo indexes?
What params should I check of MySQL?
My tables are set up as InnoDB, would MyISAM make any difference?
Thanks,
matali
https://www.percona.com/blog/2006/12/01/count-for-innodb-tables/
SELECT COUNT(*) queries are very slow without WHERE clause or without SELECT COUNT(id) ... USE INDEX (PRIMARY).
to speedup this:
SELECT gene1_id, AVG(correlation) AS avg FROM genescorrelation
WHERE gene2_id IN (176829, 176519, 176230)
GROUP BY gene1_id ORDER BY NULL
you should have composite key on (gene2_id, gene1_id, correlation) in that order. try
About index-cardinality: stats of Innodb tables are approximate, not accurate (sometimes insane). there even was (IS?) a bug-report https://bugs.mysql.com/bug.php?id=58382
Try to ANALIZE table and watch cardinality again
Related
I have some table:
mysql> show columns from room_lesson;
+-------------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------------+--------------+------+-----+---------+----------------+
| id | int | NO | PRI | NULL | auto_increment |
...
<skipped>
...
| online_users_count | int | NO | | NULL | |
+-------------------------+--------------+------+-----+---------+----------------+
and simple SELECT query:
SELECT count(*) FROM room_lesson WHERE online_users_count = 2;
which takes 0.2 seconds to execute.
room_lesson table size is 1.3 Gb.
How can I speed up this query?
For this query:
SELECT count(*) FROM room_lesson WHERE online_users_count = 2;
The optimal index is on:
CREATE INDEX idx_room_lesson_online_users_count on room_lesson(online_users_count)
However, how much the index speeds up the query depends on how often the count is 2 and how wide the rows are in your table. There should be an improvement, but it might not be dramatic (say, 50% rather than 99%).
I have a simple InnoDB table with 1M+ rows and some simple indexes.
I need to sort this table by first_public and id columns and get some of them, this is why I've indexed first_public column.
first_public is unique at the moment, but in real life it might be not.
mysql> desc table;
+--------------+-------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-------------------------+------+-----+---------+----------------+
| id | bigint unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(255) | NO | | NULL | |
| id_category | int | NO | MUL | NULL | |
| active | smallint | NO | | NULL | |
| status | enum('public','hidden') | NO | | NULL | |
| first_public | datetime | YES | MUL | NULL | |
| created_at | timestamp | YES | | NULL | |
| updated_at | timestamp | YES | | NULL | |
+--------------+-------------------------+------+-----+---------+----------------+
8 rows in set (0.06 sec)
it works well while I'm working with rows before 130000+
mysql> explain select id from table where active = 1 and status = 'public' order by first_public desc, id desc limit 24 offset 130341;
+----+-------------+--------+------------+-------+---------------+---------------------+---------+------+--------+----------+----------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+---------------------+---------+------+--------+----------+----------------------------------+
| 1 | SIMPLE | table | NULL | index | NULL | firstPublicDateIndx | 6 | NULL | 130365 | 5.00 | Using where; Backward index scan |
+----+-------------+--------+------------+-------+---------------+---------------------+---------+------+--------+----------+----------------------------------+
1 row in set, 1 warning (0.00 sec)
but when I try to get some next rows (with offset 140000+), it looks like MySQL don't use first_public column index at all.
mysql> explain select id from table where active = 1 and status = 'public' order by first_public desc, id desc limit 24 offset 140341;
+----+-------------+--------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+
| 1 | SIMPLE | table | NULL | ALL | NULL | NULL | NULL | NULL | 1133533 | 5.00 | Using where; Using filesort |
+----+-------------+--------+------------+------+---------------+------+---------+------+---------+----------+-----------------------------+
1 row in set, 1 warning (0.00 sec)
I tried to add first_public column in to select clause, but nothing changed.
What I'm doing wrong?
MySQL's optimizer tries to estimate the cost of doing your query, to decide if it's worth using an index. Sometimes it compares the cost of using the index versus just reading the rows in order, and discarding the ones that don't belong in the result.
In this case, it decided that if you use an OFFSET greater than 140k, it gives up on using the index.
Keep in mind how OFFSET works. There's no way of looking up the location of an offset by an index. Indexes help to look up rows by value, not by position. So to do an OFFSET query, it has to examine all the rows from the first matching row on up. Then it discards the rows it examined up to the offset, and then counts out the enough rows to meet the LIMIT and returns those.
It's like if you wanted to read pages 500-510 in a book, but to do this, you had to read pages 1-499 first. Then when someone asks you to read pages 511-520, and you have to read pages 1-510 over again.
Eventually the offset gets to be so large that it's less expensive to read 14000 rows in a table-scan, than to read 14000 index entries + 14000 rows.
What?!? Is OFFSET really so expensive? Yes, it is. It's much more common to look up rows by value, so MySQL is optimized for that usage.
So if you can reimagine your pagination queries to look up rows by value instead of using LIMIT/OFFSET, you'll be much happier.
For example, suppose you read "page" 1000, and you see that the highest id value on that page is 13999. When the client requests the next page, you can do the query:
SELECT ... FROM mytable WHERE id > 13999 LIMIT 24;
This does the lookup by the value of id, which is optimized because it utilizes the primary key index. Then it reads just 24 rows and returns them (MySQL is at least smart enough to stop reading after it reaches the OFFSET + LIMIT rows).
The best index is
INDEX(active, status, first_public, id)
Using huge offsets is terribly inefficient -- it must scan over 140341 + 24 rows to perform the query.
If you are trying to "walk through" the table, use the technique of "remembering where you left off". More discussion of this: http://mysql.rjweb.org/doc.php/pagination
The reason for the Optimizer to abandon the index: It decided that the bouncing back and forth between the index and the table was possibly worse than simply scanning the entire table. (The cutoff is about 20%, but varies widely.)
I recently encounter a problem involving MySQL DBSM.
The Table is like this:
CREATE TABLE `orders` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(60) DEFAULT NULL,
`age` int(11) DEFAULT NULL,
`sex` enum('男','女') DEFAULT NULL,
`amount` float(10,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `name_i` (`name`),
KEY `sex` (`sex`)
) ENGINE=InnoDB AUTO_INCREMENT=5000001 DEFAULT CHARSET=utf8
As is shown above ,I create a single colume index on col name
I want to perform a range query on name, and the explain statement is
mysql> explain select * from orders where name like '王%';
+----+-------------+--------+------------+-------+---------------+--------+---------+------+-------+----------+----------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+--------+---------+------+-------+----------+----------------------------------+
| 1 | SIMPLE | orders | NULL | range | name_i | name_i | 183 | NULL | 20630 | 100.00 | Using index condition; Using MRR |
+----+-------------+--------+------------+-------+---------------+--------+---------+------+-------+----------+----------------------------------+
1 row in set, 1 warning (0.10 sec)
so it should use the index name_i and finish the query in a flash(my classmate spent 0.07 sec)
however , this is how it turned out:
| 4998119 | 王缝 | 27 | 男 | 159.21 |
| 4998232 | 王求葬 | 19 | 男 | 335.65 |
| 4998397 | 王倘予 | 49 | 女 | 103.39 |
| 4998482 | 王厚 | 77 | 男 | 960.69 |
| 4998703 | 王啄淋 | 73 | 女 | 458.85 |
| 4999106 | 王般埋 | 70 | 女 | 700.98 |
| 4999359 | 王胆具 | 31 | 女 | 362.83 |
| 4999510 | 王铁脾 | 31 | 女 | 973.09 |
| 4999880 | 王战万 | 59 | 女 | 127.28 |
| 4999928 | 王忆 | 42 | 女 | 72.47 |
+---------+--------+------+------+--------+
11160 rows in set (3.43 sec)
And it seems to not use the index at all, because the data is sorted by the primary key id rather than col name(besides it is too slow ,comparing to 0.07 sec).
Has anyone encountered the problem too?
What percentage of the table is "Kings" (王) ? If it is more than about 20%, it will choose to do a table scan instead of use the index. (And this may actually be faster.) (Based on Comments, 0.22% of the table is Kings.)
EXPLAIN and the execution of the query are separate things. Although I don't remember proving this, it is possible that the EXPLAIN might say one thing, but the query would work another way.
Do you have 5 million rows in the table? Was the cache 'cold' when you first ran it? And it had to fetch 11,160 rows from disk? Then the second time, all was in cache, so much faster?
Was the table loaded in "alphabetical" (or whatever the Chinese word for that is) order? If so, there is a good chance the ids and the names are in the same order?
Apparently you are using utf8_general_ci COLLATION? Maybe it does not sort Chinese well. (Provide a test case; I'll do some tests.)
I do not understand why it mentioned MRR.
I, too, am baffled by "1 min 32.24sec". The ORDER BY name should have further encouraged the Optimizer to use INDEX(name). Can you turn on "Optimizer trace".
To really see whether it used the index, do this:
FLUSH STATUS;
SELECT ...;
SHOW SESSION STATUS LIKE 'Handler%';
If the big number(s) look like the number of rows in the table, then it did a table scan. If they look more like 11160, then they used the index.
So here is a very simple table 'tbl':
+---------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------+---------------------+------+-----+---------+----------------+
| val | varchar(45) | YES | MUL | NULL | |
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
+---------+---------------------+------+-----+---------+----------------+
And indexes for it:
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| tbl | 0 | PRIMARY | 1 | id | A | 201826018 | NULL | NULL | | BTREE | |
| tbl | 1 | val | 1 | val | A | 881336 | NULL | NULL | YES | BTREE | |
+--------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
I'm trying this simple select:
select val from tbl where val = 'iii';
result: 86208 rows in set (0.08 sec)
But when I want to slightly modify it:
select id, val from tbl where val = 'iii';
the result is: 86208 rows in set (47.30 sec)
I have an index right on the coumn that where points to, all I'm modifying is the result rows representation.
Why there is so terrifying delay? (I have to say that I can't reproduce this delay every time I want: even after 'reset query cache' or setting 'query_cache_type=off' command it can be done quickly).
Without actually examining your server configuration it is hard to tell, but here's an educated guess. In the first instance, MySQL is able to satisfy your query without actually reading the table data. All the information you have asked for can be retrieved from the index alone. Notice that the cardinality of the val index is only on the order of 106 rows, and the rows are going to be very short in the index.
In the second case you have asked for data NOT in the index on val. Now the engine has to actually find and read rows from the data. Here the cardinality is about 250 times larger, and since the index will retrieve rows ordered by val, finding the corresponding id values will require A LOT of jumping around in several hundred gigs of data on disk. This is going to be very much slower.
Try to add a ORDER BY and `LIMIT to the query. That should help a lot.
I think if you change the query to this it will be faster:
select id, val from tbl where val = 'iii' order by val limit 10;
You're doing a select based on two columns, but no index of both exists. Try adding a new index consisting of both id and val.
There are two ways, that I can think of, to obtain similar results from multiple tables. One is UNION and the other is JOIN. The similar questions on SO have all been answered with a UNION. Here's the coder I just found:
SELECT max(up.id) AS up, max(sc.id) AS sc, max(cl.id) AS cl
FROM updates up, chat_staff sc, change_log cl
explain:
+----+-------------+-------+------+---------------+------+---------+------+------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+------+------------------------------+
| 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
+----+-------------+-------+------+---------------+------+---------+------+------+------------------------------+
My question is -- Is this better than the following?
SELECT "up.id" AS K, max(id) AS V FROM updates
UNION
SELECT "sc.id" AS K, max(id) AS V FROM chat_staff
UNION
SELECT "cl.id" AS K, max(id) AS V FROM change_log
explain:
+----+--------------+--------------+------+---------------+------+---------+------+-------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------+--------------+------+---------------+------+---------+------+------+------------------------------+
| 1 | PRIMARY | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
| 2 | UNION | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
| 3 | UNION | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
| NULL | UNION RESULT | <union1,2,3> | ALL | NULL | NULL | NULL | NULL | NULL | |
+----+--------------+--------------+------+---------------+------+---------+------+------+------------------------------+
Both of those methods are just fine. In fact, I have another method:
SELECT
IFNULL(maxidup,0) max_id_up,
IFNULL(maxscup,0) max_sc_up,
IFNULL(maxclup,0) max_cl_up
FROM
(SELECT max(id) maxidup FROM updates) up,
(SELECT max(id) maxidsc FROM chat_staff) sc,
(SELECT max(id) maxidcl FROM change_log) cl
;
This method presents the three values side by side like your first example. It also shows 0 in the event one of the tables are empty.
mysql> DROP DATABASE IF EXISTS junk;
Query OK, 3 rows affected (0.11 sec)
mysql> CREATE DATABASE junk;
Query OK, 1 row affected (0.00 sec)
mysql> use junk
Database changed
mysql> CREATE TABLE updates (id int not null auto_increment primary key,x int);
Query OK, 0 rows affected (0.07 sec)
mysql> CREATE TABLE chat_staff LIKE updates;
Query OK, 0 rows affected (0.07 sec)
mysql> CREATE TABLE change_log LIKE updates;
Query OK, 0 rows affected (0.06 sec)
mysql> INSERT INTO updates (x) VALUES (37),(84),(12);
Query OK, 3 rows affected (0.06 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> INSERT INTO change_log (x) VALUES (37),(84),(12),(14),(35);
Query OK, 5 rows affected (0.09 sec)
Records: 5 Duplicates: 0 Warnings: 0
mysql> SELECT
-> IFNULL(maxidup,0) max_id_up,
-> IFNULL(maxidsc,0) max_sc_up,
-> IFNULL(maxidcl,0) max_cl_up
-> FROM
-> (SELECT max(id) maxidup FROM updates) up,
-> (SELECT max(id) maxidsc FROM chat_staff) sc,
-> (SELECT max(id) maxidcl FROM change_log) cl
-> ;
+-----------+-----------+-----------+
| max_id_up | max_sc_up | max_cl_up |
+-----------+-----------+-----------+
| 3 | 0 | 5 |
+-----------+-----------+-----------+
1 row in set (0.00 sec)
mysql> explain SELECT IFNULL(maxidup,0) max_id_up, IFNULL(maxidsc,0) max_sc_up, IFNULL(maxidcl,0) max_cl_up FROM (SELECT max(id) maxidup FROM updates) up, (SELECT max(id) maxidsc FROM chat_staff) sc, (SELECT max(id) maxidcl FROM change_log) cl;
+----+-------------+------------+--------+---------------+------+---------+------+------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+--------+---------------+------+---------+------+------+------------------------------+
| 1 | PRIMARY | <derived2> | system | NULL | NULL | NULL | NULL | 1 | |
| 1 | PRIMARY | <derived3> | system | NULL | NULL | NULL | NULL | 1 | |
| 1 | PRIMARY | <derived4> | system | NULL | NULL | NULL | NULL | 1 | |
| 4 | DERIVED | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
| 3 | DERIVED | NULL | NULL | NULL | NULL | NULL | NULL | NULL | No matching min/max row |
| 2 | DERIVED | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
+----+-------------+------------+--------+---------------+------+---------+------+------+------------------------------+
6 rows in set (0.02 sec)
In my EXPLAIN plan, it has Select tables optimized away just like yours. Why ?
Since id is indexed in all the tables, the index is used to retrieve the max(id) rather than the table. Thus, Select tables optimized away is the correct response.
Six of one, half dozen of the other. How you present data from there is strictly your personal preference.
UPDATE 2011-10-20 15:32 EDT
You commented : Do you know how table locking would compromise this? Let's say one of the tables in question is locked. Would this query lock the other two and keep 'em locked until the first one was freed up?
This would depend on the storage engine. If all tables in question are MyISAM, definite possibility since MyISAM performs a full table lock on INSERT, UPDATE, DELETE. If the three tables are InnoDB, you have the benefit of MVCC to provide transaction isolation. This would allow everyone their view of the data in a point-in-time. Aside from DDL and an explcit LOCK TABLES against InnoDB, your query should not be blocked.
Actually, while they're similar, there's a subtle difference. The first gives you a one-row, three-column table (with the values going "across") and the second gives you a three-row, two-column table (with the values going "down").
Provided you're happy processing or viewing that data in either form, it's probably going to come down to performance.
In my experience (and this is nothing to do specifically with MySQL), the latter query will probably be better. That's because the DBMS' I work with are able to run queries like that in parallel for efficiency, combining them at completion of all. The fact that they're on different tables means that lock contention between them will be zero.
It may be that the query analysis engine of a DBMS could do a similar optimisation for the first query but it would require a lot more intelligence than I've seen from most of them.
One quick point, if you use union all instead of just union, you tell the database not to remove duplicate rows. You won't get any duplicates in this case due to the K column being different for all three sub-queries.
But, as with all optimisations, measure, don't guess! Certainly don't take as gospel the rants of random internet roamers (yes, even me).
Put together various candidate tables with the properties you're likely to have in production, and compare the performance of each.