I have a table that I'm running a very simple query against. I've added an index to the table on a high cardinality column, so MySQL should be able to narrow the result almost instantly, but it's doing a full table scan every time. Why isn't MySQL using my index?
mysql> select count(*) FROM eventHistory;
+----------+
| count(*) |
+----------+
| 247514 |
+----------+
1 row in set (0.15 sec)
CREATE TABLE `eventHistory` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`whatID` varchar(255) DEFAULT NULL,
`whatType` varchar(255) DEFAULT NULL,
`whoID` varchar(255) DEFAULT NULL,
`createTimestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `whoID` (`whoID`,`whatID`)
) ENGINE=InnoDB;
mysql> explain SELECT * FROM eventHistory where whoID = 12551\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: eventHistory
type: ALL
possible_keys: whoID
key: NULL
key_len: NULL
ref: NULL
rows: 254481
Extra: Using where
1 row in set (0.00 sec)
I have tried adding FORCE INDEX to the query as well, and it still seems to be doing a full table scan. The performance of the query is also poor. It's currently taking about 0.65 seconds to find the appropriate row.
The above answers lead me to realize two things.
1) When using a VARCHAR index, the query criteria needs to be quoted or MySQL will refuse to use the index (implicitly casting behind the scenes?)
SELECT * FROM foo WHERE column = '123'; # do this
SELECT * FROM foo where column = 123; # don't do this
2) You're better off using/indexing an INT if at all possible.
Related
I'll go to the point. I have the following table (MySQL 5.6.30)
CREATE TABLE `rankings` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` varchar(255) DEFAULT NULL,
`group_id` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `rankings_user_id_index` (`user_id`),
KEY `rankings_group_id_index` (`group_id`)
) ENGINE=InnoDB AUTO_INCREMENT=100 DEFAULT CHARSET=latin1 |
Now, in every test, I delete the records and insert new ones, so that
mysql> SELECT * FROM rankings WHERE user_id = 'hermione' and group_id = 'enchantments';
+-----+----------+--------------+
| id | user_id | group_id |
+-----+----------+--------------+
| 122 | hermione | enchantments |
| 123 | hermione | enchantments |
+-----+----------+--------------+
2 rows in set (0.00 sec)
However, the test fails when checking the total count (for pagination purposes, I only retrieve part of the records)
mysql> SELECT count(*) FROM rankings WHERE user_id = 'hermione' and group_id = 'enchantments';
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)
Which is a very strange behavior... Digging a bit more, I've observed that:
If I don't use an auto-increment field, everything works fine.
If I insert a few more records manually, at some point the second query starts working as expected (caching issue?).
The explain query for the wrong and right results (in that order) is the following:
// WRONG
{
select_type: SIMPLE,
type: ref,
possible_keys: rankings_user_id_index,rankings_group_id_index,
key: rankings_user_id_index,
ref: const,
rows: 10
extra: Using index condition; Using where
}
// RIGHT
{
select_type: SIMPLE,
type: index_merge,
possible_keys: rankings_user_id_index,rankings_group_id_index,
key: rankings_user_id_index,rankings_medgroup_id_index,
ref: NULL,
rows: 2
extra: Using intersect(rankings_user_id_index,rankings_group_id_index); Using where; Using index
}
I do not understand the internals of MySQL, but I assume it tries to optimize the count in some way. However, it fails at that (at least in a test environment, where the table records are being deleted and recreated before every test case). Am I missing some cleaning step? Is MySQL's count caching expected to work this way?
Thank you
For faster search i have indexed two columns(composite index) client_id and batch_id.
Below is my output of indexes of my table
show indexes from authentication_codes
*************************** 3. row ***************************
Table: authentication_codes
Non_unique: 1
Key_name: client_id
Seq_in_index: 1
Column_name: client_id
Collation: A
Cardinality: 18
Sub_part: NULL
Packed: NULL
Null: YES
Index_type: BTREE
Comment:
Index_comment:
*************************** 4. row ***************************
Table: authentication_codes
Non_unique: 1
Key_name: client_id
Seq_in_index: 2
Column_name: batch_id
Collation: A
Cardinality: 18
Sub_part: NULL
Packed: NULL
Null: YES
Index_type: BTREE
Comment:
Index_comment:
4 rows in set (0.02 sec)
ERROR:
No query specified
when i use explain to check if indexing is used in query or not it gives me below output.
mysql> explain select * from authentication_codes where client_id=6 and batch_id="101" \G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: authentication_codes
type: ref
possible_keys: client_id
key: client_id
key_len: 773
ref: const,const
rows: 1044778
Extra: Using where
1 row in set (0.00 sec)
ERROR:
No query specified
********************EDIT***************************
output of show create table authentication_codes is as below
mysql> show create table authentication_codes \G;
*************************** 1. row ***************************
Table: authentication_codes
Create Table: CREATE TABLE `authentication_codes` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`code` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`batch_id` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`serial_num` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`client_id` int(11) DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `index_authentication_codes_on_code` (`code`),
KEY `client_id_batch_id` (`client_id`,`batch_id`)
) ENGINE=InnoDB AUTO_INCREMENT=48406205 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
1 row in set (0.00 sec)
my question is why batch_id column is not used for searching. why only client_id column is used for searching??
To use index on two columns you need to create two column index. MySQL cannot use two separate indexes on one table.
This query will add multi column index on client_id and batch_id
alter table authentication_codes add index client_id_batch_id (client_id,batch_id)
http://dev.mysql.com/doc/refman/5.7/en/multiple-column-indexes.html
The EXPLAIN does not match the CREATE TABLE, at least in the name of the relevant index.
Explaining the EXPLAIN (as displayed at the moment):
select_type: SIMPLE
table: authentication_codes
type: ref
possible_keys: client_id
key: client_id -- The index named "client_id" was used
key_len: 773 -- (explained below)
ref: const,const -- 2 constants were used for the first two columns in that index
rows: 1044778 -- About this many rows (2% of table) matches those two constants
Extra: Using where
773 = 2 + 3 * 255 + 1 + 4 + 1
2 = length for VARCHAR
3 = max width of a utf8 character -- do you really need utf8?
255 = max length provided in VARCHAR(255) -- do you really need that much?
1 = extra length for NULL -- perhaps your columns could/should be NOT NULL?
4 = length of INT for client_id -- if you don't need 4 billion ids, maybe a smaller INT would work? and maybe UNSIGNED, too?
So, yes, it is using both parts of client_id=6 and batch_id="101". But there are a million rows in that batch for that client, so the query takes time.
If you want to discuss how to further speed up the use of this table, please provide the other common queries. (I don't want to tweak the schema to make this query faster, only to find that other queries are made slower.)
I have a table (location_data) with 300M rows (and growing).
I want to find the id of the latest entry (MAX(id)) for a particular value of foo_id.
Is the query below optimal? It often takes a few seconds.
Does the "rows" value of 561826 mean that MySQL is reading and scanning through all the rows for a foo_id, event though it should be able to immediately jump to the last id for a foo_id, using the composite index?
Why is "keylen" 4, when it is made of two 32 bit ints?
Most importantly, how can I speed up this query?
mysql> explain
-> SELECT foo_id, MAX(id) id
-> FROM location_data l
-> WHERE l.foo_id = 253
-> GROUP BY foo_id;
+----+-------------+-------+------+---------------+---------+---------+-------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+---------+---------+-------+--------+-------------+
| 1 | SIMPLE | l | ref | Index 5 | Index 5 | 4 | const | 561826 | Using index |
+----+-------------+-------+------+---------------+---------+---------+-------+--------+-------------+
1 row in set (0.00 sec)
mysql> SHOW CREATE TABLE bar.location_data;
...
CREATE TABLE `location_data` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`foo_id` int(11) NOT NULL,
`ts_lastpos` datetime DEFAULT NULL,
`lat` double NOT NULL,
`lng` double NOT NULL,
PRIMARY KEY (`id`),
KEY `Index 5` (`foo_id`,`id`)
) ENGINE=MyISAM AUTO_INCREMENT=562767448 DEFAULT CHARSET=latin1
...
1 row in set (0.00 sec)
I tried this:
SELECT MAX(id) id
FROM location_data l
WHERE l.foo_id = 2;
The result is still MAX(id) for appropriate l.foo_id. There is no need to retrieve foo_id as you already know it before the query.
As soon as I removed GROUP BY, EXPLAIN started giving this:
mysql> EXPLAIN SELECT MAX(id) id
-> FROM location_data l
-> WHERE l.foo_id = 2\G
*************************** 1. row **********************
id: 1
select_type: SIMPLE
table: NULL
type: NULL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: NULL
Extra: Select tables optimized away
1 row in set (0.00 sec)
It means you have done a query that does nothing more than count the
number of rows in a table, and that table is a MyISAM table. MyISAM
tables are stored with a separate row count, so to do this query MySQL
doesn't need to look at any of the table row data at all. Instead it
immediately returns the pre-calculated row count. Hence the table
access is ‘optimized away’ and the query is lightning-fast.
Meaning of "Select tables optimized away" in MySQL Explain plan
So, I think, getting rid of GROUP BY will speed up your query.
The Facts:
Dedicated Server, 4 Cores, 16GB
MySQL 5.5.29-0ubuntu0.12.10.1-log - (Ubuntu)
One Table, 1.9M rows and growing
I need all sorted rows for export or a 5er chunk. The query takes 25 seconds with Copying To Tmp Table 23.3 s
I tried InnoDB and MyISAM, changing the index order, using a MD5 Hash of some_text as GROUP BY, partition the table by day.
dayis a Unix-Timestamp and alway present.
lang some_bool some_filter ano_filter rel_id could be in where clause but not need to.
Here is the MyISAM example:
The table
mysql> SHOW CREATE TABLE data \G;
*************************** 1. row ***************************
Table: data
Create Table: CREATE TABLE `data` (
`data_id` bigint(20) NOT NULL AUTO_INCREMENT,
`rel_id` int(11) NOT NULL,
`some_text` varchar(255) DEFAULT NULL,
`lang` varchar(3) DEFAULT NULL,
`some_bool` tinyint(1) DEFAULT NULL,
`some_filter` varchar(40) DEFAULT NULL,
`ano_filter` varchar(10) DEFAULT NULL,
`day` int(11) DEFAULT NULL,
PRIMARY KEY (`data_id`),
KEY `cnt_idx` (`some_filter`,`ano_filter`,`rel_id`,`lang`,`some_bool`,`some_text`,`day`)
) ENGINE=MyISAM AUTO_INCREMENT=1900099 DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
The query
mysql> EXPLAIN SELECT `some_text` , COUNT(*) AS `num` FROM `data`
WHERE `lang` = 'en' AND `day` BETWEEN '1364342400' AND
'1366934399' GROUP BY `some_text` ORDER BY `num` DESC \G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: data
type: index
possible_keys: NULL
key: cnt_idx
key_len: 947
ref: NULL
rows: 1900098
Extra: Using where; Using index; Using temporary; Using filesort
1 row in set (0.00 sec)
mysql> SELECT `some_text` , COUNT(*) AS `num` FROM `data`
WHERE `lang` = 'en' AND `day` BETWEEN '1364342400' AND '1366934399'
GROUP BY `some_text` ORDER BY `num` DESC LIMIT 5 \G;
...
*************************** 5. row ***************************
5 rows in set (24.26 sec)
Any idea how to speed up that thing?`
No index is being used because of the column order in the index. Indexes work left to right. For this query to use an index, you would need an index of lang, day.
my query is running longer than 30 minutes .it is a simple query even it contains indexes also.we are unable to find why it was taking too much execution time and it effects on our entire db performance.
yesterday it ran around: 122.6mins
any one can help me here.how to improve query performance
This is my query:
SELECT tab1.customer_id,tab1.row_mod,tab1.row_create,tab1.event_id,tab1.event_type,
tab1.new_value,tab1.old_value FROM tab1 force index (tab1_n2)where customer_id >= 1 and customer_id
< 5000000 and (tab1.row_mod >= '2012-10-01') or (tab1.row_create >= '2012-10-01' and tab1.row_create < '2012-10-13');
Explain plan
+----+-------------+------------------+------+---------------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+---------------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | tab1 | ALL | tab1_n2 | NULL | NULL | NULL | 18490530 | Using where |
+----+-------------+------------------+------+---------------------+------+---------+------+----------+-------------+
1 row in set (0.00 sec)
Table structure:
mysql> show create table tab1\G
*************************** 1. row ***************************
Table: tab1
Create Table: CREATE TABLE `tab1` (
`customer_id` int(11) NOT NULL,
`row_mod` datetime DEFAULT NULL,
`row_create` datetime DEFAULT NULL,
`event_id` int(11) DEFAULT NULL,
`event_type` varchar(45) DEFAULT NULL,
`new_value` varchar(255) DEFAULT NULL,
`old_value` varchar(255) DEFAULT NULL,
KEY `customer_id1` (`customer_id`),
KEY `new_value_n1` (`new_value`),
KEY `tab1_n1` (`row_create`),
KEY `tab1_n2` (`row_mod`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
Please help me how to tune it .even it having indexes also
Probably because you are using an index that does not make sense.
The row_mod condition is only one branch of the OR condition, so that index is not much help here. If you are forcing every lookup through the index without eliminating any rows, that could be a lot slower than a full table scan. Good rule of thumb is that an index should eliminate more than 90% of rows.
Try to do without the "force index" part.
Try using a UNION of the two conditions. That way each condition can use an index.
ALTER TABLE tab1 ADD INDEX idx_row_mod_customer_id (row_mod, customer_id);
ALTER TABLE tab1 ADD INDEX idx_row_create (row_create);
SELECT tab1.customer_id, tab1.row_mod, tab1.row_create, tab1.event_id, tab1.event_type,
tab1.new_value, tab1.old_value
FROM tab1
WHERE customer_id >= 1 and customer_id
< 5000000 AND tab1.row_mod >= '2012-10-01'
UNION
SELECT tab1.customer_id, tab1.row_mod, tab1.row_create, tab1.event_id, tab1.event_type,
tab1.new_value, tab1.old_value
FROM tab1
WHERE tab1.row_create >= '2012-10-01' AND tab1.row_create < '2012-10-13';
To optimise further, you could add all selected columns to both indices, saving MySQL from having to load the rows into memory. This will greatly increase the size of the indices, and therefore their memory requirement.