when i use id (primary key) with order by clause it uses index named PRIMARY but when i use countrycode (foreign key) with order by clause it does't uses index. my output is below.
mysql> SHOW CREATE TABLE City;
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| City | CREATE TABLE `City` (
| | `ID` int(11) NOT NULL AUTO_INCREMENT,
| | `Name` char(35) NOT NULL DEFAULT '',
| | `CountryCode` char(3) NOT NULL DEFAULT '',
| | `District` char(20) NOT NULL DEFAULT '',
| | `Population` int(11) NOT NULL DEFAULT '0',
| | PRIMARY KEY (`ID`),
| | KEY `CountryCode` (`CountryCode`),
| | CONSTRAINT `city_ibfk_1` FOREIGN KEY (`CountryCode`) REFERENCES `Country` (`Code`)
| | ) ENGINE=InnoDB AUTO_INCREMENT=4080 DEFAULT CHARSET=latin1 |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
mysql> EXPLAIN SELECT * FROM City ORDER BY ID;
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------+
| 1 | SIMPLE | City | index | NULL | PRIMARY | 4 | NULL | 4321 | |
+----+-------------+-------+-------+---------------+---------+---------+------+------+-------+
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT * FROM City ORDER BY COUNTRYCODE;
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
| 1 | SIMPLE | City | ALL | NULL | NULL | NULL | NULL | 4321 | Using filesort |
+----+-------------+-------+------+---------------+------+---------+------+------+----------------+
1 row in set (0.00 sec)
In innodb primary keys are so called "clustered indexes".
It means that the rows physically ordered according the PK values.
Because of that the rows are naturally sorted so it's cheap to read them sorted ASC or DESC.
Another story is when you order by another column.
To use that mysql would have to read both index and data pages, which dramatically increases IO. So mysql decides to sort it in memory instead (because according to its heuristics memory sort is faster than increased IO). If you want to see mysql using that index for sorting you need:
Increase total number of rows to, say several dozens of thousands
Select only a small subset, like LIMIT 10
Then mysql might decide to use index.
Related
I have a simple table like this,
CREATE TABLE `domain` (
`id` varchar(191) NOT NULL,
`time` bigint(20) DEFAULT NULL,
`task_id` bigint(20) DEFAULT NULL,
`name` varchar(512) DEFAULT NULL
PRIMARY KEY (`id`),
KEY `idx_domain_time` (`time`),
KEY `idx_domain_task_id` (`task_id`),
FULLTEXT KEY `idx_domain_name` (`name`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
And indexed like this:
mysql> show index from domain;
+--------+------------+------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Ignored |
+--------+------------+------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+
| domain | 0 | PRIMARY | 1 | id | A | 2036092 | NULL | NULL | | BTREE | | | NO |
| domain | 1 | idx_domain_name | 1 | name | NULL | NULL | NULL | NULL | YES | FULLTEXT |
+--------+------------+------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+
Index is used when I select only the id field:
mysql> explain SELECT id FROM `domain` WHERE task_id = '3';
+------+-------------+--------+------+--------------------+--------------------+---------+-------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+--------+------+--------------------+--------------------+---------+-------+---------+-------------+
| 1 | SIMPLE | domain | ref | idx_domain_task_id | idx_domain_task_id | 9 | const | 1018046 | Using index |
+------+-------------+--------+------+--------------------+--------------------+---------+-------+---------+-------------+
1 row in set (0.00 sec)
When I select all fields, it does not work:
mysql> explain SELECT * FROM `domain` WHERE task_id = '3';
+------+-------------+--------+------+--------------------+------+---------+------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+--------+------+--------------------+------+---------+------+---------+-------------+
| 1 | SIMPLE | domain | ALL | idx_domain_task_id | NULL | NULL | NULL | 2036092 | Using where |
+------+-------------+--------+------+--------------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)
mysql> explain SELECT id, name FROM `domain` WHERE task_id = '3';
+------+-------------+--------+------+--------------------+------+---------+------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+--------+------+--------------------+------+---------+------+---------+-------------+
| 1 | SIMPLE | domain | ALL | idx_domain_task_id | NULL | NULL | NULL | 2036092 | Using where |
+------+-------------+--------+------+--------------------+------+---------+------+---------+-------------+
1 row in set (0.00 sec)
What's wrong?
Indexes other than the Primary Key work by storing data for the indexed field(s) in index order, along with the primary key.
So when you SELECT the primary key by the indexed field, there is enough information in the index to completely satisfy the query. When you add other fields, there's no longer enough information in the index. That doesn't mean the database won't use the index, but now it's no longer as much of a slam dunk, and it comes down more to table statistics.
MySql optimizer will try to achieve the best performance so it may ignore an index. You can force optimizer to use the index you want if you are sure that will give you better performance. You can use :
SELECT * FROM `domain` USE INDEX (idx_domain_task_id) WHERE task_id = '3';
For more details please see this page Index Hints .
I has table with the same schema
CREATE TABLE `stock` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`currency` varchar(3) COLLATE utf8_unicode_ci NOT NULL,
`against` varchar(3) COLLATE utf8_unicode_ci NOT NULL,
`date` date NOT NULL,
`time` time NOT NULL,
`rate` double(8,4) NOT NULL,
`ask` double(8,4) NOT NULL,
`bid` double(8,4) NOT NULL,
`created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`id`),
KEY `stock_currency_index` (`currency`),
KEY `stock_against_index` (`against`),
KEY `stock_date_index` (`date`),
KEY `stock_time_index` (`time`),
KEY `created_at_index` (`created_at`)
) ENGINE=InnoDB AUTO_INCREMENT=244221 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
When i execute this query mysql has using index
mysql> explain select max(id) from stock group by currency;
+----+-------------+-------+------------+-------+----------------------+----------------------+---------+------+------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+----------------------+----------------------+---------+------+------+----------+--------------------------+
| 1 | SIMPLE | stock | NULL | range | stock_currency_index | stock_currency_index | 11 | NULL | 2 | 100.00 | Using index for group-by |
+----+-------------+-------+------------+-------+----------------------+----------------------+---------+------+------+----------+--------------------------+
1 row in set, 1 warning (0.00 sec)
Also when i am executing this query mysql has using primary index too
mysql> explain select * from stock where id in (244221, 244222);
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | stock | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.00 | Using where |
+----+-------------+-------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0.00 sec)
BUT when i am combine these two queries PRIMARY index are not using... i am confused. What i am doing wrong?
mysql> explain select * from stock where id in (select max(id) from stock group by currency);
+----+-------------+-------+------------+-------+----------------------+----------------------+---------+------+--------+----------+--------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+----------------------+----------------------+---------+------+--------+----------+--------------------------+
| 1 | PRIMARY | stock | NULL | ALL | NULL | NULL | NULL | NULL | 221800 | 100.00 | Using where |
| 2 | SUBQUERY | stock | NULL | range | stock_currency_index | stock_currency_index | 11 | NULL | 2 | 100.00 | Using index for group-by |
+----+-------------+-------+------------+-------+----------------------+----------------------+---------+------+--------+----------+--------------------------+
2 rows in set, 1 warning (0.00 sec)
First, try rewriting the query as:
select s.*
from stock s join
(select max(id) as maxid
from stock
group by currency
) ss
on ss.maxid = s.id;
Second, I would be tempted to put an index on stock(currency, id) and to use:
select s.*
from stock s
where s.id = (select max(s2.id) from stock s2 where s2.currency = s.currency);
Do either of these perform better?
I have the following MySQL (MyISAM) table with about 3 Million rows.
CREATE TABLE `tasks` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`node` smallint(6) NOT NULL,
`pid` int(11) NOT NULL,
`job` int(11) NOT NULL,
`a_id` int(11) DEFAULT NULL,
`user_id` int(11) NOT NULL,
`state` int(11) NOT NULL,
`start_time` int(11) NOT NULL,
`end_time` int(11) NOT NULL,
`stop_time` int(11) NOT NULL,
`end_stream` int(11) NOT NULL,
`message` varchar(255) DEFAULT NULL,
`rate` float NOT NULL,
`exiting` int(11) NOT NULL DEFAULT '0',
`bytes` int(11) NOT NULL,
`motion` tinyint(4) NOT NULL,
PRIMARY KEY (`id`),
KEY `a_id` (`a_id`),
KEY `job` (`job`),
KEY `state` (`state`),
KEY `end_time` (`end_time`),
KEY `start_time` (`start_time`),
) ENGINE=MyISAM AUTO_INCREMENT=100 DEFAULT CHARSET=utf8;
Now when I run the following query, MySQL is only using the a_id index and needs to scan a few thousand rows.
SELECT count(id) AS tries FROM `tasks` WHERE ( job='1' OR job='3' )
AND a_id='614' AND state >'80' AND state < '100' AND start_time >='1386538013';
When I add an additional index KEY newkey (a_id,state,start_time), MySQL is still trying to use a_id only and not newkey. Only when using the hint / force index in the query, it's been used. Changing the fields in the query around does not help.
Any ideas? I don't necessarily want hints in my statements. The fact that MySQL is not doing this automatically indicates to me that there is an issue with my table, keys or query somewhere. Any help is highly appreciated.
Additional info:
mysql> show index in tasks;
+-------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| tasks | 0 | PRIMARY | 1 | id | A | 3130554 | NULL | NULL | | BTREE | | |
| tasks | 1 | a_id | 1 | a_id | A | 2992 | NULL | NULL | YES | BTREE | | |
| tasks | 1 | job | 1 | job | A | 5 | NULL | NULL | | BTREE | | |
| tasks | 1 | state | 1 | state | A | 9 | NULL | NULL | | BTREE | | |
| tasks | 1 | end_time | 1 | end_time | A | 1565277 | NULL | NULL | | BTREE | | |
| tasks | 1 | newkey | 1 | a_id | A | 2992 | NULL | NULL | YES | BTREE | | |
| tasks | 1 | newkey | 2 | state | A | 8506 | NULL | NULL | | BTREE | | |
| tasks | 1 | newkey | 3 | start_time | A | 3130554 | NULL | NULL | | BTREE | | |
+-------+------------+-----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
EXPLAIN with and without quotes:
mysql> DESCRIBE SELECT count(id) AS tries FROM `tasks` WHERE ( job='1' OR job='3' ) AND a_id='614' AND state >'80' AND state < '100' AND start_time >='1386538013';
+----+-------------+-------+------+----------------------------+-----------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+----------------------------+-----------+---------+-------+------+-------------+
| 1 | SIMPLE | tasks | ref | a_id,job,state,newkey | a_id | 5 | const | 740 | Using where |
+----+-------------+-------+------+----------------------------+-----------+---------+-------+------+-------------+
1 row in set (0.10 sec)
mysql> DESCRIBE SELECT count(id) AS tries FROM `tasks` WHERE ( job=1 OR job=3 ) AND a_id = 614 AND state > 80 AND state < 100 AND start_time >= 1386538013;
+----+-------------+-------+------+----------------------------+-----------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+----------------------------+-----------+---------+-------+------+-------------+
| 1 | SIMPLE | tasks | ref | a_id,job,state,newkey | a_id | 5 | const | 740 | Using where |
+----+-------------+-------+------+----------------------------+-----------+---------+-------+------+-------------+
1 row in set (0.01 sec)
A few things... I would have a SINGLE compound index on
( a_id, job, state, start_time )
This to help optimize the query on all the criteria, in what I believe is the best tuned sequence. A single "A_ID", then two jobs, a small state range, then time based. Next, notice no quotes... It appears you were converting numeric to string comparisons, leave them as numeric for compare -- faster than strings.
Also, by having them all as part of the index, it is a COVERING index meaning it does NOT have to go to the raw page data to get the other values to test the qualifying records to include or not.
SELECT
count(*) AS tries
FROM
tasks
WHERE
a_id = 614
AND job IN ( 1, 3 )
AND state > 80 AND state < 100
AND start_time >= 1386538013;
Now, the why the index... consider the following scenario. You have two rooms that have boxes... In the first room, each box is an "a_id", within that are the jobs in order, within each job are the state ranges, and finally by start time.
In another room, your boxes are sorted by start time, within that a_id are sorted, and finally state.
Which would be easier to find what you need. That is how you should think on the indexes. I would rather go to one box for "A_ID = 614", then jump to Job 1 and another for Job 3. Within each Job 1, Job 3, grab 80-100, then time. You however know better your data and volume in each criteria consideration and may adjust.
Finally, the count(ID) vs count(*). All I care about is a record qualified. I don't need to know the actual ID as the filtering criteria already qualified as include or not, why look (in this case) for the actual "ID".
Probably mysql thinks that using the a_id key will using less IO.
Probably the cardinality of the key a_id is good enough.
What explains of the hinted/hintless queries say?
Most of a_id=614's state has > 80 and < 100, then it could be happened. Have you tried one of below indexes?
INDEX(a_id, start_time, state)
INDEX(start_time, a_id, state)
I'm trying to order by and use an index but it seems to still be using filesort
Table
CREATE TABLE `teachers` (
`id` int(10) unsigned NOT NULL,
`name` VARCHAR(10) NOT NULL,
`lastName` tinytext NOT NULL,
KEY `lastName` (lastName(10))
)
Query
mysql> EXPLAIN SELECT name,lastName FROM teachers ORDER BY lastName DESC;
+----+-------------+----------+------+---------------+------+---------+------+------+----------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+------+---------------+------+---------+------+------+----------------+
| 1 | SIMPLE | teachers | ALL | NULL | NULL | NULL | NULL | 1546 | Using filesort |
+----+-------------+----------+------+---------------+------+---------+------+------+----------------+
You need to create a covering index last name, name
CREATE TABLE `teachers` (
`id` int(10) unsigned NOT NULL,
`name` varchar(10) NOT NULL,
`lastName` varchar(255) NOT NULL,
PRIMARY KEY (`id`),
KEY `last_first` (`lastName`,`name`)
)
Now that query
EXPLAIN SELECT `name`, lastName FROM teachers ORDER BY lastName DESC;
Produces the following result
+----+-------------+----------+-------+---------------+------------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+-------+---------------+------------+---------+------+------+-------------+
| 1 | SIMPLE | teachers | index | NULL | last_first | 269 | NULL | 1 | Using index |
+----+-------------+----------+-------+---------------+------------+---------+------+------+-------------+
Your problem is in key definition: KEY lastName (lastName(10))
Especially that you have defined a KEY length here
You index only a prefix of a column named in the ORDER BY clause. In this case, the index cannot be used to fully resolve the sort order. For example, if you have a CHAR(20) column, but index only the first 10 bytes, the index cannot distinguish values past the 10th byte and a filesort will be needed.
Proof link
You can write KEY lastName (lastName) and it will work as you need.
According to MySQL docs a composite index will still be used if the leftmost fields are part of the criteria. However, this table will not join correctly with the primary key; I had to add another index of the left two fields which is then used.
One of the tables is memory, and I know that by default memory uses a hash index which can't be used for group/order. However I'm using all rows of the memory table and not the index, so I don't think that relates to the problem.
What am I missing?
mysql> show create table pr_temp;
| pr_temp | CREATE TEMPORARY TABLE `pr_temp` (
`player_id` int(10) unsigned NOT NULL,
`insert_date` date NOT NULL,
[...]
PRIMARY KEY (`player_id`,`insert_date`) USING BTREE,
KEY `insert_date` (`insert_date`)
) ENGINE=MEMORY DEFAULT CHARSET=utf8 |
mysql> show create table player_game_record;
| player_tank_record | CREATE TABLE `player_game_record` (
`player_id` int(10) unsigned NOT NULL,
`game_id` smallint(5) unsigned NOT NULL,
`insert_date` date NOT NULL,
[...]
PRIMARY KEY (`player_id`,`insert_date`,`game_id`),
KEY `insert_date` (`insert_date`),
KEY `player_date` (`player_id`,`insert_date`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 DATA DIRECTORY='...' INDEX DIRECTORY='...' |
mysql> explain select pgr.* from player_game_record pgr inner join pr_temp on pgr.player_id = pr_temp.player_id and pgr.insert_date = pr_temp.date_prev;
+----+-------------+---------+------+---------------------------------+-------------+---------+-------------------------------------------------------------------------+--------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+------+---------------------------------+-------------+---------+-------------------------------------------------------------------------+--------+-------+
| 1 | SIMPLE | pr_temp | ALL | PRIMARY | NULL | NULL | NULL | 174683 | |
| 1 | SIMPLE | pgr | ref | PRIMARY,insert_date,player_date | player_date | 7 | test_gamedb.pr_temp.player_id,test_gamedb.pr_temp.date_prev | 21 | |
+----+-------------+---------+------+---------------------------------+-------------+---------+-------------------------------------------------------------------------+--------+-------+
2 rows in set (0.00 sec)
mysql> explain select pgr.* from player_game_record pgr force index (primary) inner join pr_temp on pgr.player_id = pr_temp.player_id and pgr.insert_date = pr_temp.date_prev;
+----+-------------+---------+------+---------------+---------+---------+-------------------------------------------------------------------------+---------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+------+---------------+---------+---------+-------------------------------------------------------------------------+---------+-------+
| 1 | SIMPLE | pr_temp | ALL | PRIMARY | NULL | NULL | NULL | 174683 | |
| 1 | SIMPLE | pgr | ref | PRIMARY | PRIMARY | 7 | test_gamedb.pr_temp.player_id,test_gamedb.pr_temp.date_prev | 2873031 | |
+----+-------------+---------+------+---------------+---------+---------+-------------------------------------------------------------------------+---------+-------+
2 rows in set (0.00 sec)
I think the primary key should work, with the two left columns (player_id, insert_date) being used. However it will use the player_date index by default, and if I force it to use the primary index it looks like it's only using one field rather than both.
Update2: Mysql version 5.5.27-log
Update3:
(note this is after removing the player_date index while trying some other tests)
mysql> show indexes in player_game_record;
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| player_game_record | 0 | PRIMARY | 1 | player_id | A | NULL | NULL | NULL | | BTREE | | |
| player_game_record | 0 | PRIMARY | 2 | insert_date | A | NULL | NULL | NULL | | BTREE | | |
| player_game_record | 0 | PRIMARY | 3 | game_id | A | 576276246 | NULL | NULL | | BTREE | | |
| player_game_record | 1 | insert_date | 1 | insert_date | A | 33304 | NULL | NULL | | BTREE | | |
+--------------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
4 rows in set (1.08 sec)
mysql> select count(*) from player_game_record;
+-----------+
| count(*) |
+-----------+
| 576276246 |
+-----------+
1 row in set (0.00 sec)
I agree that your use of the MEMORY storage engine for one of the tables should not at all be an issue here, since we're talking about the other table.
I also agree that the leftmost prefix of an index can be used exactly how you are trying to use it, and I cannot think of any reason why the primary key could not be used in exactly the same way as any other index.
This has been a head-scratcher. The new index you created "should" be the same as the left side of the primary key, so why don't they behave the same way? I have two thoughts, both of which lead me to the same recommendation, even though I am not as familiar with the internals of MyISAM as I am with InnoDB. (As an aside, I'd recommend InnoDB over MyISAM.)
The index on your primary key was presumably on the table when you began inserting data, while the new index was added while most or all of the data was already there. This suggests that your new index is nice and cleanly-organized internally, while your primary key index may be highly fragmented, having been built as the data was loaded.
The row count the optimizer shows is based on index statistics, which may be inaccurate on your primary key due to the insert order.
The fragmentation theory may explain why querying with the primary key as your index is not as fast; the index statistics theory may explain why the optimizer comes up with such a different row count and it may explain why the optimizer might have been choosing a full table scan instead of using that index (which is only a guess, since we don't have the explain available).
The thing I would suggest based on these two thoughts is running OPTIMIZE TABLE on your table. If it took 12 hours to build that new index, then optimizing the table may very possibly take that long or longer.
Possibly helpful: http://www.dbasquare.com/2012/07/09/data-fragmentation-problem-in-mysql-myisam/