I have very strange problem with mySQL and simple query with simple index. I am trying to use this query:
SELECT * FROM `counter_links` WHERE `link_id`=1544;
as you can see there is also index named link_id:
mysql> show indexes from counter_links;
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
| counter_links | 1 | link_id | 1 | link_id | A | NULL | NULL | NULL | | BTREE | disabled | |
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
1 row in set (0.12 sec)
But look what EXPLAIN returned:
mysql> explain SELECT * FROM `counter_links` WHERE `link_id`=1544;
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | counter_links | ALL | NULL | NULL | NULL | NULL | 22103687 | Using where |
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
1 row in set (0.03 sec)
Why mySQL don't use index there? I see that the same is working good on smaller version of the table, but I am not able to recognize where the problem is. You do?
It is more stranger that it seems, because lower IDs sometimes works with the index.
Thanks you for all comments in advance!
Cheers!
Jakub
The comment field for th eindex says that the index is disabled, not sure why.
ALTER TABLE ... ENABLE KEYS may work
Related
i'am having these indexes on my table:
mysql> SHOW INDEXES from sous_categories;
+-----------------+------------+----------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression |
+-----------------+------------+----------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| sous_categories | 0 | PRIMARY | 1 | id_sous_categorie | A | 16 | NULL | NULL | | BTREE | | | YES | NULL |
| sous_categories | 0 | PRIMARY | 2 | categorie_id | A | 16 | NULL | NULL | | BTREE | | | YES | NULL |
+-----------------+------------+----------+--------------+-------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
2 rows in set (0.01 sec)
and i want to remove the second column, by checking MySQL INDEX STATMENT, i figured out that i can drop only by index_name, but in my case both has the same key_name.
How can i do that please ? (I know it's possible because i can do it with phpmyadmin but i want to know how to do it with command line)
ALTER TABLE sous_categories
DROP PRIMAR KEY,
ADD PRIMARY KEY(id_sous_categorie);
However, to be the PK, id_sous_categorie must be Unique (no dups) and NOT NULL.
The only effects will be
Time spent rebuilding the table. (Any change to the PK requires a rebuild.)
A fatal error if id_sous_categorie is not already unique. (Previously the pair of columns was constrained to be unique, but not each column.)
I found that the same query gets logged in my slow-log again and again, but I don't really know how to make it any faster. I believe I did most of the optimization possible. The query is as follows:
SELECT merk_id FROM profile_merk WHERE profile_id = '51485e0d9ce6d'
The profile_id varies with every query of course. The table has about 40 mln rows and 2 columns. One column is merk_id and the other is profile_id. I have the following indexes on the table:
Tbl | Non_unique | Key_name | Seq_in_index | Column_name | Collation| Cardinality | Sub_part
profile_merk | 0 | PRIMARY | 1 | merk_id | A | 4125 | NULL | NULL | BTREE
profile_merk | 0 | PRIMARY | 2 | profile_id | A | 39101247 | NULL | NULL | BTREE
profile_merk | 1 | profile_id | 1 | profile_id | A| 131212 | NULL | NULL | BTREE
profile_merk | 1 | merk_id | 1 | merk_id | A | 4125 | NULL | NULL | BTREE
And the explain statement of the query gives the following:
id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
1 | SIMPLE | profile_merk | ref | profile_id | profile_id | 62 | const | 1688 | Using where
What more can I do to optimize the performance? I realise it's kind of an open question, but any help would be very much appreciated!
For this simple query:
SELECT merk_id
FROM profile_merk
WHERE profile_id = '51485e0d9ce6d';
The best performance is using a composite key:
create index idx_profilemark_profileid_merkid on profile_mark(profile_id, merk_id)
The problem with the primary key is that the two columns are not in the order needed for this query. There is a big difference between an index on profile_mark(profile_id, merk_id) and profile_mark(merk_id, profile_id). In essence, the index will be used from the left to the right with where clause getting the first chance to use index columns.
I have a table named test:
mysql> describe test;
+-------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+-------+
| coll1 | int(11) | YES | MUL | NULL | |
| coll2 | int(11) | YES | | NULL | |
| coll3 | int(11) | YES | | NULL | |
| coll4 | int(11) | YES | | NULL | |
+-------+---------+------+-----+---------+-------+
And I have an index on the table:
mysql> show index from test;
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------ +------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| test | 1 | hi | 1 | coll1 | A | 0 | NULL | NULL | YES | BTREE | | |
| test | 1 | hi | 2 | coll2 | A | 0 | NULL | NULL | YES | BTREE | | |
| test | 1 | hi | 3 | coll3 | A | 0 | NULL | NULL | YES | BTREE | | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
Because of the leftmost prefix index strategy, the following output is easy to understand.
mysql> explain select * from test where coll2=1;
+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | test | ALL | NULL | NULL | NULL | NULL | 1 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
And I know about the covering index stuff, but how does this happen? Does this mean the covering index got a higher priority than the leftmost prefix index strategy?
mysql> explain select coll1 from test where coll2=1;
+----+-------------+-------+-------+---------------+------+---------+------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+------+---------+------+------+--------------------------+
| 1 | SIMPLE | test | index | NULL | hi | 15 | NULL | 1 | Using where; Using index |
+----+-------------+-------+-------+---------------+------+---------+------+------+--------------------------+
P.S.
I'm using mysql version 5.6.13
Sorry for my bad English.
Files of table are consist of DATA file + INDEX files. Usually one composite INDEXed columns are subset all columns. This means certain INDEX file size is less than DATA file size. So following query is using hi INDEX scan which reads less disk than DATA full scan.
SELECT coll1 FROM test WHERE coll2=1;
But when you retrieve coll1, coll4 as follows, coll4 is not a part of hi index. So, full data scan is faster.
SELECT coll1, coll4 FROM test WHERE coll2=1;
Simple answer No
Long answer (still very simple explained)
No the MySQL optimizer will use cost based analyses on your query to calculate the lowest execution cost.
The covering index happens because the query is forcing MySQL optimizer into using an covering index because this is the cheapest execution possible.
select coll1 from test where coll2=1
Because coll1 and coll2 are both indexed MySQL optimizer sees it can satisfy the query result from accessing the index data only (covering index)
Motto off the story the cheapest access plan will get a higher priority
I've run into a serious MySQL performance bottleneck which I'm unable to understand and resolve. Here are the table structures, indexes and record counts (bear with me, it's only two tables):
mysql> desc elggobjects_entity;
+-------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------------+------+-----+---------+-------+
| guid | bigint(20) unsigned | NO | PRI | NULL | |
| title | text | NO | MUL | NULL | |
| description | text | NO | | NULL | |
+-------------+---------------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql> show index from elggobjects_entity;
+--------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| elggobjects_entity | 0 | PRIMARY | 1 | guid | A | 613637 | NULL | NULL | | BTREE | |
| elggobjects_entity | 1 | title | 1 | title | NULL | 131 | NULL | NULL | | FULLTEXT | |
| elggobjects_entity | 1 | title | 2 | description | NULL | 131 | NULL | NULL | | FULLTEXT | |
+--------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
3 rows in set (0.00 sec)
mysql> select count(*) from elggobjects_entity;
+----------+
| count(*) |
+----------+
| 613637 |
+----------+
1 row in set (0.00 sec)
mysql> desc elggentity_relationships;
+--------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+---------------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| guid_one | bigint(20) unsigned | NO | MUL | NULL | |
| relationship | varchar(50) | NO | MUL | NULL | |
| guid_two | bigint(20) unsigned | NO | MUL | NULL | |
| time_created | int(11) | NO | | NULL | |
+--------------+---------------------+------+-----+---------+----------------+
5 rows in set (0.00 sec)
mysql> show index from elggentity_relationships;
+--------------------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------------------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+
| elggentity_relationships | 0 | PRIMARY | 1 | id | A | 11408236 | NULL | NULL | | BTREE | |
| elggentity_relationships | 0 | guid_one | 1 | guid_one | A | NULL | NULL | NULL | | BTREE | |
| elggentity_relationships | 0 | guid_one | 2 | relationship | A | NULL | NULL | NULL | | BTREE | |
| elggentity_relationships | 0 | guid_one | 3 | guid_two | A | 11408236 | NULL | NULL | | BTREE | |
| elggentity_relationships | 1 | relationship | 1 | relationship | A | 11408236 | NULL | NULL | | BTREE | |
| elggentity_relationships | 1 | guid_two | 1 | guid_two | A | 11408236 | NULL | NULL | | BTREE | |
+--------------------------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+
6 rows in set (0.00 sec)
mysql> select count(*) from elggentity_relationships;
+----------+
| count(*) |
+----------+
| 11408236 |
+----------+
1 row in set (0.00 sec)
Now I'd like to use an INNER JOIN on those two tables and perform a full text search.
Query:
SELECT
count(DISTINCT o.guid) as total
FROM
elggobjects_entity o
INNER JOIN
elggentity_relationships r on (r.relationship="image" AND r.guid_one = o.guid)
WHERE
((MATCH (o.title, o.description) AGAINST ('scelerisque' )))
This gave me a 6 minute (!) response time.
On the other hand this one
SELECT
count(DISTINCT o.guid) as total
FROM
elggobjects_entity o
INNER JOIN
elggentity_relationships r on (r.relationship="image" AND r.guid_one = o.guid)
WHERE
((o.title like "%scelerisque%") OR (o.description like "%scelerisque%"))
returned the same count value in 0.02 seconds.
How is that possible? What am I missing here?
(MySQL info: mysql Ver 14.14 Distrib 5.1.49, for debian-linux-gnu (x86_64) using readline 6.1)
EDIT
EXPLAINing the first query (using match .. against) gives:
+----+-------------+-------+----------+-----------------------+--------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+----------+-----------------------+--------------+---------+-------+------+-------------+
| 1 | SIMPLE | r | ref | guid_one,relationship | relationship | 152 | const | 6145 | Using where |
| 1 | SIMPLE | o | fulltext | PRIMARY,title | title | 0 | | 1 | Using where |
+----+-------------+-------+----------+-----------------------+--------------+---------+-------+------+-------------+
while the second query (using LIKE "%..%"):
+----+-------------+-------+--------+-----------------------+--------------+---------+---------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------+--------------+---------+---------------------+------+-------------+
| 1 | SIMPLE | r | ref | guid_one,relationship | relationship | 152 | const | 6145 | Using where |
| 1 | SIMPLE | o | eq_ref | PRIMARY | PRIMARY | 8 | elgg1710.r.guid_one | 1 | Using where |
+----+-------------+-------+--------+-----------------------+--------------+---------+---------------------+------+-------------+
By combining your experience and EXPLAIN's results, it seems that fulltext index is not as useful as you expect in this particular case. This depends on particular data in your database, on database structure or/and particular query.
Usually database engines use no more than one index per table. So when the table has more than one index, query optimizer tries to use the better one. But optimizer is not always clever enough.
EXPLAIN's output shows that database query optimizer decided to use indexes for relationship and title. The relationship filter reduces table elggentity_relationships to 6145 rows. And the title filter reduces the table elggobjects_entity to 72697 rows. Then MySQL needs to join those tables (6145 x 72697 = 446723065 filtering operations) without using any index because indexes have already been used for filtering. In this case this can be too much. MySQL can even make a decision to keep intermediate calculations in the hard drive by trying to keep enough free space in memory.
Now let's take a look into another query. It uses relationship and PRIMARY KEY (of table elggobjects_entity) as its indexes. The relationship filter reduces table elggentity_relationships to 6145 rows. By joining those tables on PRIMARY KEY index, the result gets only 3957 rows. This is not much for the last filter (i.e. LIKE "%scelerisque%"), even if index is NOT used for this purpose at all.
As you can see the speed much depends on indexes selected for a query. So, in this particular case the PRIMARY KEY index is much more useful than fulltext title index, because PRIMARY KEY has bigger impact for result reduction than title.
MySQL is not always clever to set the right indexes. We can do this manually, by using clauses like IGNORE INDEX (index_name), FORCE INDEX (index_name), etc.
But in your case the problem is that if we use MATCH() AGAINST() in a query then the fulltext index is required, because MATCH() AGAINST() doesn't work without fulltext index at all. So this is the main reason why MySQL has chosen incorrect indexes for the query.
UPDATE
OK, I did some investigation.
Firstly, you may try to force MySQL to use guid_one index instead of relationship on table elggentity_relationships: USE INDEX (guid_one).
But for even better performance I think you can try to create one index for the composition of two columns (guid_one, membership). Current index guid_one is very similar, but for 3 columns, not for 2. In this query there are only 2 columns used. In my opinion after index creation MySQL should automatically use the right index. If not, force MySQL to use it.
Note: After index creation don't forget to remove old USE INDEX instruction from your query, because this may prevent query from using the newly created index. :)
I have a simple Message table, with 2 indexes:
mysql> show keys from Message;
+---------+------------+-----------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------+------------+-----------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+
| Message | 0 | PRIMARY | 1 | id | A | 5643295 | NULL | NULL | | BTREE | |
| Message | 1 | timestamp | 1 | startTimestamp | A | 5643295 | NULL | NULL | | BTREE | |
+---------+------------+-----------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+
When issuing an order by query, a very large number of rows is examined:
mysql> explain SELECT * from Message ORDER BY startTimestamp LIMIT 0,20;
+----+-------------+---------+-------+---------------+-----------+---------+------+---------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+-------+---------------+-----------+---------+------+---------+-------+
| 1 | SIMPLE | Message | index | NULL | timestamp | 8 | NULL | 5643592 | |
+----+-------------+---------+-------+---------------+-----------+---------+------+---------+-------+
The total row count is:
mysql> select count(*) from Message;
+----------+
| count(*) |
+----------+
| 5837363 |
+----------+
This query touches 96.7% of the rows. The index is BTREE, so as far as I know it should simply yank out the top 20 rows and return them. As it stands, it's using an index to access almost all of the table's rows, which is presumably slower than a full table scan.
Am I mistaken in my assumption that it should simply pick the top 20 rows using the index and return them?
The MySQL server version is 5.0.45 and the table type is InnoDB.
in the EXPLAIN, MySQL estimates the number of rows scanned to 5643592 but it's using the right index(order by column:timestamp) and is limited to 20 rows, so don't worry it's doing the right thing and will stop as soon as 20 rows are sent. Can you give us the query execution time?
With EXPLAIN SELECT ... ORDER BY, you
can check whether MySQL can use
indexes to resolve the query. It
cannot if you see Using filesort in
the Extra column.
It is using the index..