I have the following MYSQL query:
SELECT statusdate,consigneenamee,weight,productcode,
pieces,statustime,statuscode,statusdesc,generatingiata,
shipmentdate,shipmenttime,consigneeairportcode,signatory
FROM notes
where (shipperref='180908184' OR shipperref='180908184'
OR shipperref='180908184' OR shipperref='180908184 '
OR shipperref like 'P_L_%180908184')
order by edicheckpointdate asc, edicheckpointtime asc;
I added an index to speed up this query using the MYSQL Workbench but when executing the EXPLAIN command, it still does not show the key and shows as NULL:
+----+-------------+---------------+------+---------------+------+---------+------+---------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+------+---------------+------+---------+------+---------+-----------------------------+
| 1 | SIMPLE | dhltracking_2 | ALL | index2 | NULL | NULL | NULL | 3920874 | Using where; Using filesort |
+----+-------------+---------------+------+---------------+------+---------+------+---------+-----------------------------+
Any reason why this is happening and how I can speed up this query?
My Index:
You have LIKE statement in your query and I think that your index spans on more than 20-30% of table rows (or more..) and that's why MySQL can ignore it for performance reasons.
My proposal:
Add FORCE INDEX as #Solarflare proposes
Use FULLTEXT index (works on CHAR and VARCHAR also) and use MATCH ... AGAINST search (https://dev.mysql.com/doc/refman/8.0/en/fulltext-boolean.html)
Related
I have a query in Ruby on Rails application with a strange unary condition in where:
SELECT * FROM messages WHERE (active) ORDER BY id DESC;
I didn't even know that such conditions are allowed and can't find documentation describing this syntax anywhere. Experiments show that this is some kind of an equivalent to
SELECT * FROM messages WHERE active!=0 ORDER BY id DESC;
The problem is that Mysql uses index for the second variany only:
mysql> explain SELECT * FROM messages WHERE (active) ORDER BY id DESC;
+----+-------------+----------+------+---------------+------+---------+------+--------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+------+---------------+------+---------+------+--------+-----------------------------+
| 1 | SIMPLE | messages | ALL | NULL | NULL | NULL | NULL | 560646 | Using where; Using filesort |
+----+-------------+----------+------+---------------+------+---------+------+--------+-----------------------------+
mysql> explain SELECT * FROM messages WHERE active!=0 ORDER BY id DESC;
+----+-------------+----------+-------+------------------+--------+---------+------+------+---------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+-------+------------------+--------+---------+------+------+---------------------------------------+
| 1 | SIMPLE | messages | range | active_id,active | active | 2 | NULL | 1394 | Using index condition; Using filesort |
+----+-------------+----------+-------+------------------+--------+---------+------+------+---------------------------------------+
I can't change the query text since, as it was explained to me, the application generates queries on the fly and they are not stored anywhere. So my questions are:
Do I understand the meaning of this unary clause correctly?
Why such queries don't use indices?
Is it possible to make Mysql to use an index on this one without changing the query text?
You are right, both clauses should have the same result (assuming an int type).
The server code doesn't appear to recognise the equivalence. Testing on MySQL-8.0 and the same query plan exists. MariaDB-10.2, and 10.3 appear to both use an index for both cases.
No.
If the range of values in active is 0 or 1; A SELECT * FROM messages WHERE active=1 ORDER BY id DESC query will be able to use the index for ordering (hence no filesort), if id is a primary key;
I am running this query
EXPLAIN SELECT id, timestamp from foo where id IN (23,67,78,90) order by ASC
here id is indexed. But then too when I am running Explain I am getting this in Using where;Using Index in Extra
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | foo | range | PRIMARY | PRIMARY | 8 | NULL | 12 | Using where;Using Index|
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
But when I am running this same query with single id nothing is in Extra its working as expected in the case of index
EXPLAIN SELECT id, timestamp from foo where id = 23`
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
| id | select_type | table| type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
| 1 | SIMPLE | foo | range | PRIMARY | PRIMARY | 8 | NULL | 1 | |
+----+-------------+---------------+-------+---------------+---------+---------+------+------+-------------+
I think something wrong with IN. Can anyone tell me the way to optimize it ?
As per my knowledge,
IN query will take more time than Single "=". If "IN" have single value then it is equal to single "=" query.
Because it is MULTIPLE OR conditions of "=" OR "=".
id will match with every 4 values in "IN" array.
Only way to optimise is to have index.
Update:
If the Extra column also says Using where, it means the index is being used to perform lookups of key values. Without Using where, the optimizer may be reading the index to avoid reading data rows but not using it for lookups. For example, if the index is a covering index for the query, the optimizer may scan it without using it for lookups. See explanation
Please, help me!
How to optimize a query like:
SELECT idu
FROM `user`
WHERE `username`!='manager'
AND `username`!='user1#yahoo.com'
ORDER BY lastdate DESC
This is the explain:
explain SELECT idu FROM `user` WHERE `username`!='manager' AND `username`!='ser1#yahoo.com' order by lastdate DESC;
+----+-------------+-------+------+----------------------------+------+---------+------+--------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+----------------------------+------+---------+------+--------+-----------------------------+
| 1 | SIMPLE | user | ALL | username,username-lastdate | NULL | NULL | NULL | 208478 | Using where; Using filesort |
+----+-------------+-------+------+----------------------------+------+---------+------+--------+-----------------------------+
1 row in set (0.00 sec)
To avoid file sorting in a big database.
Since this query is just scanning all rows, you need an index on lastdate to avoid MySQL from having to order the results manually (using filesort, which isn't always to disk/temp table).
For super read performance, add the following multi-column "covering" index:
user(lastdate, username, idu)
A "covering" index would allow MySQL to just scan the index instead of the actual table data.
If using InnoDB and any of the above columns are your primary key, you don't need it in the index.
Please consider following schema
CREATE table articles (
id Int UNSIGNED NOT NULL AUTO_INCREMENT,
cat_id Int UNSIGNED NOT NULL,
status Int UNSIGNED NOT NULL,
date_added Datetime,
Primary Key (id)) ENGINE = InnoDB;
CREATE INDEX cat_list_INX ON articles (cat_id, status, date_added);
CREATE INDEX categories_list_INX ON articles (cat_id, status);
I have written following two queries which can be completely satisfied by the above two indicies but MySQL is putting where in extra column.
mysql> EXPLAIN SELECT cat_id FROM articles USE INDEX (cat_list_INX) WHERE cat_id=158 AND status=2 ORDER BY date_added DESC LIMIT 500, 5;
+----+-------------+----------+------+---------------+--------------+---------+-------------+-------+--------------------------+
| id | select_type | table ref | | type | possible_keys | key | key_len | rows | Extra |
+----+-------------+----------+------+---------------+--------------+---------+-------------+-------+--------------------------+
| 1 | SIMPLE | articles | ref | cat_list_INX | cat_list_INX | 5 | const,const | 50698 | Using where; Using index |
+----+-------------+----------+------+---------------+--------------+---------+-------------+-------+--------------------------+
mysql> EXPLAIN SELECT cat_id FROM articles USE INDEX (categories_list_INX) WHERE cat_id=158 AND status=2;
+----+-------------+----------+------+---------------------+---------------------+---------+-------------+-------+--------------------------+
| id | select_type | tab key |le | type | possible_keys | key_len | ref | rows | Extra |
+----+-------------+----------+------+---------------------+---------------------+---------+-------------+-------+--------------------------+
| 1 | SIMPLE | articles | ref | categories_list_INX | categories_list_INX | 5 | const,const | 52710 | Using where; Using index |
+----+-------------+----------+------+---------------------+---------------------+---------+-------------+-------+--------------------------+
As far as I know where requires an additional disk seek. Why it's not just using index?
The first query is filtering records at the mysql level outside of the storage engine because of your "ORDER BY" clause using date_added field.
This can be mitigated by moving the date_added field first in the index like this
CREATE INDEX cat_list_INX ON articles (date_added, cat_id, status);
The 2nd query - my version of mysql is not showing a "Using where" - I would not expect to either - maybe its because I have no records.
mysql> EXPLAIN SELECT cat_id FROM articles USE INDEX (categories_list_INX) WHERE cat_id=158 AND status=2;
+----+-------------+----------+------+---------------------+---------------------+---------+-------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+------+---------------------+---------------------+---------+-------------+------+-------------+
| 1 | SIMPLE | articles | ref | categories_list_INX | categories_list_INX | 8 | const,const | 1 | Using index |
+----+-------------+----------+------+---------------------+---------------------+---------+-------------+------+-------------+
1 row in set (0.00 sec)
Extra column info from High Performance MySQL:
Using Index: This indicates that MySQL will use a covering index to avoid accessing the table. Don't confuse covering indexes with the index access type.
Using Where: This means the MySQL server will post-filter rows after the storage engine retrieves them. Many WHERE conditions that involve columns in an index can be checked by the storage engine when (and if) it reads the index, so not all queries with a WHERE clause will show "Using where". Sometimes the presence of "Using where" is a hint that the query can benefit from different indexing.
I'm wondering if MySQL takes collation into account when generating an index, or if the index is generated the same regardless of collation, the collation only being taken into account when later traversing that index.
For my purposes, I'd like to use the collation utf8_unicode_ci on a field. I know this particular collation has a relatively high performance penalty, but it's still important to me to use it.
I have an index on that field which is being used to satisfy an ORDER BY clause, retrieving the rows in order quickly (avoiding a filesort). However, I'm not sure whether using this collation is going to affect the speed of rows as they are read back from the index, or if the index stores data in an already-normalised state according to that collation, allowing for the performance penalty to be entirely in generating the index and not reading it back.
I believe that the btree structure will be different because it has to compare the column values differently.
Look at these two query plans:
mysql> explain select * from sometable where keycol = '3';
+----+-------------+-------+------+---------------+---------+---------+-------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+---------------+---------+---------+-------+------+--------------------------+
| 1 | SIMPLE | pro | ref | PRIMARY | PRIMARY | 66 | const | 34 | Using where; Using index |
+----+-------------+-------+------+---------------+---------+---------+-------+------+--------------------------+
mysql> explain select * from sometable where binary keycol = '3';
+----+-------------+-------+-------+---------------+---------+---------+------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+-------+--------------------------+
| 1 | SIMPLE | pro | index | NULL | PRIMARY | 132 | NULL | 14417 | Using where; Using index |
+----+-------------+-------+-------+---------------+---------+---------+------+-------+--------------------------+
If we change the collation for the comparison, suddenly it isn't even able to seek the index anymore and has to scan every row. The actual values stored in the index will be the same regardless of collation, for instance, because it will still return the value in its original casing regardless of whether it's using a case sensitive or case insensitive collation.
So lookups against a case insensitive collation should be a little less efficient.
However, I doubt you'd ever be able to notice the difference; note that MySQL makes everything case insensitive by default, so the impact can't be that terrible.
UPDATE:
You can see a similar effect for order by operations:
mysql> explain select * from sometable order by keycol collate latin1_general_cs;
+----+-------------+-------+-------+---------------+---------+---------+------+-------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+-------+-----------------------------+
| 1 | SIMPLE | pro | index | NULL | PRIMARY | 132 | NULL | 14417 | Using index; Using filesort |
+----+-------------+-------+-------+---------------+---------+---------+------+-------+-----------------------------+
mysql> explain select * from sometable order by keycol ;
+----+-------------+-------+-------+---------------+---------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------+---------+---------+------+-------+-------------+
| 1 | SIMPLE | pro | index | NULL | PRIMARY | 132 | NULL | 14417 | Using index |
+----+-------------+-------+-------+---------------+---------+---------+------+-------+-------------+
Note the extra 'filesort' stage required to execute the query. That means mysql is queuing up the result in a temporary buffer and sorting it itself using a quicksort in an extra stage, throwing out whatever the index order was. Using the original collation this step is uneccessary as mysql knows the order from index initially.
MySQL will use the collation of the column for the index. So if you make a utf8_unicode_ci field, then the index will also be in utf8_unicode_ci order effectively.
Keep in mind that using the index will not always 100% bypass the performance impact, but for most practical purposes it will.
Many database systems aren't CPU bound, so I doubt you would notice the impact.