Trouble with subquery but only in view - mysql

I have some trouble with the following query:
DROP VIEW IF EXISTS own_inst_stakes_detail_all;
CREATE VIEW `own_inst_stakes_detail_all` AS
SELECT
`own_inst_stakes_detail`.`FactSet_Entity_ID` AS `factset_entity_id`,
`own_inst_stakes_detail`.`FSYM_ID` AS `fsym_id`,
`own_inst_stakes_detail`.`as_of_date` AS `report_date`,
`own_inst_stakes_detail`.`Position` AS `adj_holding`,
`own_inst_stakes_detail`.`Position` / (SELECT own_sec_prices.adj_shares_outstanding FROM own_sec_prices WHERE own_sec_prices.`FSYM_ID` = own_inst_stakes_detail.`FSYM_ID` ORDER BY ABS(DATEDIFF(own_sec_prices.price_date, `own_inst_stakes_detail`.`as_of_date`)) DESC LIMIT 1) AS `adj_ratio`,
`own_inst_stakes_detail`.`Position` AS `reported_holding`,
`sym_coverage`.`Proper_Name` AS `security_proper_name`,
`sym_entity`.`Entity_Proper_Name` AS `entity_proper_name`
FROM
`own_inst_stakes_detail`
LEFT JOIN `sym_entity` ON `sym_entity`.`FactSet_Entity_ID` = `own_inst_stakes_detail`.`FactSet_Entity_ID`
LEFT JOIN `sym_coverage` ON `sym_coverage`.`FSYM_ID` = `own_inst_stakes_detail`.`FSYM_ID`;
It uses the following tables, which are of particular interest:
CREATE TABLE `own_inst_stakes_detail` (
`FactSet_Entity_ID` CHAR(8) NOT NULL,
`FSYM_ID` CHAR(8) NOT NULL,
`as_of_date` DATE NOT NULL,
`Position` DOUBLE DEFAULT NULL,
PRIMARY KEY (`FactSet_Entity_ID`,`FSYM_ID`,`as_of_date`),
KEY `idx_own_inst_stakes_detail_FactSet_Entity_ID` (`FactSet_Entity_ID`),
KEY `idx_own_inst_stakes_detail_FSYM_ID` (`FSYM_ID`),
KEY `idx_own_inst_stakes_detail_as_of_date` (`as_of_date`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
CREATE TABLE `own_sec_prices` (
`FSYM_ID` CHAR(8) NOT NULL,
`price_date` DATE NOT NULL,
`unadj_shares_outstanding` DOUBLE NOT NULL,
`adj_shares_outstanding` DOUBLE NOT NULL,
PRIMARY KEY (`FSYM_ID`,`price_date`),
KEY `idx_own_sec_prices_FSYM_ID` (`FSYM_ID`),
KEY `idx_own_sec_prices_price_date` (`price_date`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
The query in the view works really well running just the query but it is a catastrophe if that is wrapped in a view. The query never completes, probably because of a full table scan. The tables are really big which I am using here.
I have tried that with MySQL Server 5.7 (btw., with MariaDB 10.1 there are no problems). Is there a way to improve that?
The EXPLAINs look as follows (Maria and My):
MariaDB [factset]> explain SELECT * FROM own_inst_stakes_detail_all WHERE FSYM_ID = 'WK13LJ-S' ORDER BY report_date DESC, adj_ratio DESC;
+------+--------------------+------------------------+--------+------------------------------------+------------------------------------+---------+--------------------------------------------------+------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+--------------------+------------------------+--------+------------------------------------+------------------------------------+---------+--------------------------------------------------+------+----------------------------------------------------+
| 1 | PRIMARY | own_inst_stakes_detail | ref | idx_own_inst_stakes_detail_FSYM_ID | idx_own_inst_stakes_detail_FSYM_ID | 8 | const | 8 | Using index condition; Using where; Using filesort |
| 1 | PRIMARY | sym_entity | eq_ref | PRIMARY | PRIMARY | 8 | factset.own_inst_stakes_detail.FactSet_Entity_ID | 1 | |
| 1 | PRIMARY | sym_coverage | const | PRIMARY | PRIMARY | 8 | const | 1 | Using where |
| 3 | DEPENDENT SUBQUERY | own_sec_prices | ref | PRIMARY,idx_own_sec_prices_FSYM_ID | idx_own_sec_prices_FSYM_ID | 8 | factset.own_inst_stakes_detail.FSYM_ID | 10 | Using temporary; Using filesort |
+------+--------------------+------------------------+--------+------------------------------------+------------------------------------+---------+--------------------------------------------------+------+----------------------------------------------------+
4 rows in set (0.01 sec)
mysql> explain SELECT * FROM own_inst_stakes_detail_all WHERE FSYM_ID = 'WK13LJ-S' ORDER BY report_date DESC, adj_ratio DESC;
+----+--------------------+------------------------+------------+--------+------------------------------------+----------------------------+---------+--------------------------------------------------+--------+----------+---------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------------+------------------------+------------+--------+------------------------------------+----------------------------+---------+--------------------------------------------------+--------+----------+---------------------------------+
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 8 | const | 10 | 100.00 | Using where; Using filesort |
| 2 | DERIVED | own_inst_stakes_detail | NULL | ALL | NULL | NULL | NULL | NULL | 817961 | 100.00 | NULL |
| 2 | DERIVED | sym_entity | NULL | eq_ref | PRIMARY | PRIMARY | 8 | factset.own_inst_stakes_detail.FactSet_Entity_ID | 1 | 100.00 | NULL |
| 2 | DERIVED | sym_coverage | NULL | eq_ref | PRIMARY | PRIMARY | 8 | factset.own_inst_stakes_detail.FSYM_ID | 1 | 100.00 | NULL |
| 3 | DEPENDENT SUBQUERY | own_sec_prices | NULL | ref | PRIMARY,idx_own_sec_prices_FSYM_ID | idx_own_sec_prices_FSYM_ID | 8 | factset.own_inst_stakes_detail.FSYM_ID | 10 | 100.00 | Using temporary; Using filesort |
+----+--------------------+------------------------+------------+--------+------------------------------------+----------------------------+---------+--------------------------------------------------+--------+----------+---------------------------------+
5 rows in set, 3 warnings (0.08 sec)
To me it looks like MySQL cannot use the index idx_own_inst_stakes_detail_FSYM_ID. But if I change the column with the subquery to "1 as adj_ratio", the following happens:
mysql> explain SELECT * FROM own_inst_stakes_detail_all WHERE FSYM_ID = 'WK13LJ-S';
+----+-------------+------------------------+------------+--------+------------------------------------+------------------------------------+---------+--------------------------------------------------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------------------+------------+--------+------------------------------------+------------------------------------+---------+--------------------------------------------------+------+----------+-------+
| 1 | SIMPLE | own_inst_stakes_detail | NULL | ref | idx_own_inst_stakes_detail_FSYM_ID | idx_own_inst_stakes_detail_FSYM_ID | 8 | const | 8 | 100.00 | NULL |
| 1 | SIMPLE | sym_entity | NULL | eq_ref | PRIMARY | PRIMARY | 8 | factset.own_inst_stakes_detail.FactSet_Entity_ID | 1 | 100.00 | NULL |
| 1 | SIMPLE | sym_coverage | NULL | const | PRIMARY | PRIMARY | 8 | const | 1 | 100.00 | NULL |
+----+-------------+------------------------+------------+--------+------------------------------------+------------------------------------+---------+--------------------------------------------------+------+----------+-------+
3 rows in set, 1 warning (0.00 sec)
That works well! Any help is greatly appreciated. Thanks!

Related

MySQL: Out of sort memory, consider increasing server sort buffer size

I cant find explanation of MySQL behavior. Examples are below.
Fields name and name_full are text type and field price_steps is json type. None of them has index.
SELECT
name,
name_full,
price_steps
FROM
`lots`
WHERE
EXISTS (
SELECT
*
FROM
`categories`
INNER JOIN `category_lot` ON `categories`.`id` = `category_lot`.`category_id`
WHERE
`lots`.`id` = `category_lot`.`lot_id`
AND `category_id` IN (25)
)
ORDER BY
`created_at` DESC
LIMIT 31 OFFSET 0
MySQL throws error: [Err] 1038 - Out of sort memory, consider increasing server sort buffer size.
Ok, let it be.
Then I added extra field into select part.
SUBSTRING(`name_full`, 1, 200000000000) as name_full2
and query runs successfully (Why? Extra field should lead to extra memory allocation, isnt it?).
Then I decided to make query heavier and replace string
AND `category_id` IN (25)
with
AND `category_id` IN (1,2,3,4,5,6,7,8,9,10, 25)
and the query also finishes successfully.
Count of rows with category = 25 is only about 250, but with categories in (1,2,3,4,5,6,7,8,9,10, 25) is about 40000 rows. This must lead to extra memory demands, but mysql doesnt throw error. Why?
Any explanation to this paradox? Thanks in advance!
UPDATE1
EXPLAIN with failing query
mysql> EXPLAIN SELECT name, name_full, price_steps FROM `lots` WHERE EXISTS ( SELECT * FROM `categories` INNER JOIN `category_lot` ON `categories`.`id` = `category_lot`.`category_id` WHERE `lots`.`id` = `category_lot`.`lot_id` AND `category_id` IN(25) ) ORDER BY `created_at` DESC;
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
| 1 | SIMPLE | categories | NULL | const | PRIMARY | PRIMARY | 8 | const | 1 | 100.00 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | category_lot | NULL | ref | category_lot_lot_id_foreign,category_lot_category_id_foreign | category_lot_category_id_foreign | 8 | const | 1099 | 100.00 | Start temporary |
| 1 | SIMPLE | lots | NULL | eq_ref | PRIMARY | PRIMARY | 8 | torgs.category_lot.lot_id | 1 | 100.00 | End temporary |
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
3 rows in set, 2 warnings (0.00 sec)
EXPLAIN with success query (added 4th field)
mysql> EXPLAIN SELECT name, name_full, price_steps,SUBSTRING(`name_full`, 1, 200000000000) as name_full2 FROM `lots` WHERE EXISTS ( SELECT * FROM `categories` INNER JOIN `category_lot` ON `categories`.`id` = `category_lot`.`category_id` WHERE `lots`.`id` = `category_lot`.`lot_id` AND `category_id` IN(25) ) ORDER BY `created_at` DESC;
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
| 1 | SIMPLE | categories | NULL | const | PRIMARY | PRIMARY | 8 | const | 1 | 100.00 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | category_lot | NULL | ref | category_lot_lot_id_foreign,category_lot_category_id_foreign | category_lot_category_id_foreign | 8 | const | 1099 | 100.00 | Start temporary |
| 1 | SIMPLE | lots | NULL | eq_ref | PRIMARY | PRIMARY | 8 | torgs.category_lot.lot_id | 1 | 100.00 | End temporary |
+----+-------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------------+------+----------+----------------------------------------------+
3 rows in set, 2 warnings (0.00 sec)
EXPLAIN with success query (added 4th field and category_id in (1,2,3,4,5,6,7,8,9,10,25))
mysql> EXPLAIN SELECT name, name_full, price_steps, SUBSTRING(`name_full`, 1, 200000000000) as name_full2 FROM `lots` WHERE EXISTS ( SELECT * FROM `categories` INNER JOIN `category_lot` ON `categories`.`id` = `category_lot`.`category_id` WHERE `lots`.`id` = `category_lot`.`lot_id` AND `category_id` IN(1,2,3,4,5,6,7,8,9,10,25) ) ORDER BY `created_at` DESC;
+----+--------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------+------+----------+---------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------+------+----------+---------------------------------+
| 1 | SIMPLE | <subquery2> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | lots | NULL | eq_ref | PRIMARY | PRIMARY | 8 | <subquery2>.lot_id | 1 | 100.00 | NULL |
| 2 | MATERIALIZED | categories | NULL | range | PRIMARY | PRIMARY | 8 | NULL | 11 | 100.00 | Using where; Using index |
| 2 | MATERIALIZED | category_lot | NULL | ref | category_lot_lot_id_foreign,category_lot_category_id_foreign | category_lot_category_id_foreign | 8 | torgs.categories.id | 1883 | 100.00 | NULL |
+----+--------------+--------------+------------+--------+--------------------------------------------------------------+----------------------------------+---------+---------------------+------+----------+---------------------------------+
4 rows in set, 2 warnings (0.00 sec)
UPDATE2
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Note | 1276 | Field or reference 'torgs.lots.id' of SELECT #2 was resolved in SELECT #1 |
| Note | 1003 | /* select#1 */ select `torgs`.`lots`.`name` AS `name`,`torgs`.`lots`.`name_full` AS `name_full`,`torgs`.`lots`.`price_steps` AS `price_steps`,substr(`torgs`.`lots`.`name_full`,1,200000000000) AS `name_full2` from `torgs`.`lots` semi join (`torgs`.`categories` join `torgs`.`category_lot`) where ((`torgs`.`category_lot`.`category_id` = `torgs`.`categories`.`id`) and (`torgs`.`lots`.`id` = `<subquery2>`.`lot_id`) and (`torgs`.`categories`.`id` in (1,2,3,4,5,6,7,8,9,10,25))) order by `torgs`.`lots`.`created_at` desc |
+-------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Query uses temporary table without force index

Query
SELECT SQL_NO_CACHE contacts.id,
contacts.date_modified contacts__date_modified
FROM contacts
INNER JOIN
(SELECT tst.team_set_id
FROM team_sets_teams tst
INNER JOIN team_memberships team_membershipscontacts ON (team_membershipscontacts.team_id = tst.team_id)
AND (team_membershipscontacts.user_id = '5daa2e92-c347-11e9-afc5-525400a80916')
AND (team_membershipscontacts.deleted = 0)
GROUP BY tst.team_set_id) contacts_tf ON contacts_tf.team_set_id = contacts.team_set_id
LEFT JOIN contacts_cstm contacts_cstm ON contacts_cstm.id_c = contacts.id
WHERE contacts.deleted = 0
ORDER BY contacts.date_modified DESC,
contacts.id DESC
LIMIT 21;
Takes extremely long (2 minutes on 2M records). I cant change this query, since it is system generated
This is it's explain:
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| 1 | PRIMARY | contacts | NULL | ref | idx_contacts_tmst_id,idx_del_date_modified,idx_contacts_del_last,idx_cont_del_reports,idx_del_id_user | idx_del_date_modified | 2 | const | 1113718 | 100.00 | Using temporary; Using filesort |
| 1 | PRIMARY | <derived3> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 50.00 | Using where; Using join buffer (Block Nested Loop) |
| 1 | PRIMARY | contacts_cstm | NULL | eq_ref | PRIMARY | PRIMARY | 144 | sugarcrm.contacts.id | 1 | 100.00 | Using index |
| 3 | DERIVED | team_membershipscontacts | NULL | ref | idx_team_membership,idx_teammemb_team_user,idx_del_team_user | idx_team_membership | 145 | const | 2 | 99.36 | Using index condition; Using where; Using temporary; Using filesort |
| 3 | DERIVED | tst | NULL | ref | idx_ud_set_id,idx_ud_team_id,idx_ud_team_set_id,idx_ud_team_id_team_set_id | idx_ud_team_id_team_set_id | 144 | sugarcrm.team_membershipscontacts.team_id | 1 | 100.00 | Using index |
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
But when I use force index(idx_del_date_modified) (which is the same index used in explain), the query takes just 0.01s and I get slightly different explain.
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| 1 | PRIMARY | contacts | NULL | ref | idx_del_date_modified | idx_del_date_modified | 2 | const | 1113718 | 100.00 | Using where |
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 50.00 | Using where |
| 1 | PRIMARY | contacts_cstm | NULL | eq_ref | PRIMARY | PRIMARY | 144 | sugarcrm.contacts.id | 1 | 100.00 | Using index |
| 2 | DERIVED | team_membershipscontacts | NULL | ref | idx_team_membership,idx_teammemb_team_user,idx_del_team_user | idx_team_membership | 145 | const | 2 | 99.36 | Using index condition; Using where; Using temporary; Using filesort |
| 2 | DERIVED | tst | NULL | ref | idx_ud_set_id,idx_ud_team_id,idx_ud_team_set_id,idx_ud_team_id_team_set_id | idx_ud_team_id_team_set_id | 144 | sugarcrm.team_membershipscontacts.team_id | 1 | 100.00 | Using index |
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
The first query uses temporary table and file sort, but the query with force index uses just where. Shouldn't the queries be the same? Why is the query with force index so much faster - used index is still the same.
According to MySQL manual:
Temporary tables can be created under conditions such as these:
If there is an ORDER BY clause and a different GROUP BY clause, or if
the ORDER BY or GROUP BY contains columns from tables other than the
first table in the join queue, a temporary table is created.
DISTINCT combined with ORDER BY may require a temporary table.
If you use the SQL_SMALL_RESULT option, MySQL uses an in-memory
temporary table, unless the query also contains elements (described
later) that require on-disk storage.
Likely, you have better performance because in MySQL there is the query optimizer component.
If you create index the query optimizer could not use the index column even though the index exists.
Using force index(..) you are forcing MySql to use index, instead.
Please consider a detailed example here.

TPCH Query Optimization

The following query is taking 5 hours so far to run:
INSERT $LINEITEM_PUBLIC SELECT *
FROM LINEITEM
WHERE L_PARTKEY IN ( SELECT P_PARTKEY FROM $PART_PUBLIC )
AND L_SUPPKEY IN ( SELECT S_SUPPKEY FROM $SUPPLIER_PUBLIC )
AND L_ORDERKEY IN ( SELECT O_ORDERKEY FROM $ORDERS_PUBLIC );
I added all required indexes but nothing seems to be helping. The Query Explain Plan prints the following:
+----+-------------+------------------+------------+--------+--------------------------------+-------------+---------+--------------------------------+----------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------------+------------+--------+--------------------------------+-------------+---------+--------------------------------+----------+----------+-------------+
| 1 | INSERT | $LINEITEM_PUBLIC | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | NULL |
| 1 | SIMPLE | $ORDERS_PUBLIC | NULL | index | PRIMARY | O_ORDERDATE | 3 | NULL | 12826617 | 100.00 | Using index |
| 1 | SIMPLE | LINEITEM | NULL | ref | PRIMARY,LINEITEM_FK2,L_SUPPKEY | PRIMARY | 4 | TPCH.$ORDERS_PUBLIC.O_ORDERKEY | 3 | 100.00 | NULL |
| 1 | SIMPLE | $SUPPLIER_PUBLIC | NULL | eq_ref | PRIMARY | PRIMARY | 4 | TPCH.LINEITEM.L_SUPPKEY | 1 | 100.00 | Using index |
| 1 | SIMPLE | $PART_PUBLIC | NULL | eq_ref | PRIMARY | PRIMARY | 4 | TPCH.LINEITEM.L_PARTKEY | 1 | 100.00 | Using index |
+----+-------------+------------------+------------+--------+--------------------------------+-------------+---------+--------------------------------+----------+----------+-------------+
Any recommendations on how this query can be optimized?
Update:
The size of the tables in the previous query is as follows:
LINEITEM: 60M records
$ORDERS_PUBLIC: 13M records
$SUPPLIER_PUBLIC: 92K records
$PART_PUBLIC: 2M records
Make sure there is an index starting with O_ORDERKEY.
IN (SELECT ...) may be optimized poorly (depending on version); try this:
INSERT $LINEITEM_PUBLIC
SELECT l.*
FROM LINEITEM AS l
WHERE EXISTS( SELECT * FROM $PART_PUBLIC WHERE P_PARTKEY = L_PARTKEY )
AND EXISTS( SELECT * FROM $SUPPLIER_PUBLIC WHERE S_SUPPKEY = L_SUPPKEY )
AND EXISTS( SELECT * FROM $ORDERS_PUBLIC WHERE O_ORDERKEY = L_ORDERKEY );

Mysql. JOIN to small table blocks using indexes on the main one

MySQL version 5.7.17
When I use JOIN for small tables, MySQL don't allow me to use index on the main table.
Compare:
EXPLAIN SELECT `ko_base_client_orders`.*
FROM `ko_base_client_orders`
LEFT JOIN `ko_base_new_perspe` AS `ko_of`
ON (`ko_of`.`id` = `ko_base_client_orders`.`office`)
ORDER BY `order_time` DESC
LIMIT 10 OFFSET 0;
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
| 1 | SIMPLE | ko_base_client_orders | NULL | ALL | NULL | NULL | NULL | NULL | 60737 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | ko_of | NULL | index | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.00 | Using where; Using index; Using join buffer (Block Nested Loop) |
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
2 rows in set, 1 warning (0.00 sec)
and the same query with index hint:
EXPLAIN SELECT `ko_base_client_orders`.*
FROM `ko_base_client_orders`
LEFT JOIN `ko_base_new_perspe` AS `ko_of`
FORCE INDEX FOR JOIN (`PRIMARY`)
ON (`ko_of`.`id` = `ko_base_client_orders`.`office`)
ORDER BY `order_time` DESC
LIMIT 10 OFFSET 0;
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
| 1 | SIMPLE | ko_base_client_orders | NULL | index | NULL | order_time | 6 | NULL | 10 | 100.00 | NULL |
| 1 | SIMPLE | ko_of | NULL | eq_ref | PRIMARY | PRIMARY | 4 | e3.ko_base_client_orders.office | 1 | 100.00 | Using index |
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
In second case MySQL uses key for order_time, but in first example it ignores index and scans the entire table.
I can not predict which table would be small, and which would not (it differs from project to project), so I don't want to use index hints all the time. Is there another solution for this problem?

MySQL index slowing down query

MySQL Server version: 5.0.95
Tables All: InnoDB
I am having an issue with a MySQL db query. Basically I am finding that if I index a particular varchar(50) field tag.name, my queries take longer (x10) than not indexing the field. I am trying to speed this query up, however my efforts seem to be counter productive.
The culprit line and field seems to be:
WHERE `t`.`name` IN ('news','home')
I have noticed that if i query the tag table directly without a join using the same criteria and with the name field indexed, i do not have the issue.. It actually works faster as expected.
EXAMPLE Query **
SELECT `a`.*, `u`.`pen_name`
FROM `tag_link` `tl`
INNER JOIN `tag` `t`
ON `t`.`tag_id` = `tl`.`tag_id`
INNER JOIN `article` `a`
ON `a`.`article_id` = `tl`.`link_id`
INNER JOIN `user` `u`
ON `a`.`user_id` = `u`.`user_id`
WHERE `t`.`name` IN ('news','home')
AND `tl`.`type` = 'article'
AND `a`.`featured` = 'featured'
GROUP BY `article_id`
LIMIT 0 , 5
EXPLAIN with index **
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------+---------+---------+-------------------+------+-----------------------------------------------------------+
| 1 | SIMPLE | t | range | PRIMARY,name | name | 152 | NULL | 4 | Using where; Using index; Using temporary; Using filesort |
| 1 | SIMPLE | tl | ref | tag_id,link_id,link_id_2 | tag_id | 4 | portal.t.tag_id | 10 | Using where |
| 1 | SIMPLE | a | eq_ref | PRIMARY,fk_article_user1 | PRIMARY | 4 | portal.tl.link_id | 1 | Using where |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | portal.a.user_id | 1 | |
+----+-------------+-------+--------+--------------------------+---------+---------+-------------------+------+-----------------------------------------------------------+
EXPLAIN without index **
+----+-------------+-------+--------+--------------------------+---------+---------+---------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+--------------------------+---------+---------+---------------------+------+-------------+
| 1 | SIMPLE | a | index | PRIMARY,fk_article_user1 | PRIMARY | 4 | NULL | 8742 | Using where |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | portal.a.user_id | 1 | |
| 1 | SIMPLE | tl | ref | tag_id,link_id,link_id_2 | link_id | 4 | portal.a.article_id | 3 | Using where |
| 1 | SIMPLE | t | eq_ref | PRIMARY | PRIMARY | 4 | portal.tl.tag_id | 1 | Using where |
+----+-------------+-------+--------+--------------------------+---------+---------+---------------------+------+-------------+
TABLE CREATE
CREATE TABLE `tag` (
`tag_id` int(11) NOT NULL auto_increment,
`name` varchar(50) NOT NULL,
`type` enum('layout','image') NOT NULL,
`create_dttm` datetime default NULL,
PRIMARY KEY (`tag_id`)
) ENGINE=InnoDB AUTO_INCREMENT=43077 DEFAULT CHARSET=utf8
INDEXS
SHOW INDEX FROM tag_link;
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| tag_link | 0 | PRIMARY | 1 | tag_link_id | A | 42023 | NULL | NULL | | BTREE | |
| tag_link | 1 | tag_id | 1 | tag_id | A | 10505 | NULL | NULL | | BTREE | |
| tag_link | 1 | link_id | 1 | link_id | A | 14007 | NULL | NULL | | BTREE | |
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
SHOW INDEX FROM article;
+---------+------------+------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+---------+------------+------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| article | 0 | PRIMARY | 1 | article_id | A | 5723 | NULL | NULL | | BTREE | |
| article | 1 | fk_article_user1 | 1 | user_id | A | 1 | NULL | NULL | | BTREE | |
| article | 1 | create_dttm | 1 | create_dttm | A | 5723 | NULL | NULL | YES | BTREE | |
+---------+------------+------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
Final Solution
It seems that MySQL is just sorted the data incorrectly. In the end it turned out faster to look at the tag table as a sub query returning the ids.
It seems that article_id is the primary key for the article table.
Since you're grouping by article_id, MySQL needs to return the records in order by that column, in order to perform the GROUP BY.
You can see that without the index, it scans all records in the article table, but they're at least in order by article_id, so no later sort is required. The LIMIT optimization can be applied here, since it's already in order, it can just stop after it gets five rows.
In the query with the index on tag.name, instead of scanning the entire articles table, it utilizes the index, but against the tag table, and starts there. Unfortunately, when doing this, the records must later be sorted by article.article_id in order to complete the GROUP BY clause. The LIMIT optimization can't be applied since it must return the entire result set, then order it, in order to get the first 5 rows.
In this case, MySQL just guesses wrongly.
Without the LIMIT clause, I'm guessing that using the index is faster, which is maybe what MySQL was guessing.
How big are your tables?
I noticed in the first explain you have a "Using temporary; Using filesort" which is bad. Your query is likely being dumped to disc which makes it way slower than in memory queries.
Also try to avoid using "select *" and instead query the minimum fields needed.