I am trying to optimize a stored procedure. I have identified some issues with it, but I don't know enough to actually correct the problems. One of the subqueries looks like this
select d.districtCode,
b.year Year,
if(bli.releaseAdjustment = 3 and bli.id not in (select billLineItem_id from AppliedDiscount),
bli.amount ,0) *-1 Refund
from Bill b
join BillLineItem bli on bli.bill_id = b.id
left join Bill_District d on d.bill_id = b.id
and bli.type = d.type
left join AppliedDiscount a on a.billLineItem_id = bli.id
where bli.releaseAdjustment in (1,2,3)
and bli.type in (4,6)
group by d.districtCode, b.year
The EXPLAIN outputs this
+----+------------------------+---------------+--------------+--------------------------------+-----------------+---------+---------------------+---------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+------------------------+---------------+--------------+--------------------------------+-----------------+---------+---------------------+---------+----------------------------------------------+
| 1 | PRIMARY | bli | ALL | FKF4236A,type,releaseAdjustment| NULL | NULL | NULL | 2787322 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | b | eq_ref | PRIMARY | PRIMARY | 8 | tax.bli.bill_id | 1 | |
| 1 | PRIMARY | d | ref | bill_id,type | bill_id | 8 | tax.bli.bill_id | 1 | |
| 1 | PRIMARY | a | ref | billLineItem_idx | billLineItem_idx| 8 | tax.bli.id | 1 | Using index |
| 1 | DEPENDENT SUBQUERY |AppliedDiscount|index_subquery| billLineItem_idx | billLineItem_idx| 8 | func | 1 | Using index |
+----+------------------------+---------------+--------------+--------------------------------+-----------------+---------+---------------------+---------+----------------------------------------------+
How would you suggest I fix this? This problem, or one very similar, is found throughout this stored procedure numerous times. AppliedDiscount only consists of 3 columns, all of which are indexed already.
Edit: Removing the group by changes the first row of the explain to
| 1 | PRIMARY | bli | ALL | FKF4236A,type,releaseAdjustment,bill_id| NULL | NULL | NULL | 2613847 | Using where |
That's better and technically answers my question, but that just means that I was asking the wrong question.
The 'type' is still ALL. What can I do to improve that?
This query takes 6+ minutes to run:
SELECT null as primary_id, t211.appln_id as appln_id, t212.pat_publn_id, t212.citn_id, citn_origin, cited_pat_publn_id, cited_appln_id, pat_citn_seq_nr, relevant_claim,
t212.cited_npl_publn_id, npl_citn_seq_nr, citn_gener_auth, npl_biblio, npl_type, publn_nr_original, t215.citn_categ, t215.CITN_REPLENISHED, npl_author,
npl_title1, npl_title2, npl_editor, npl_volume, npl_issue, npl_publn_date, npl_publn_end_date, npl_publisher, npl_page_first, npl_page_last, npl_abstract_nr, npl_doi, npl_isbn, npl_issn, online_availability, online_classification, online_search_date
FROM
tls211_pat_publn as t211
LEFT JOIN tls212_citation as t212 on t211.pat_publn_id = t212.pat_publn_id
LEFT JOIN tls214_npl_publn as t214 on t214.npl_publn_id = t212.cited_npl_publn_id
LEFT JOIN tls215_citn_categ as t215 on t215.pat_publn_id = t212.pat_publn_id and t215.citn_id = t212.citn_id
where t211.pat_publn_id=777;
This query takes 0 seconds to run:
SELECT null as primary_id, t211.appln_id as appln_id, t212.pat_publn_id, t212.citn_id, citn_origin, cited_pat_publn_id, cited_appln_id, pat_citn_seq_nr, relevant_claim,
t212.cited_npl_publn_id, npl_citn_seq_nr, citn_gener_auth, npl_biblio, npl_type, publn_nr_original, t215.citn_categ, t215.CITN_REPLENISHED, npl_author,
npl_title1, npl_title2, npl_editor, npl_volume, npl_issue, npl_publn_date, npl_publn_end_date, npl_publisher, npl_page_first, npl_page_last, npl_abstract_nr, npl_doi, npl_isbn, npl_issn, online_availability, online_classification, online_search_date
FROM
tls211_pat_publn as t211
LEFT JOIN tls212_citation as t212 on t211.pat_publn_id = t212.pat_publn_id
LEFT JOIN tls214_npl_publn as t214 on t214.npl_publn_id = t212.cited_npl_publn_id
LEFT JOIN tls215_citn_categ as t215 on t215.pat_publn_id = t212.pat_publn_id
where t211.pat_publn_id=777;
The only difference is adding: and t215.citn_id = t212.citn_id at the end of the last LEFT JOIN.
Both t212 and t215 have indexes for citn_id.
I've run "explain" on this an absurd number of times. Here's the explain on the slow one:
+----+-------------+-------+------------+--------+-------------------------+-------------+---------+-------------------------------------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+--------+-------------------------+-------------+---------+-------------------------------------------+------+----------+-------------+
| 1 | SIMPLE | t211 | NULL | const | PRIMARY | PRIMARY | 4 | const | 1 | 100.00 | NULL |
| 1 | SIMPLE | t212 | NULL | ref | PRIMARY,ppi | ppi | 4 | const | 7 | 100.00 | NULL |
| 1 | SIMPLE | t214 | NULL | eq_ref | PRIMARY | PRIMARY | 4 | patstat_2019_fall.t212.cited_npl_publn_id | 1 | 100.00 | NULL |
| 1 | SIMPLE | t215 | NULL | ref | PRIMARY,idx_citn_id,ppi | idx_citn_id | 2 | patstat_2019_fall.t212.citn_id | 11 | 100.00 | Using where |
+----+-------------+-------+------------+--------+-------------------------+-------------+---------+-------------------------------------------+------+----------+-------------+
4 rows in set, 1 warning (0.00 sec)
Here's the explain on the fast one:
+----+-------------+-------+------------+--------+---------------+---------+---------+-------------------------------------------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+--------+---------------+---------+---------+-------------------------------------------+------+----------+-------+
| 1 | SIMPLE | t211 | NULL | const | PRIMARY | PRIMARY | 4 | const | 1 | 100.00 | NULL |
| 1 | SIMPLE | t212 | NULL | ref | PRIMARY,ppi | ppi | 4 | const | 7 | 100.00 | NULL |
| 1 | SIMPLE | t214 | NULL | eq_ref | PRIMARY | PRIMARY | 4 | patstat_2019_fall.t212.cited_npl_publn_id | 1 | 100.00 | NULL |
| 1 | SIMPLE | t215 | NULL | ref | PRIMARY,ppi | ppi | 4 | patstat_2019_fall.t212.pat_publn_id | 47 | 100.00 | NULL |
+----+-------------+-------+------------+--------+---------------+---------+---------+-------------------------------------------+------+----------+-------+
4 rows in set, 1 warning (0.02 sec)
I see that the slow one says, "Using where" in the Extra column. This sounds ominous, but says only 11 rows.
My slow query log says:
Query_time: 269.658475 Lock_time: 0.000000 Rows_sent: 9 Rows_examined: 113106063
That's a lot of rows examined... but I have failed to understand WHY.
I'm confident this is obvious to someone, but I'm at the end of my rope.
Help?!
Only one index can be used for each join. If you have separate indexes for pat_publn_id and citn_id, it has to pick one of them to use, and then scan for the other column.
Since you're joining on two columns, to get the most effective indexing you need a multi-column index containing both columns.
ALTER TABLE tls212_citation ADD INDEX (pat_publn_id, citn_id);
ALTER TABLE tls215_citn_categ ADD INDEX (pat_publn_id, citn_id);
Does citn_id has single row entry in both t212 & t215? Or t212 has single but t215 has multiple entry. The second one cause speed issue. Even query can stop working.
You need to look on this, whether both table has single entry.
If multiple entry then left join won't work. You need to format the SQL in other way.
Query
SELECT SQL_NO_CACHE contacts.id,
contacts.date_modified contacts__date_modified
FROM contacts
INNER JOIN
(SELECT tst.team_set_id
FROM team_sets_teams tst
INNER JOIN team_memberships team_membershipscontacts ON (team_membershipscontacts.team_id = tst.team_id)
AND (team_membershipscontacts.user_id = '5daa2e92-c347-11e9-afc5-525400a80916')
AND (team_membershipscontacts.deleted = 0)
GROUP BY tst.team_set_id) contacts_tf ON contacts_tf.team_set_id = contacts.team_set_id
LEFT JOIN contacts_cstm contacts_cstm ON contacts_cstm.id_c = contacts.id
WHERE contacts.deleted = 0
ORDER BY contacts.date_modified DESC,
contacts.id DESC
LIMIT 21;
Takes extremely long (2 minutes on 2M records). I cant change this query, since it is system generated
This is it's explain:
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| 1 | PRIMARY | contacts | NULL | ref | idx_contacts_tmst_id,idx_del_date_modified,idx_contacts_del_last,idx_cont_del_reports,idx_del_id_user | idx_del_date_modified | 2 | const | 1113718 | 100.00 | Using temporary; Using filesort |
| 1 | PRIMARY | <derived3> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 50.00 | Using where; Using join buffer (Block Nested Loop) |
| 1 | PRIMARY | contacts_cstm | NULL | eq_ref | PRIMARY | PRIMARY | 144 | sugarcrm.contacts.id | 1 | 100.00 | Using index |
| 3 | DERIVED | team_membershipscontacts | NULL | ref | idx_team_membership,idx_teammemb_team_user,idx_del_team_user | idx_team_membership | 145 | const | 2 | 99.36 | Using index condition; Using where; Using temporary; Using filesort |
| 3 | DERIVED | tst | NULL | ref | idx_ud_set_id,idx_ud_team_id,idx_ud_team_set_id,idx_ud_team_id_team_set_id | idx_ud_team_id_team_set_id | 144 | sugarcrm.team_membershipscontacts.team_id | 1 | 100.00 | Using index |
+----+-------------+--------------------------+------------+--------+-------------------------------------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
But when I use force index(idx_del_date_modified) (which is the same index used in explain), the query takes just 0.01s and I get slightly different explain.
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
| 1 | PRIMARY | contacts | NULL | ref | idx_del_date_modified | idx_del_date_modified | 2 | const | 1113718 | 100.00 | Using where |
| 1 | PRIMARY | <derived2> | NULL | ALL | NULL | NULL | NULL | NULL | 2 | 50.00 | Using where |
| 1 | PRIMARY | contacts_cstm | NULL | eq_ref | PRIMARY | PRIMARY | 144 | sugarcrm.contacts.id | 1 | 100.00 | Using index |
| 2 | DERIVED | team_membershipscontacts | NULL | ref | idx_team_membership,idx_teammemb_team_user,idx_del_team_user | idx_team_membership | 145 | const | 2 | 99.36 | Using index condition; Using where; Using temporary; Using filesort |
| 2 | DERIVED | tst | NULL | ref | idx_ud_set_id,idx_ud_team_id,idx_ud_team_set_id,idx_ud_team_id_team_set_id | idx_ud_team_id_team_set_id | 144 | sugarcrm.team_membershipscontacts.team_id | 1 | 100.00 | Using index |
+----+-------------+--------------------------+------------+--------+----------------------------------------------------------------------------+----------------------------+---------+-------------------------------------------+---------+----------+---------------------------------------------------------------------+
The first query uses temporary table and file sort, but the query with force index uses just where. Shouldn't the queries be the same? Why is the query with force index so much faster - used index is still the same.
According to MySQL manual:
Temporary tables can be created under conditions such as these:
If there is an ORDER BY clause and a different GROUP BY clause, or if
the ORDER BY or GROUP BY contains columns from tables other than the
first table in the join queue, a temporary table is created.
DISTINCT combined with ORDER BY may require a temporary table.
If you use the SQL_SMALL_RESULT option, MySQL uses an in-memory
temporary table, unless the query also contains elements (described
later) that require on-disk storage.
Likely, you have better performance because in MySQL there is the query optimizer component.
If you create index the query optimizer could not use the index column even though the index exists.
Using force index(..) you are forcing MySql to use index, instead.
Please consider a detailed example here.
MySQL version 5.7.17
When I use JOIN for small tables, MySQL don't allow me to use index on the main table.
Compare:
EXPLAIN SELECT `ko_base_client_orders`.*
FROM `ko_base_client_orders`
LEFT JOIN `ko_base_new_perspe` AS `ko_of`
ON (`ko_of`.`id` = `ko_base_client_orders`.`office`)
ORDER BY `order_time` DESC
LIMIT 10 OFFSET 0;
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
| 1 | SIMPLE | ko_base_client_orders | NULL | ALL | NULL | NULL | NULL | NULL | 60737 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | ko_of | NULL | index | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.00 | Using where; Using index; Using join buffer (Block Nested Loop) |
+----+-------------+-----------------------+------------+-------+---------------+---------+---------+------+-------+----------+-----------------------------------------------------------------+
2 rows in set, 1 warning (0.00 sec)
and the same query with index hint:
EXPLAIN SELECT `ko_base_client_orders`.*
FROM `ko_base_client_orders`
LEFT JOIN `ko_base_new_perspe` AS `ko_of`
FORCE INDEX FOR JOIN (`PRIMARY`)
ON (`ko_of`.`id` = `ko_base_client_orders`.`office`)
ORDER BY `order_time` DESC
LIMIT 10 OFFSET 0;
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
| 1 | SIMPLE | ko_base_client_orders | NULL | index | NULL | order_time | 6 | NULL | 10 | 100.00 | NULL |
| 1 | SIMPLE | ko_of | NULL | eq_ref | PRIMARY | PRIMARY | 4 | e3.ko_base_client_orders.office | 1 | 100.00 | Using index |
+----+-------------+-----------------------+------------+--------+---------------+------------+---------+---------------------------------+------+----------+-------------+
In second case MySQL uses key for order_time, but in first example it ignores index and scans the entire table.
I can not predict which table would be small, and which would not (it differs from project to project), so I don't want to use index hints all the time. Is there another solution for this problem?
I have a query that is taking way too long to execute (4 seconds) even though all the fields i am querying against are indexed. Below are the query and the explain results. Any ideas what the problem is? (mysql CPU usage shoots up to 100% when executing the query
EXPLAIN SELECT count(hd.did) as NumPo, `hd`.`sid`, `src`.`Name`
FROM (`hd`)
JOIN `result` ON `result`.`did` = `hd`.`did`
JOIN `sf` ON `sf`.`fid` = `hd`.`fid`
JOIN `src` ON `src`.`sid` = `hd`.`sid`
WHERE `sf`.`tid` = 2
AND `result`.`set` = 'xxxxxxx'
GROUP BY `hd`.`sid`
ORDER BY `NumPo` DESC
LIMIT 10;
+----+-------------+--------------+--------+-------------------------+---------+---------+--------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+--------+-------------------------+---------+---------+--------------------------+------+----------------------------------------------+
| 1 | SIMPLE | sf | ref | PRIMARY,type | type | 2 | const | 4 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | hd | ref | PRIMARY,sid,fid | FeedID | 4 | f2.sf.fid | 3 | |
| 1 | SIMPLE | result | ALL | resultset | NULL | NULL | NULL | 5322 | Using where; Using join buffer |
| 1 | SIMPLE | src | eq_ref | PRIMARY | PRIMARY | 4 | f2.hd.sid | 1 | |
+----+-------------+--------------+--------+-------------------------+---------+---------+--------------------------+------+----------------------------------------------+
| 1 | SIMPLE | result | ALL | resultset | NULL | NULL | NULL | 5322 | Using where; Using join buffer |
It looks like it's not using an index on the biggest table. I'm having trouble guessing what this query is supposed to do, but it looks like you have an index on result.set, so I'd try adding one to result.did and see if it helps.