I have 2 mysql servers. 1 node is master and the other acting as slave, replicating from the master
The 2 nodes have identical data and schema.
However, 1 particular query is executed differently from mysql when run on both nodes
query
EXPLAIN SELECT t.*, COUNT(h.id)
FROM tags t
INNER JOIN tags2articles s
ON t.id = s.tag_id
INNER JOIN tag_hits h
ON h.id = s.tag_id
INNER JOIN articles art
ON art.id = s.`article_id`
WHERE art.source_id IN (SELECT id FROM feeds WHERE source_id = 15074)
AND time_added > DATE_SUB(NOW(), INTERVAL 1 DAY)
AND t.type = '1'
GROUP BY t.id
HAVING COUNT(h.id) > 4
ORDER BY COUNT(h.id) DESC
LIMIT 15
Below is the outpout from EXPLAIN query run on both nodes. Note that the master node is outputting
the correct one
output on master node
+----+--------------------+-------+-----------------+-----------------------------+---------------------+---------+----------------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+-----------------+-----------------------------+---------------------+---------+----------------+--------+----------------------------------------------+
| 1 | PRIMARY | art | ALL | PRIMARY | NULL | NULL | NULL | 100270 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | s | ref | PRIMARY,FK_tags2articles | FK_tags2articles | 4 | art.id | 12 | Using index |
| 1 | PRIMARY | h | ref | tags_hits_idx | tags_hits_idx | 4 | s.tag_id | 1 | Using index |
| 1 | PRIMARY | t | eq_ref | PRIMARY,tags_type_idx | PRIMARY | 4 | s.tag_id | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | feeds | unique_subquery | PRIMARY,f_source_id_idx | PRIMARY | 4 | func | 1 | Using where |
+----+--------------------+-------+-----------------+-----------------------------+---------------------+---------+----------------+--------+----------------------------------------------+
output on slave node
+----+--------------------+-------+-----------------+-----------------------------+------------------+---------+--------------------+--------+----------------------------------------
------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------+-----------------+-----------------------------+------------------+---------+--------------------+--------+----------------------------------------------+
| 1 | PRIMARY | t | ref | PRIMARY,tags_type_idx | tags_type_idx | 2 | const | 206432 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | h | ref | tags_hits_idx | tags_hits_idx | 4 | t.id | 1 | Using index |
| 1 | PRIMARY | s | ref | PRIMARY,FK_tags2articles | PRIMARY | 4 | h.id | 2 | Using where; Using index |
| 1 | PRIMARY | art | eq_ref | PRIMARY | PRIMARY | 4 | s.article_id | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | feeds | unique_subquery | PRIMARY,f_source_id_idx | PRIMARY | 4 | func | 1 | Using where |
+----+--------------------+-------+-----------------+-----------------------------+------------------+---------+--------------------+--------+----------------------------------------------+
I cannot understand why this discrepancy exists. Any help?
Thanks
They can have different statistics for indexes / keys and that causes differences in index usage. If possible (locks table, so not always recommended) run ANALYZE TABLE for all participating tables and then query plan is likely same.
Related
I ran following query in Mysql 5.5 and Mariadb 10.2. it took only 4 seconds to run the query in Mariadb while it took about 6 minutes in Mysql.
SELECT
' ' AS max_claim_amount,
claims_only_A.*
FROM
(
SELECT
employee.emp_number AS emp_number,
' ' AS emp_id,
' ' AS emp_name,
NULL AS estimate_id,
NULL AS estimate_submitted_date,
NULL AS estimate_state,
NULL AS currency_for_reimbursement,
NULL AS cash_in_advance,
NULL AS estimate_purpose,
NULL AS estimate_exp_type,
NULL AS estimate_foreign_currency,
NULL AS estimate_exchange_rate,
NULL AS estimate_amount,
NULL AS claim_id,
NULL AS claim_currency_for_reimbursement,
NULL AS claimed_date,
NULL AS claim_exp_type,
cety.id AS claim_exp_type_id,
claim_cc.currency_id AS claim_foreign_currency,
cex.exchange_rate AS claim_exchange_rate,
cex.amount AS claim_amount,
cex.remarks AS claim_remarks,
employee.deleted_at AS emp_deleted_at,
employee.purged_at AS emp_purged_at,
employee.termination_id AS emp_termination_id,
employee.emp_lastname AS emp_lastname,
el.location_id AS emp_location_id,
employee.job_title_code AS emp_job_title_code,
employee.work_station AS emp_work_station,
cr.request_id AS claim_request_id,
employee.emp_status AS emp_status,
NULL AS estimate_sort_id,
cex.id AS claim_exp_id
FROM
`claim_request` cr
LEFT JOIN `claim_expense` cex ON cex.request_id = cr.id
LEFT JOIN `claim_expense_type` cety ON cex.expense_type_id = cety.id
LEFT JOIN `_employee` AS employee ON cr.emp_number = employee.emp_number
LEFT JOIN claim_currency claim_cc ON (claim_cc.id = cex.currency_id)
LEFT JOIN claim_currency claim_req_cc ON (claim_req_cc.id = cr.currency_id)
LEFT JOIN _emp_locations el ON(employee.emp_number = el.emp_number)
WHERE
cr.id NOT IN (
SELECT
claim_request_id
FROM
`claim_estimation_claiming`
)
) AS claims_only_A
WHERE
(claim_request_id, claim_amount) NOT IN (
SELECT
claim_request_id,
MAX(claim_amount)
FROM
(
SELECT
cr.request_id AS claim_request_id,
cex.amount AS claim_amount,
cety.name AS claim_expense_type,
cex.id AS claim_exp_id
FROM
`claim_request` cr
LEFT JOIN `claim_expense` cex ON cex.request_id = cr.id
LEFT JOIN `claim_expense_type` cety ON cex.expense_type_id = cety.id
WHERE
cr.id NOT IN (
SELECT
claim_request_id
FROM
`claim_estimation_claiming`
)
) AS A
GROUP BY
claim_request_id,
claim_expense_type
)
Explain of the queried run were the followings,
-- MYSQL 5.5
+----+--------------------+--------------------------------+----------------+------------------+------------------+---------+-----------------------------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+--------------------------------+----------------+------------------+------------------+---------+-----------------------------------------+------+---------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2876 | Using where |
| 4 | DEPENDENT SUBQUERY | <derived5> | ALL | NULL | NULL | NULL | NULL | 2876 | Using temporary; Using filesort |
| 5 | DERIVED | cr | ALL | NULL | NULL | NULL | NULL | 1131 | Using where |
| 5 | DERIVED | cex | ref | request_id | request_id | 5 | dbname.cr.id | 1 | |
| 5 | DERIVED | cety | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cex.expense_type_id | 1 | |
| 6 | DEPENDENT SUBQUERY | claim_estimation_claiming | index_subquery | claim_request_id | claim_request_id | 5 | func | 2 | Using index |
| 2 | DERIVED | cr | ALL | NULL | NULL | NULL | NULL | 1131 | Using where |
| 2 | DERIVED | cex | ref | request_id | request_id | 5 | dbname.cr.id | 1 | |
| 2 | DERIVED | cety | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cex.expense_type_id | 1 | Using index |
| 2 | DERIVED | employee | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cr.emp_number | 1 | |
| 2 | DERIVED | claim_cc | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cex.currency_id | 1 | |
| 2 | DERIVED | claim_req_cc | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cr.currency_id | 1 | Using index |
| 2 | DERIVED | el | ref | PRIMARY | PRIMARY | 4 | dbname.employee.emp_number | 1 | Using index |
| 3 | DEPENDENT SUBQUERY | claim_estimation_claiming | index_subquery | claim_request_id | claim_request_id | 5 | func | 2 | Using index |
+----+--------------------+--------------------------------+----------------+------------------+------------------+---------+-----------------------------------------+------+---------------------------------+
-- MARIADB 10.2
+------+--------------+--------------------------------+--------+------------------+------------------+---------+-----------------------------------------+------+------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+--------------+--------------------------------+--------+------------------+------------------+---------+-----------------------------------------+------+------------------------------+
| 1 | PRIMARY | cr | ALL | NULL | NULL | NULL | NULL | 920 | Using where |
| 1 | PRIMARY | cex | ref | request_id | request_id | 5 | dbname.cr.id | 1 | Using where |
| 1 | PRIMARY | cety | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cex.expense_type_id | 1 | Using where; Using index |
| 1 | PRIMARY | employee | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cr.emp_number | 1 | Using where |
| 1 | PRIMARY | claim_cc | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cex.currency_id | 1 | Using where |
| 1 | PRIMARY | el | ref | PRIMARY | PRIMARY | 4 | dbname.employee.emp_number | 1 | Using where; Using index |
| 4 | MATERIALIZED | cr | ALL | NULL | NULL | NULL | NULL | 920 | Using where; Using temporary |
| 4 | MATERIALIZED | cex | ref | request_id | request_id | 5 | dbname.cr.id | 1 | |
| 4 | MATERIALIZED | cety | eq_ref | PRIMARY | PRIMARY | 4 | dbname.cex.expense_type_id | 1 | Using where |
| 6 | MATERIALIZED | claim_estimation_claiming | index | claim_request_id | claim_request_id | 5 | NULL | 1 | Using index |
| 3 | MATERIALIZED | claim_estimation_claiming | index | claim_request_id | claim_request_id | 5 | NULL | 1 | Using index |
+------+--------------+--------------------------------+--------+------------------+------------------+---------+-----------------------------------------+------+------------------------------+
I tried running sub-queries separately and the sub-queries didn't show any delay in Mysql. The problem seems to be only when the query is run as a whole.
As I feel, according to the explains, the issue seems to be because Mysql 5.5 has more All values in type field (it means mysql has to go through all the values in a subset).
Anyone has a better reasoning or Anyway to improve this query?
Turn
NOT IN ( SELECT ... )
into either
NOT EXISTS ( SELECT ... )
or
LEFT JOIN ... WHERE .. IS NULL
Then see if you can get rid of more subqueries.
If those don't speed it up enough, I'll look again.
The likely reason is that MariaDB has a feature that MySQL does not have: subquery caching. Also, MySQL has 3 major versions after 5.5, and there are some significant optimization improvements in them.
It would be interesting to see SHOW VARIABLES and SHOW GLOBAL STATUS from each server (plus RAM size). From that, I think I can point out that the caching is in use.
Meanwhile, my goal in the reformulation suggestions is to speed up MySQL (and maybe MariaDB).
Looks like there is a redundant left join which isn't used anywhere, so it will be helpful to remove it: claim_req_cc
Try to modify NOT IN to NOT EXISTS, as Rick mentioned.
EXPLAIN on this query
select v.type,sum(c.rank)
from
(select distinct power,color,type from vehicle) v
join configuration c using (power,color)
group by v.type
gives
+----+-------------+---------------+------------+-------+---------------+-------------+---------+-----------------------------------------+---------+----------+---------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+-------+---------------+-------------+---------+-----------------------------------------+---------+----------+---------------------------------+
| 1 | PRIMARY | configuration | NULL | ALL | veh | NULL | NULL | NULL | 76658 | 100.00 | Using temporary; Using filesort |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 6 | configuration.power,configuration.color | 65 | 100.00 | NULL |
| 2 | DERIVED | vehicle | NULL | index | cov | cov | 20 | NULL | 5058658 | 100.00 | Using index |
+----+-------------+---------------+------------+-------+---------------+-------------+---------+-----------------------------------------+---------+----------+---------------------------------+
The index on configuration (power,color) is not used even if I set force index
If I use a table instead of a subquery
create table tmp select distinct power,color,type from vehicle
then Explain on the 'same' query
select v.type,sum(c.rank)
from
tmp v
join configuration c using (power,color)
group by type
becomes
+----+-------------+---------------+------------+------+---------------+------+---------+---------------------+---------+----------+---------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------------+------------+------+---------------+------+---------+---------------------+---------+----------+---------------------------------+
| 1 | SIMPLE | tmp | NULL | ALL | NULL | NULL | NULL | NULL | 1016144 | 100.00 | Using temporary; Using filesort |
| 1 | SIMPLE | configuration | NULL | ref | veh | veh | 6 | tmp.power,tmp.color | 2 | 100.00 | NULL |
+----+-------------+---------------+------------+------+---------------+------+---------+---------------------+---------+----------+---------------------------------+
and this is 4 times faster
How can I avoid using a hard table ?
In the first case the optimizer thinks it is better to do it the other way around, by using the auto generated key in the derived table.
In the second case there is no key in the temp table, so the best plan is to go for tmp first.
You should be able to force the table order by using STRAIGHT_JOIN instead of JOIN.
I have a query which joins 3 tables (one inner join, one left join).
I need 2 WHERE conditions.
If I use either condition by itself, the search is quick. If I use both, the search will not complete.
I don't think I can add the search clause directly to the join as I don't want to exclude results where the search term is not present in that join.
SELECT job_entry.Job_Number,
job_entry.subject_ref,
contacts_library.company
FROM job_entry
LEFT JOIN multi_part_wind_instructions
ON job_entry.Job_Number = multi_part_wind_instructions.job_number
INNER JOIN contacts_library
ON job_entry.ContactID = contacts_library.ContactID
WHERE
company LIKE '%example%'
OR
multi_part_wind_instructions.address LIKE '%example%'
LIMIT 10
Explain Results query with both WHERE conditions:
+----+-------------+------------------------------+--------+---------------+---------+---------+-----------------------------+-------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------------------+--------+---------------+---------+---------+-----------------------------+-------+----------------------------------------------------+
| 1 | SIMPLE | job_entry | ALL | NULL | NULL | NULL | NULL | 16234 | Using where |
| 1 | SIMPLE | contacts_library | eq_ref | PRIMARY | PRIMARY | 4 | euroims.job_entry.ContactID | 1 | Using index condition; Using where |
| 1 | SIMPLE | multi_part_wind_instructions | ALL | NULL | NULL | NULL | NULL | 39447 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+------------------------------+--------+---------------+---------+---------+-----------------------------+-------+----------------------------------------------------+
Explain results from single WHERE condition (quick):
+----+-------------+------------------------------+--------+---------------+---------+---------+-----------------------------+-------+----------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------------------+--------+---------------+---------+---------+-----------------------------+-------+----------------------------------------------------+
| 1 | SIMPLE | job_entry | ALL | NULL | NULL | NULL | NULL | 16234 | Using where |
| 1 | SIMPLE | contacts_library | eq_ref | PRIMARY | PRIMARY | 4 | euroims.job_entry.ContactID | 1 | Using index condition; Using where |
| 1 | SIMPLE | multi_part_wind_instructions | ALL | NULL | NULL | NULL | NULL | 39447 | Using where; Using join buffer (Block Nested Loop) |
+----+-------------+------------------------------+--------+---------------+---------+---------+-----------------------------+-------+----------------------------------------------------+
upon enabling the slow_query_log in mysql. I found that the query below was one of slowest query with 15 secs or more. How can I optimize this query. All tables are Innodb and
server: Server version: 5.6 Thank you
EXPLAIN EXTENDED SELECT
ntv_staffs.fname,
ntv_staffs.id,
ntv_staffs.mname,
ntv_staffs.profile_pic,
ntv_staffs.lname,
ntv_staff_office.cug,
ntv_staff_office.extension,
ntv_staff_office.off_email,
ntv_staff_office.job_title,
ntv_designations.`name` as designation_name,
ntv_branches.`name` as branch_name,
ntv_departments.`name` as department_name,
ntv_departments.id as department_id
FROM
ntv_staffs
INNER JOIN ntv_staff_office ON ntv_staffs.id = ntv_staff_office.pid
INNER JOIN ntv_departments ON ntv_staff_office.department_id = ntv_departments.id
INNER JOIN ntv_branches ON ntv_staff_office.branch_id = ntv_branches.id
INNER JOIN ntv_designations ON ntv_staff_office.designation = ntv_designations.id
where ntv_staffs.id='662';
+----+-------------+------------------+--------+---------------+---------+---------+----------------------------------------------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------------+--------+---------------+---------+---------+----------------------------------------------+------+----------+-------------+
| 1 | SIMPLE | ntv_staffs | const | PRIMARY | PRIMARY | 4 | const | 1 | 100.00 | |
| 1 | SIMPLE | ntv_staff_office | ALL | NULL | NULL | NULL | NULL | 247 | 100.00 | Using where |
| 1 | SIMPLE | ntv_branches | eq_ref | PRIMARY | PRIMARY | 4 | abc_portal_new.sks_staff_office.branch_id | 1 | 100.00 | |
| 1 | SIMPLE | ntv_designations | eq_ref | PRIMARY | PRIMARY | 4 | abc_portal_new.sks_staff_office.designation | 1 | 100.00 | Using where |
| 1 | SIMPLE | ntv_departments | eq_ref | PRIMARY | PRIMARY | 4 | abc_portal_new.sks_staff_office.department_id | 1 | 100.00 | |
+----+-------------+------------------+--------+---------------+---------+---------+----------------------------------------------+------+----------+-------------+
Try setting a btree-index for ntv_staff_office.pid.
I was analizing a query (working on a wordpress plugin named nextgen gallery), this is what I got
query:
EXPLAIN
SELECT title, filename
FROM wp_ngg_pictures wnp
LEFT JOIN wp_ngg_gallery wng
ON wng.gid = wnp.galleryid
GROUP BY wnp.galleryid
LIMIT 5
result:
+----+-------------+-------+--------+---------------+---------+---------+-----------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+---------+---------+-----------------------+------+---------------------------------+
| 1 | SIMPLE | wnp | ALL | NULL | NULL | NULL | NULL | 439 | Using temporary; Using filesort |
| 1 | SIMPLE | wng | eq_ref | PRIMARY | PRIMARY | 8 | web1db1.wnp.galleryid | 1 | |
+----+-------------+-------+--------+---------------+---------+---------+-----------------------+------+---------------------------------+
so I do:
ALTER TABLE wp_ngg_pictures ADD INDEX(galleryid);
and on my local test system I get:
+----+-------------+-------+--------+---------------+-----------+---------+--------------------+------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+-----------+---------+--------------------+------+-------+
| 1 | SIMPLE | wnp | index | galleryid | galleryid | 8 | NULL | 30 | |
| 1 | SIMPLE | wng | eq_ref | PRIMARY | PRIMARY | 8 | test.wnp.galleryid | 1 | |
+----+-------------+-------+--------+---------------+-----------+---------+--------------------+------+-------+
which seems fine, but on the final server I get
+----+-------------+-------+--------+---------------+-----------+---------+-----------------------+------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+-----------+---------+-----------------------+------+-------+
| 1 | SIMPLE | wnp | index | galleryid | galleryid | 8 | NULL | 439 | |
| 1 | SIMPLE | wng | eq_ref | PRIMARY | PRIMARY | 8 | web1db1.wnp.galleryid | 1 | |
+----+-------------+-------+--------+---------------+-----------+---------+-----------------------+------+-------+
so the index is used but all the rows are scanned anyway? Why is this happening?
Only difference I can see is mysql version which is 5.1.47 (local) vs 5.0.45 (remote), data is the same on both systems.
The rows column in the EXPLAIN SELECT output is an estimate of the number of rows that MySQL believes it must examine to execute the query, so I guess it is possible that your local version (5.1.47) is better at estimating than your remote version.
Without the EXPLAIN clause, do both queries produce the same output? What happens if you change the query to use a STRAIGHT_JOIN?