I have got the query operating on rather small tables (most of 4000 rows each). THe query lasts about 4 seconds and contains several joins, however, all joins use indexes.
Here is the query:
SELECT count(DISTINCT p0_.id) AS sclr0
FROM promoter p0_
LEFT JOIN user u1_ ON p0_.user_id = u1_.id
LEFT JOIN profile p2_ ON p0_.id = p2_.promoter_id
LEFT JOIN promoter_city p4_ ON p0_.id = p4_.user_id
LEFT JOIN city c3_ ON c3_.id = p4_.city_id
LEFT JOIN promoter_language p6_ ON p0_.id = p6_.user_id
LEFT JOIN language l5_ ON l5_.id = p6_.language_id
LEFT JOIN promoter_worker_type p8_ ON p0_.id = p8_.promoter_id
LEFT JOIN worker_type w7_ ON w7_.id = p8_.workertype_id
LEFT JOIN practise p9_ ON p0_.id = p9_.promoter_id
LEFT JOIN contract c11_ ON p0_.id = c11_.promoter_id
And here is the output of the explain:
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | p0_ | index | |UNIQ_BCB929A3A76ED| 5 | | 4161 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | u1_ | eq_ref | PRIMARY | PRIMARY | 4 |promoteri.p0_.user| 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | p2_ | ref |type,IDX_8157AA0F4| type | 5 | promoteri.p0_.id | 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | p4_ | ref |PRIMARY,IDX_183C53| PRIMARY | 4 | promoteri.p0_.id | 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | c3_ | eq_ref | PRIMARY | PRIMARY | 4 |promoteri.p4_.city| 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | p6_ | ref |PRIMARY,IDX_19EE2A| PRIMARY | 4 | promoteri.p0_.id | 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | l5_ | eq_ref | PRIMARY | PRIMARY | 4 |promoteri.p6_.lang| 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | p8_ | ref |PRIMARY,IDX_37AC17| PRIMARY | 4 | promoteri.p0_.id | 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | w7_ | eq_ref | PRIMARY | PRIMARY | 4 |promoteri.p8_.work| 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | p9_ | ref |IDX_352E261F4B84B2|IDX_352E261F4B84B2| 5 | promoteri.p0_.id | 1 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
| 1 | SIMPLE | c11_ | ref |IDX_E98F28594B84B2|IDX_E98F28594B84B2| 5 | promoteri.p0_.id | 6 | Using index |
+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+------------------+
I do know, that the JOINS are unnecessary in this query, however I use lot of similar queries having some where statements applied, where it really matters to have these joins.
Is there any explanation, why this query takes so long?
Try changing all of those left joins to inner joins. When you use left join, you force the optimizer to join the tables in the sequence you've provided... when you use inner join, the optimizer has the option of picking a better starting position for joining the tables.
As far as a reference for left join forcing a specific read order - from the MySQL Docs:
The table read order forced by LEFT JOIN or STRAIGHT_JOIN helps the
join optimizer do its work much more quickly, because there are fewer
table permutations to check.
... Indicating that left join forces read order the same as STRAIGHT_JOIN.
Related
I am having issue with mysql query.
SELECT Count(*) AS aggregate
FROM (SELECT Group_concat(gateways.public_name) AS client_gateways,
`clients`.`id`,
`clients`.`name`,
`clients`.`status`,
`clients`.`api_key`,
`clients`.`user_name`,
`clients`.`psp_id`,
`clients`.`suspend`,
`clients`.`secret_key`,
`clients`.`created_at`,
`companies`.`name` AS `company_name`,
`mid_groups_mid`.`mid_id`,
`mid_groups_mid`.`mid_group_id`,
`mid_groups`.`id` AS `group_id`,
`mid_groups`.`user_id`,
`mids`.`mid_group_id` AS `id_of_mid`
FROM `clients`
LEFT JOIN `client_site_gateways`
ON `clients`.`id` = `client_site_gateways`.`client_id`
LEFT JOIN `gateways`
ON `client_site_gateways`.`gateway_id` = `gateways`.`id`
LEFT JOIN `client_broker`
ON `client_broker`.`client_id` = `clients`.`id`
LEFT JOIN `mid_groups`
ON `mid_groups`.`user_id` = `clients`.`psp_id`
LEFT JOIN `mid_groups_mid`
ON `mid_groups_mid`.`mid_group_id` = `mid_groups`.`id`
LEFT JOIN `mids`
ON `mids`.`mid_group_id` = `mid_groups_mid`.`mid_group_id`
INNER JOIN `companies`
ON `companies`.`id` = `clients`.`company_id`
WHERE `is_corp` = 0
AND `clients`.`suspend` = '0'
AND ( `clients`.`company_id` = 1 )
AND `clients`.`deleted_at` IS NULL
GROUP BY `clients`.`id`,
`clients`.`name`,
`clients`.`status`,
`clients`.`api_key`,
`clients`.`suspend`,
`clients`.`secret_key`,
`clients`.`created_at`,
`companies`.`name`,
`clients`.`user_name`,
`clients`.`psp_id`,
`mid_groups_mid`.`mid_id`,
`mid_groups_mid`.`mid_group_id`,
`mid_groups`.`id`,
`mid_groups`.`user_id`,
`mids`.`mid_group_id`) count_row_table
all table have few hundreds records. here is explain query result
+------+-------------+----------------------+--------+-------------------------------------+-------------------------------------+---------+----------------------------------------------+------------+-------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+----------------------+--------+-------------------------------------+-------------------------------------+---------+----------------------------------------------+------------+-------------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2849642280 | |
| 2 | DERIVED | companies | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 2 | DERIVED | clients | ref | clients_company_id_foreign | clients_company_id_foreign | 4 | const | 543 | Using where |
| 2 | DERIVED | client_site_gateways | ref | client_id | client_id | 4 | knox_staging.clients.id | 5 | |
| 2 | DERIVED | gateways | eq_ref | PRIMARY | PRIMARY | 4 | knox_staging.client_site_gateways.gateway_id | 1 | Using where |
| 2 | DERIVED | client_broker | ALL | NULL | NULL | NULL | NULL | 6 | Using where; Using join buffer (flat, BNL join) |
| 2 | DERIVED | mid_groups | ref | mid_groups_user_id_foreign | mid_groups_user_id_foreign | 4 | knox_staging.clients.psp_id | 1 | Using where; Using index |
| 2 | DERIVED | mid_groups_mid | ref | mid_groups_mid_mid_group_id_foreign | mid_groups_mid_mid_group_id_foreign | 8 | knox_staging.mid_groups.id | 433 | Using where |
| 2 | DERIVED | mids | ref | mids_mid_group_id_foreign | mids_mid_group_id_foreign | 9 | knox_staging.mid_groups_mid.mid_group_id | 404 | Using where; Using index |
+------+-------------+----------------------+--------+-------------------------------------+-------------------------------------+---------+----------------------------------------------+------------+-------------------------------------------------+
in explain results what is causing to have 2849642280 row. while tables have only few hundreds records. all tables have proper indexing.
what i am thinking causing storage full is tmp table with above records. i tried to scale storage upto 60GB database size is few MBs. all storage filled up as soon as i run above query. i am not sure what causing left join to filter 2849642280 rows
The problem is probably the "aggregate." If the only thing you need is the count of records, you should write a new query which gets that count.
I have an issue when adding an ORDER BY on my query.
without ORDER BY query takes around 26ms, as soon as I add a ORDER BY it's taking around 20s.
I have tried a few different ways but can seem to reduce the time.
Tried FORCE INDEX (PRIMARY)
Tried wrapping SELECT * FROM (...)
Does anyone have any other ideas?
QUERY:
SELECT
orders.id AS order_number,
orders.created AS order_created,
CONCAT_WS(' ', clients.name, office.name) AS client_office,
order_item.code AS product_code,
order_item.name AS product_name,
order_item.description AS product_description,
order_item.sale_price,
jobs.rating AS job_rating,
job_role.start_booking AS book_start,
job_role.end_booking AS book_end,
CONCAT_WS(' ', admin_staffs.first_name, admin_staffs.last_name) AS soul,
admin_roles.name AS role,
services.name AS service,
COUNT(job_item.id) AS total_assets,
COALESCE(orders.project_name, CONCAT_WS(' ', orders.location_unit_no, orders.location_street_number, orders.location_street_number, orders.location_street_name
)
) AS order_title
FROM jobs
LEFT JOIN job_item ON jobs.id = job_item.job_id
LEFT JOIN job_role ON jobs.id = job_role.job_id
LEFT JOIN admin_roles ON admin_roles.id = job_role.role_id
LEFT JOIN services ON services.id = jobs.service_id
LEFT JOIN admin_staffs ON admin_staffs.id = job_role.staff_id
LEFT JOIN order_item ON jobs.item_id = order_item.id
LEFT JOIN orders ON orders.id = order_item.order_id
LEFT JOIN clients ON orders.client_id = clients.id
LEFT JOIN office ON orders.office_id = office.id
LEFT JOIN client_users ON orders.user_id = client_users.id
GROUP BY jobs.id
ORDER BY jobs.order_id DESC
LIMIT 50
EXPLAIN:
+----+-------------+--------------+--------+----------------+-----------+---------+--------------------------------------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+--------+----------------+-----------+---------+--------------------------------------+-------+---------------------------------+
| 1 | SIMPLE | jobs | ALL | PRIMARY | NULL | NULL | NULL | 49555 | Using temporary; Using filesort |
| 1 | SIMPLE | job_item | ref | job_id | job_id | 5 | Wx3392vf_UCO_app.jobs.id | 8 | Using where; Using index |
| 1 | SIMPLE | job_role | ref | job_id | job_id | 4 | Wx3392vf_UCO_app.jobs.id | 1 | Using where |
| 1 | SIMPLE | admin_roles | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.job_role.role_id | 1 | Using where |
| 1 | SIMPLE | services | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.jobs.service_id | 1 | Using where |
| 1 | SIMPLE | admin_staffs | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.job_role.staff_id | 1 | NULL |
| 1 | SIMPLE | order_item | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.jobs.item_id | 1 | Using where |
| 1 | SIMPLE | orders | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.order_item.order_id | 1 | Using where |
| 1 | SIMPLE | clients | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.orders.client_id | 1 | Using where |
| 1 | SIMPLE | office | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.orders.office_id | 1 | Using where |
| 1 | SIMPLE | client_users | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.orders.user_id | 1 | Using index |
+----+-------------+--------------+--------+----------------+-----------+---------+--------------------------------------+-------+---------------------------------+
Try to start from this query:
SELECT * FROM jobs ORDER BY jobs.order_id DESC LIMIT 50
And then increase its complexity by adding more joins:
SELECT * FROM jobs
LEFT JOIN job_item ON jobs.id = job_item.job_id
ORDER BY jobs.order_id DESC LIMIT 50
etc.
Make profiling at each step and you will be able to find a bottleneck. Probably, it is some TEXT field in one of joined tables that slows down the query or some other reason.
I have a MySQL query which has a JOIN of 12 tables. When I explain the query, It showing 394699 rows for one table and 185368 rows for another table. All other tables has 1-3 rows. The total result which I am getting from the query id 472 rows only. But for that, it is taking more than 1 minute.
Is there any way to check how many rows has been analyzed to produce such a result? So that, I can find which is the table costs the higher time.
I am giving the query structure below. As the table structure is too high, I am not able to provide it here.
SELECT h.nid,h.attached_nid,h.created, s.field_species_value as species, g.field_gender_value as gender, u.field_unique_id_value as unqid, n.title, dob.field_adult_healthy_weight_value as birth_date, dcolor.field_dog_primary_color_value as dogcolor, ccolor.field_primary_color_value as catcolor, sdcolor.field_dog_secondary_color_value as sdogcolor, sccolor.field_secondary_color_value as scatcolor, dpattern.field_dog_pattern_value as dogpattern, cpattern.field_cat_pattern_value as catpattern
FROM table1 h
JOIN table2 n ON n.nid = h.nid
JOIN table3 s ON n.nid = s.entity_id
JOIN table4 u ON n.nid = u.entity_id
LEFT JOIN table5 g ON n.nid = g.entity_id
LEFT JOIN table6 dob ON n.nid = dob.entity_id
LEFT JOIN table7 AS dcolor ON n.nid = dcolor.entity_id
LEFT JOIN table8 AS ccolor ON n.nid = ccolor.entity_id
LEFT JOIN table9 AS sdcolor ON n.nid = sdcolor.entity_id
LEFT JOIN table10 AS sccolor ON n.nid = sccolor.entity_id
LEFT JOIN table11 AS dpattern ON n.nid = dpattern.entity_id
LEFT JOIN table12 AS cpattern ON n.nid = cpattern.entity_id
WHERE h.title = '4208'
AND ((h.created BETWEEN 1483257600 AND 1485935999))
AND h.uid!=1
AND h.uid IN(
SELECT etid
FROM `table`
WHERE gid=464
AND entity_type='user')
AND h.attached_nid>0
ORDER BY CAST(h.created as UNSIGNED) DESC;
Below is the EXPLAIN result which I get
+------+--------------+---------------+--------+----------------------+---------------------+---------+----------------------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+--------------+---------------+--------+----------------------+---------------------+---------+----------------------+--------+----------------------------------------------+
| 1 | PRIMARY | s | index | entity_id | field_species_value | 772 | NULL | 394699 | Using index; Using temporary; Using filesort |
| 1 | PRIMARY | u | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | n | eq_ref | PRIMARY | PRIMARY | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | g | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | dob | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | dcolor | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | ccolor | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | sdcolor | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | sccolor | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | dpattern | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | cpattern | ref | entity_id | entity_id | 4 | pantheon.s.entity_id | 1 | |
| 1 | PRIMARY | h | ref | attached_nid,nid,uid | nid | 5 | pantheon.s.entity_id | 3 | Using index condition; Using where |
| 1 | PRIMARY | <subquery2> | eq_ref | distinct_key | distinct_key | 4 | func | 1 | Using where |
| 2 | MATERIALIZED | og_membership | ref | entity,gid | gid | 4 | const | 185368 | Using where |
+------+--------------+---------------+--------+----------------------+---------------------+---------+----------------------+--------+----------------------------------------------+
You can find the ROWS_EXAMINED by using the Performance Schema.
Here is a link to the performance schema quick start guide.
https://dev.mysql.com/doc/refman/5.5/en/performance-schema-quick-start.html
This is the query I run in PHP applications, to find out what queries I need to optimize. You should be able to adapt it pretty easily.
The query finds the stats on the query that was run before this one. So in my apps, I run query after every query I run, store the results, then at the end of the PHP script I output the stats for every query I ran during the request.
SELECT `EVENT_ID`, TRUNCATE(`TIMER_WAIT`/1000000000000,6) as Duration,
`SQL_TEXT`, `DIGEST_TEXT`, `NO_INDEX_USED`, `NO_GOOD_INDEX_USED`, `ROWS_AFFECTED`, `ROWS_SENT`, `ROWS_EXAMINED`
FROM `performance_schema`.`events_statements_history`
WHERE
`CURRENT_SCHEMA` = '{$database}' AND `EVENT_NAME` LIKE 'statement/sql/%'
AND `THREAD_ID` = (SELECT `THREAD_ID` FROM `performance_schema`.`threads` WHERE `performance_schema`.`threads`.`PROCESSLIST_ID` = CONNECTION_ID())
ORDER BY `EVENT_ID` DESC LIMIT 1;
To decrease the number of rows accessed from og_membership, try adding an index containing the gid, entity_type, and etid fields. Including gid and entity_type should make the lookup more performant and including etid will make the index a covering index.
After adding the index, run EXPLAIN again to look at the results. Based on the new explain plan, either keep the index, remove the index, and/or add an additional index. Keep doing this until you get results you are satisfied with.
For sure, you will want to try and eliminate any mentions of Using temporary or Using filesort. Using temporary implies a temporary table is being used to make this query probably for the sheer size of your intermittent. Using filesort implies ordering isn't being satisfied with an index and is being done by examining the matching rows.
An detail explanation about EXPLAIN can be found at https://dev.mysql.com/doc/refman/5.7/en/explain-output.html.
Key-Value (EAV) schema sucks.
Indexes:
table1: INDEX(title, created)
table1: INDEX(uid, title, created)
table: INDEX(gid, entity_type, etid)
table* -- Is `entity_id` already an index? Can it be the PRIMARY KEY?
Does nid need to be NULL instead of NOT NULL?
If those don't do enough, try:
And turn the IN ( SELECT ... ) into a JOIN ( SELECT ... ) USING(hid)
If you still need help, please provide SHOW CREATE TABLE and EXPLAIN SELECT ...
I have a query which is very slow in INNER JOIN condition, but is faster when used in WHERE IN clause:
Slower Inner join:
SELECT *
FROM cases
left join
(
select tst.team_set_id
from team_sets_teams tst
INNER JOIN team_memberships team_memberships
ON tst.team_id = team_memberships.team_id
AND team_memberships.user_id = '1'
AND team_memberships.deleted=0 group by tst.team_set_id
) cases_tf
ON cases_tf.team_set_id = cases.team_set_id
LEFT JOIN contacts_cases
ON contacts_cases.case_id = cases.id
AND contacts_cases.deleted = 0
where cases.deleted=0
ORDER BY cases.name LIMIT 0,20;
Faster where in:
SELECT *
FROM cases
LEFT JOIN contacts_cases
ON contacts_cases.case_id = cases.id
AND contacts_cases.deleted = 0
where cases.deleted=0
and cases.team_set_id in (
select tst.team_set_id
from team_sets_teams tst
INNER JOIN team_memberships team_memberships
ON tst.team_id = team_memberships.team_id
AND team_memberships.user_id = '1'
AND team_memberships.deleted=0
group by tst.team_set_id
)
ORDER BY cases.name LIMIT 0,20;
The explain plan for INNER JOIN and WHERE IN clause are below:
Inner Join:
+----+-------------+------------------+------+--------------------------------------------+---------------------+---------+-----------------------------------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------+------+--------------------------------------------+---------------------+---------+-----------------------------------+--------+----------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 4 | Using temporary; Using filesort |
| 1 | PRIMARY | cases | ref | idx_cases_tmst_id | idx_cases_tmst_id | 109 | cases_tf.team_set_id | 446976 | Using where |
| 1 | PRIMARY | contacts_cases | ref | idx_con_case_case | idx_con_case_case | 111 | sugarcrm.cases.id | 1 | |
| 2 | DERIVED | team_memberships | ref | idx_team_membership,idx_teammemb_team_user | idx_team_membership | 109 | | 2 | Using where; Using temporary; Using filesort |
| 2 | DERIVED | tst | ref | idx_ud_team_id | idx_ud_team_id | 109 | sugarcrm.team_memberships.team_id | 1 | Using where |
+----+-------------+------------------+------+--------------------------------------------+---------------------+---------+-----------------------------------+--------+----------------------------------------------+
While in condition:
------+-----------------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+------------------+-------+--------------------------------------------+---------------------+---------+-----------------------------------+------+----------------------------------------------+
| 1 | PRIMARY | cases | index | NULL | idx_case_name | 768 | NULL | 20 | Using where |
| 1 | PRIMARY | contacts_cases | ref | idx_con_case_case | idx_con_case_case | 111 | sugarcrm.cases.id | 1 | |
| 2 | DEPENDENT SUBQUERY | team_memberships | ref | idx_team_membership,idx_teammemb_team_user | idx_team_membership | 109 | const | 2 | Using where; Using temporary; Using filesort |
| 2 | DEPENDENT SUBQUERY | tst | ref | idx_ud_team_id | idx_ud_team_id | 109 | sugarcrm.team_memberships.team_id | 1 | Using where |
+----+--------------------+------------------+-------+--------------------------------------------+---------------------+---------+-----------------------------------+------+----------------------------------------------+
Though there are indexes, I couldn't figure out what the problem. Please help me out. Thanks.
(This is a query in sugarcrm)
It is impossible to convert IN condition to INNER JOIN.
A query with IN condition is called semi-join.
A semi-join returns rows from one table that would join with another table, but without performing a complete join.
Here is a simple example of a semi-join query using IN operator:
SELECT *
FROM table1
WHERE some-column IN (
SELECT some-other-column
FROM table2
WHERE some-conditions
)
the above semi-join can be converted to semantically equivalent query
(eqivalent - means giving exactly same results)
using EXISTS operator and dependend subquery:
SELECT *
FROM table1
WHERE EXISTS(
SELECT 1
FROM table2
WHERE some-conditions
AND table1.some-column = table2.some-other-column
)
Most leading databases use the same plan for both of the above queries, and their speed is the same,
unfortunately this is not always true for MySql.
Joins and semi joins are totally different queries, with completely different execution plans, so comparing their speed is like comparing apples to onions.
You can try to convert the first query with IN into a query with EXIST, but not into the join.
I am using the following query, and as you can see via the results below, it is taking 2+ hours to return the results. I'd love to know how I could speed this up -- I believe I've already built the indexes correctly but it is still taking an incredible amount of time to return this data.
EXPLAIN SELECT Import_Values.base_vehicle_id,
Import_Values.qty,
Import_Values.part_type_id,
Import_Values.part_id,
Import_Values.position_id,
Import_Values.note,
Parts.partterminologyname,
BaseVehicle.YearID,
Make.MakeName,
Model.modelname,
SubModel.SubModelName,
EngineDesignation.EngineDesignationName,
EngineVIN.EngineVINName,
EngineBase.Liter,
EngineBase.CC,
EngineBase.CID,
EngineBase.Cylinders,
EngineBase.BlockType,
EngineBase.EngBoreIn,
EngineBase.EngBoreMetric,
EngineBase.EngStrokeIn,
EngineBase.EngStrokeMetric,
FuelDeliveryType.FuelDeliveryTypeName,
FuelDeliverySubType.FuelDeliverySubTypeName,
FuelSystemControlType.FuelSystemControlTypeName,
FuelSystemDesign.FuelSystemDesignName,
Aspiration.AspirationName,
CylinderHeadType.CylinderHeadTypeName,
FuelType.FuelTypeName,
IgnitionSystemType.IgnitionSystemTypeName,
Mfr.MfrName,
EngineVersion.EngineVersion,
Valves.ValvesPerEngine,
BedLength.BedLength,
BedLength.BedLengthMetric,
BedType.BedTypeName
FROM
Import_Values
INNER JOIN BaseVehicle
ON Import_Values.base_vehicle_id=BaseVehicle.BaseVehicleID
INNER JOIN Parts
ON Import_Values.part_type_id=Parts.PartTerminologyID
INNER JOIN Make
ON BaseVehicle.MakeID=Make.MakeID
INNER JOIN Model
ON BaseVehicle.ModelID=Model.ModelID
INNER JOIN Vehicle
ON Import_Values.base_vehicle_id=Vehicle.BaseVehicleID
INNER JOIN SubModel
ON Vehicle.SubModelID=SubModel.SubModelID
INNER JOIN VehicleConfig
ON Vehicle.VehicleID=VehicleConfig.VehicleID
INNER JOIN EngineConfig
ON VehicleConfig.EngineConfigID=EngineConfig.EngineConfigID
INNER JOIN EngineDesignation
ON EngineConfig.EngineDesignationID=EngineDesignation.EngineDesignationID
INNER JOIN EngineVIN
ON EngineConfig.EngineVINID=EngineVIN.EngineVINID
INNER JOIN EngineBase
ON EngineConfig.EngineBaseID=EngineBase.EngineBaseID
INNER JOIN FuelDeliveryConfig
ON EngineConfig.FuelDeliveryConfigID=FuelDeliveryConfig.FuelDeliveryConfigID
INNER JOIN FuelDeliveryType
ON FuelDeliveryConfig.FuelDeliveryTypeID=FuelDeliveryType.FuelDeliveryTypeID
INNER JOIN FuelDeliverySubType
ON FuelDeliveryConfig.FuelDeliverySubTypeID=FuelDeliverySubType.FuelDeliverySubTypeID
INNER JOIN FuelSystemControlType
ON FuelDeliveryConfig.FuelSystemControlTypeID=FuelSystemControlType.FuelSystemControlTypeID
INNER JOIN FuelSystemDesign
ON FuelDeliveryConfig.FuelSystemDesignID=FuelSystemDesign.FuelSystemDesignID
INNER JOIN Aspiration
ON EngineConfig.AspirationID=Aspiration.AspirationID
INNER JOIN CylinderHeadType
ON EngineConfig.CylinderHeadTypeID=CylinderHeadType.CylinderHeadTypeID
INNER JOIN FuelType
ON EngineConfig.FuelTypeID=FuelType.FuelTypeID
INNER JOIN IgnitionSystemType
ON EngineConfig.IgnitionSystemTypeID=IgnitionSystemType.IgnitionSystemTypeID
INNER JOIN Mfr
ON EngineConfig.EngineMfrID=Mfr.MfrID
INNER JOIN EngineVersion
ON EngineConfig.EngineVersionID=EngineVersion.EngineVersionID
INNER JOIN Valves
ON EngineConfig.ValvesID=Valves.Valvesid
INNER JOIN BedConfig
ON VehicleConfig.BedConfigID=BedConfig.BedConfigID
INNER JOIN BedLength
ON BedConfig.BedLengthID=BedLength.BedLengthID
INNER JOIN BedType
ON BedConfig.BedTypeID=BedType.BedTypeID
And the results...
+----+-------------+-----------------------+--------+-----------------------+---------+---------+-----------------------------------------------------------------+--------+-------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------------+--------+-----------------------+---------+---------+-----------------------------------------------------------------+--------+-------------------+
| 1 | SIMPLE | VehicleConfig | ALL | NULL | NULL | NULL | NULL | 171375 | |
| 1 | SIMPLE | Import_Values | ALL | base_vehicle_id | NULL | NULL | NULL | 18933 | Using join buffer |
| 1 | SIMPLE | BedConfig | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.VehicleConfig.BedConfigID | 1 | |
| 1 | SIMPLE | BedType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.BedConfig.BedTypeID | 1 | |
| 1 | SIMPLE | BedLength | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.BedConfig.BedLengthID | 1 | |
| 1 | SIMPLE | EngineConfig | eq_ref | PRIMARY,EngineBaseID | PRIMARY | 4 | icarcare_importfeeds.VehicleConfig.EngineConfigID | 1 | |
| 1 | SIMPLE | FuelType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.FuelTypeID | 1 | |
| 1 | SIMPLE | Valves | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.ValvesID | 1 | |
| 1 | SIMPLE | EngineVIN | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineVINID | 1 | |
| 1 | SIMPLE | EngineVersion | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineVersionID | 1 | |
| 1 | SIMPLE | IgnitionSystemType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.IgnitionSystemTypeID | 1 | |
| 1 | SIMPLE | Mfr | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineMfrID | 1 | |
| 1 | SIMPLE | Aspiration | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.AspirationID | 1 | |
| 1 | SIMPLE | FuelDeliveryConfig | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.FuelDeliveryConfigID | 1 | |
| 1 | SIMPLE | FuelSystemDesign | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.FuelDeliveryConfig.FuelSystemDesignID | 1 | |
| 1 | SIMPLE | FuelDeliverySubType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.FuelDeliveryConfig.FuelDeliverySubTypeID | 1 | |
| 1 | SIMPLE | EngineDesignation | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineDesignationID | 1 | |
| 1 | SIMPLE | EngineBase | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineBaseID | 1 | |
| 1 | SIMPLE | CylinderHeadType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.CylinderHeadTypeID | 1 | |
| 1 | SIMPLE | Parts | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.Import_Values.part_type_id | 1 | Using where |
| 1 | SIMPLE | FuelSystemControlType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.FuelDeliveryConfig.FuelSystemControlTypeID | 1 | |
| 1 | SIMPLE | BaseVehicle | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.Import_Values.base_vehicle_id | 1 | Using where |
| 1 | SIMPLE | Make | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.BaseVehicle.MakeID | 1 | |
| 1 | SIMPLE | Model | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.BaseVehicle.ModelID | 1 | |
| 1 | SIMPLE | Vehicle | eq_ref | PRIMARY,BaseVehicleID | PRIMARY | 4 | icarcare_importfeeds.VehicleConfig.VehicleID | 1 | Using where |
| 1 | SIMPLE | SubModel | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.Vehicle.SubModelID | 1 | |
| 1 | SIMPLE | FuelDeliveryType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.FuelDeliveryConfig.FuelDeliveryTypeID | 1 | |
+----+-------------+-----------------------+--------+-----------------------+---------+---------+-----------------------------------------------------------------+--------+-------------------+
27 rows in set (2 hours 39 min 30.51 sec)
Is there anything I can do? I've tried analyzing the tables, optimizing them, etc. It just seems there must be more to do than let it run for 2 hours, lol.
Are you sure that your query is logically correct? I know nothing about your database so I can't tell what's going on, but here is a tiny bit of the query that seems to go against the pattern set down by the rest of the lookups in the query.
Here it is:
INNER JOIN Vehicle
ON Import_Values.base_vehicle_id=Vehicle.BaseVehicleID
This looks strange, given the rest of your query. I would have expected the ON condition to be looking up one entry in Vehicle based on Vehicle.VehicleID which is probably the indexed field in that table. Instead, it's looking it up based on Vehicle.BaseVehicleID which is probably an unindexed field.
It's entirely possible that this is logically correct. But if it's logically incorrect, it could explain the long delay. If this turns out to be the case, then your query will deliver wrong results in addition to taking too long to complete.
Edit:
What is the relationship between ImportValues, Vehicle, and VehicleConfig?
Are there many VehicleConfigs for one Vehicle? Or is it vice versa?
It looks like The explain plan ended up using a plan that amounts to a Cartesian Join between ImportValues and VehicleConfig. It's very unusual for a Cartesian join to produce the desired results. In any event, you can expect a Cartesian Join to take a long, long time. Even if the query doesn't yield a Cartesian join, if the plan uses the same algorithm, it's still going to take a long, long time.
Second Edit:
There is another join condition that looks anomalous. It's the join condition under VehicleConfig.
INNER JOIN VehicleConfig
ON Vehicle.VehicleID=VehicleConfig.VehicleID
This is a pretty good join condition, but not for the VehicleConfig table. What it says, in plain English, is that VehicleConfig determines Vehicle. Not terribly surprising given the names of the tables and columns.
But what determines VehicleConfig?
If the answer is "nothing", then it's no wonder that poor old MySQL is doing a full scan on the table, in addition to a full scan on Import_Values.
My conclusion: fix your query so that 26 of the 27 tables are determined by an FK that references the table's PK. 24 of your joins already meet this criterion. When you are finally done, and you do an explain plan, you should see only one table doing a full table scan, namely the Import_Values table.
It should run faster, and it's more likely to produce correct results.
In looking at your explain plan, it looks like it's fine for everything except for Import_Values and VehicleConfig. It doesn't even look like it is considering any keys for VehicleConfig and just going straight into a full table scan which is bad.
Try creating a composite index for VehicleConfig on VehicleID and BedConfigID and an index for Import_Values on base_vehicle_id and part_type_id. If they already have indexes, it may just be a matter of updating statistics which you can do with the following commands. Hope that helps.
ANALYZE TABLE VehicleConfig
ANALYZE TABLE Import_Values
Please try applying an Execution Plan when executing this Query and see if doesn't suggest any missing keys/ indexes.
Try to set forceplan on and run the query. Another suggesstion is to
break the query into multiple queries with 6-7 table joins in each
query.