MYSQL slow ORDER BY - mysql

I have an issue when adding an ORDER BY on my query.
without ORDER BY query takes around 26ms, as soon as I add a ORDER BY it's taking around 20s.
I have tried a few different ways but can seem to reduce the time.
Tried FORCE INDEX (PRIMARY)
Tried wrapping SELECT * FROM (...)
Does anyone have any other ideas?
QUERY:
SELECT
orders.id AS order_number,
orders.created AS order_created,
CONCAT_WS(' ', clients.name, office.name) AS client_office,
order_item.code AS product_code,
order_item.name AS product_name,
order_item.description AS product_description,
order_item.sale_price,
jobs.rating AS job_rating,
job_role.start_booking AS book_start,
job_role.end_booking AS book_end,
CONCAT_WS(' ', admin_staffs.first_name, admin_staffs.last_name) AS soul,
admin_roles.name AS role,
services.name AS service,
COUNT(job_item.id) AS total_assets,
COALESCE(orders.project_name, CONCAT_WS(' ', orders.location_unit_no, orders.location_street_number, orders.location_street_number, orders.location_street_name
)
) AS order_title
FROM jobs
LEFT JOIN job_item ON jobs.id = job_item.job_id
LEFT JOIN job_role ON jobs.id = job_role.job_id
LEFT JOIN admin_roles ON admin_roles.id = job_role.role_id
LEFT JOIN services ON services.id = jobs.service_id
LEFT JOIN admin_staffs ON admin_staffs.id = job_role.staff_id
LEFT JOIN order_item ON jobs.item_id = order_item.id
LEFT JOIN orders ON orders.id = order_item.order_id
LEFT JOIN clients ON orders.client_id = clients.id
LEFT JOIN office ON orders.office_id = office.id
LEFT JOIN client_users ON orders.user_id = client_users.id
GROUP BY jobs.id
ORDER BY jobs.order_id DESC
LIMIT 50
EXPLAIN:
+----+-------------+--------------+--------+----------------+-----------+---------+--------------------------------------+-------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+--------+----------------+-----------+---------+--------------------------------------+-------+---------------------------------+
| 1 | SIMPLE | jobs | ALL | PRIMARY | NULL | NULL | NULL | 49555 | Using temporary; Using filesort |
| 1 | SIMPLE | job_item | ref | job_id | job_id | 5 | Wx3392vf_UCO_app.jobs.id | 8 | Using where; Using index |
| 1 | SIMPLE | job_role | ref | job_id | job_id | 4 | Wx3392vf_UCO_app.jobs.id | 1 | Using where |
| 1 | SIMPLE | admin_roles | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.job_role.role_id | 1 | Using where |
| 1 | SIMPLE | services | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.jobs.service_id | 1 | Using where |
| 1 | SIMPLE | admin_staffs | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.job_role.staff_id | 1 | NULL |
| 1 | SIMPLE | order_item | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.jobs.item_id | 1 | Using where |
| 1 | SIMPLE | orders | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.order_item.order_id | 1 | Using where |
| 1 | SIMPLE | clients | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.orders.client_id | 1 | Using where |
| 1 | SIMPLE | office | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.orders.office_id | 1 | Using where |
| 1 | SIMPLE | client_users | eq_ref | PRIMARY | PRIMARY | 4 | Wx3392vf_UCO_app.orders.user_id | 1 | Using index |
+----+-------------+--------------+--------+----------------+-----------+---------+--------------------------------------+-------+---------------------------------+

Try to start from this query:
SELECT * FROM jobs ORDER BY jobs.order_id DESC LIMIT 50
And then increase its complexity by adding more joins:
SELECT * FROM jobs
LEFT JOIN job_item ON jobs.id = job_item.job_id
ORDER BY jobs.order_id DESC LIMIT 50
etc.
Make profiling at each step and you will be able to find a bottleneck. Probably, it is some TEXT field in one of joined tables that slows down the query or some other reason.

Related

mysql Query make Db Storage full

I am having issue with mysql query.
SELECT Count(*) AS aggregate
FROM (SELECT Group_concat(gateways.public_name) AS client_gateways,
`clients`.`id`,
`clients`.`name`,
`clients`.`status`,
`clients`.`api_key`,
`clients`.`user_name`,
`clients`.`psp_id`,
`clients`.`suspend`,
`clients`.`secret_key`,
`clients`.`created_at`,
`companies`.`name` AS `company_name`,
`mid_groups_mid`.`mid_id`,
`mid_groups_mid`.`mid_group_id`,
`mid_groups`.`id` AS `group_id`,
`mid_groups`.`user_id`,
`mids`.`mid_group_id` AS `id_of_mid`
FROM `clients`
LEFT JOIN `client_site_gateways`
ON `clients`.`id` = `client_site_gateways`.`client_id`
LEFT JOIN `gateways`
ON `client_site_gateways`.`gateway_id` = `gateways`.`id`
LEFT JOIN `client_broker`
ON `client_broker`.`client_id` = `clients`.`id`
LEFT JOIN `mid_groups`
ON `mid_groups`.`user_id` = `clients`.`psp_id`
LEFT JOIN `mid_groups_mid`
ON `mid_groups_mid`.`mid_group_id` = `mid_groups`.`id`
LEFT JOIN `mids`
ON `mids`.`mid_group_id` = `mid_groups_mid`.`mid_group_id`
INNER JOIN `companies`
ON `companies`.`id` = `clients`.`company_id`
WHERE `is_corp` = 0
AND `clients`.`suspend` = '0'
AND ( `clients`.`company_id` = 1 )
AND `clients`.`deleted_at` IS NULL
GROUP BY `clients`.`id`,
`clients`.`name`,
`clients`.`status`,
`clients`.`api_key`,
`clients`.`suspend`,
`clients`.`secret_key`,
`clients`.`created_at`,
`companies`.`name`,
`clients`.`user_name`,
`clients`.`psp_id`,
`mid_groups_mid`.`mid_id`,
`mid_groups_mid`.`mid_group_id`,
`mid_groups`.`id`,
`mid_groups`.`user_id`,
`mids`.`mid_group_id`) count_row_table
all table have few hundreds records. here is explain query result
+------+-------------+----------------------+--------+-------------------------------------+-------------------------------------+---------+----------------------------------------------+------------+-------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+----------------------+--------+-------------------------------------+-------------------------------------+---------+----------------------------------------------+------------+-------------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2849642280 | |
| 2 | DERIVED | companies | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 2 | DERIVED | clients | ref | clients_company_id_foreign | clients_company_id_foreign | 4 | const | 543 | Using where |
| 2 | DERIVED | client_site_gateways | ref | client_id | client_id | 4 | knox_staging.clients.id | 5 | |
| 2 | DERIVED | gateways | eq_ref | PRIMARY | PRIMARY | 4 | knox_staging.client_site_gateways.gateway_id | 1 | Using where |
| 2 | DERIVED | client_broker | ALL | NULL | NULL | NULL | NULL | 6 | Using where; Using join buffer (flat, BNL join) |
| 2 | DERIVED | mid_groups | ref | mid_groups_user_id_foreign | mid_groups_user_id_foreign | 4 | knox_staging.clients.psp_id | 1 | Using where; Using index |
| 2 | DERIVED | mid_groups_mid | ref | mid_groups_mid_mid_group_id_foreign | mid_groups_mid_mid_group_id_foreign | 8 | knox_staging.mid_groups.id | 433 | Using where |
| 2 | DERIVED | mids | ref | mids_mid_group_id_foreign | mids_mid_group_id_foreign | 9 | knox_staging.mid_groups_mid.mid_group_id | 404 | Using where; Using index |
+------+-------------+----------------------+--------+-------------------------------------+-------------------------------------+---------+----------------------------------------------+------------+-------------------------------------------------+
in explain results what is causing to have 2849642280 row. while tables have only few hundreds records. all tables have proper indexing.
what i am thinking causing storage full is tmp table with above records. i tried to scale storage upto 60GB database size is few MBs. all storage filled up as soon as i run above query. i am not sure what causing left join to filter 2849642280 rows
The problem is probably the "aggregate." If the only thing you need is the count of records, you should write a new query which gets that count.

Sql query optimization and debug

I am trying to optimize sql query in mysql db. Tried different ways of rewriting it , adding/removing indexes, but nothing seems to decrease the load. Maybe I am missing something.
Query:
select co.country_name as state, ci.city_name as city, ci.city_id, ci.country_id,
count(l.id) as num
FROM cities ci
INNER JOIN countries co ON (ci.country_id = co.country_id)
INNER JOIN dancers l ON (l.city_id = ci.city_id AND l.closed = 0 AND l.approved = 1 )
WHERE 1 AND ci.main=1
GROUP BY ci.city_id
ORDER BY city
Duration : 2.01sec - 2.20sec
Optimized query:
select co.country_name as state, ci.city_name as city, ci.city_id, ci.country_id, count(l.id) as num from
(select ci1.city_name, ci1.city_id, ci1.country_id from cities ci1
where ci1.main=1) as ci
INNER JOIN countries co ON (ci.country_id = co.country_id)
INNER JOIN dancers l ON (l.city_id = ci.city_id AND l.closed = 0 AND l.approved = 1 ) GROUP BY ci.city_id ORDER BY city
Duration : 0.82sec - 0.90sec
But i feel that this query can be optimized even more but not getting the ideea how to optimized it. There are 3 tables
Table 1 : countries ( country_id, country_name)
Table 2 : cities ( city_id, city_name, main, country_id)
Table 3 : dancers ( id, country_id, city_id, closed, approved)
I am trying to get all the cities which have main=1 and for each to count all the profiles that are into those cities joining with countries to get the country_name.
Any ideas are welcomed, thank you.
Later edit : - first query explain
+----+-------------+-------+-------------+---------------------------------------------------------------------+-----------------+---------+------------------+-------+--------------------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------------+---------------------------------------------------------------------+-----------------+---------+------------------+-------+--------------------------------------------------------------------------------+
| 1 | SIMPLE | l | index_merge | city_id,closed,approved,city_id_2 | closed,approved | 1,2 | NULL | 75340 | Using intersect(closed,approved); Using where; Using temporary; Using filesort |
| 1 | SIMPLE | ci | eq_ref | PRIMARY,state_id_2,state_id,city_name,lat,city_name_shorter,city_id | PRIMARY | 4 | db.l.city_id | 1 | Using where |
| 1 | SIMPLE | co | eq_ref | PRIMARY | PRIMARY | 4 | db.ci.country_id | 1 | Using where |
+----+-------------+-------+-------------+---------------------------------------------------------------------+-----------------+---------+------------------+-------+--------------------------------------------------------------------------------+
Second query explain :
+----+-------------+------------+------+-----------------------------------+-------------+---------+------------------+-------+------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+------+-----------------------------------+-------------+---------+------------------+-------+------------------------------------+
| 1 | PRIMARY | co | ALL | PRIMARY | NULL | NULL | NULL | 51 | Using temporary; Using filesort |
| 1 | PRIMARY | <derived2> | ref | <auto_key1> | <auto_key1> | 4 | db.co.country_id | 176 | Using where |
| 1 | PRIMARY | l | ref | city_id,closed,approved,city_id_2 | city_id_2 | 4 | ci.city_id | 44 | Using index condition; Using where |
| 2 | DERIVED | ci1 | ALL | NULL | NULL | NULL | NULL | 11765 | Using where |
+----+-------------+------------+------+-----------------------------------+-------------+---------+------------------+-------+------------------------------------+
#used_by_already query explain :
+----+-------------+------------+-------------+-----------------------------------+-----------------+---------+------------------+-------+--------------------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------+-------------+-----------------------------------+-----------------+---------+------------------+-------+--------------------------------------------------------------------------------+
| 1 | PRIMARY | co | ALL | PRIMARY | NULL | NULL | NULL | 51 | NULL |
| 1 | PRIMARY | <derived2> | ref | <auto_key0> | <auto_key0> | 4 | db.co.country_id | 565 | Using where |
| 2 | DERIVED | l | index_merge | city_id,closed,approved,city_id_2 | closed,approved | 1,2 | NULL | 75341 | Using intersect(closed,approved); Using where; Using temporary; Using filesort |
| 2 | DERIVED | ci1 | eq_ref | PRIMARY,state_id_2,city_id | PRIMARY | 4 | db.l.city_id | 1 | Using where |
+----+-------------+------------+-------------+-----------------------------------+-----------------+---------+------------------+-------+--------------------------------------------------------------------------------+
I suggest you try this:
SELECT
co.country_name AS state
, ci.city_name AS city
, ci.city_id
, ci.country_id
, ci.num
FROM (
SELECT
ci1.city_id
, ci1.city_name
, ci1.country_id
, COUNT(l.id) AS num
FROM cities ci1
INNER JOIN dancers l ON l.city_id = ci1.city_id
AND l.closed = 0
AND l.approved = 1
WHERE ci1.main = 1
GROUP BY
ci1.city_id
, ci1.city_name
, ci1.country_id
) AS ci
INNER JOIN countries co ON ci.country_id = co.country_id
;
And that you provide the explain plan output for further analysis if needed. When optimizing knowing what indexes exist, and having he explain plan, are essentials.
Not also, that MySQL does permit non-standard GROUP BY syntax (where only one or some of the columns in the select list are included under the group by clause).
In recent versions of MySQL the default behaviour for GROUP BY has changed to SQL standard syntax (where all "non-aggregating" columns in the select list must be included under the group by clause). While your existing queries use the non-standard group by syntax, the query supplied here dis compliant.

How to speed up a complex SQL query with JOINs

I am using the following query, and as you can see via the results below, it is taking 2+ hours to return the results. I'd love to know how I could speed this up -- I believe I've already built the indexes correctly but it is still taking an incredible amount of time to return this data.
EXPLAIN SELECT Import_Values.base_vehicle_id,
Import_Values.qty,
Import_Values.part_type_id,
Import_Values.part_id,
Import_Values.position_id,
Import_Values.note,
Parts.partterminologyname,
BaseVehicle.YearID,
Make.MakeName,
Model.modelname,
SubModel.SubModelName,
EngineDesignation.EngineDesignationName,
EngineVIN.EngineVINName,
EngineBase.Liter,
EngineBase.CC,
EngineBase.CID,
EngineBase.Cylinders,
EngineBase.BlockType,
EngineBase.EngBoreIn,
EngineBase.EngBoreMetric,
EngineBase.EngStrokeIn,
EngineBase.EngStrokeMetric,
FuelDeliveryType.FuelDeliveryTypeName,
FuelDeliverySubType.FuelDeliverySubTypeName,
FuelSystemControlType.FuelSystemControlTypeName,
FuelSystemDesign.FuelSystemDesignName,
Aspiration.AspirationName,
CylinderHeadType.CylinderHeadTypeName,
FuelType.FuelTypeName,
IgnitionSystemType.IgnitionSystemTypeName,
Mfr.MfrName,
EngineVersion.EngineVersion,
Valves.ValvesPerEngine,
BedLength.BedLength,
BedLength.BedLengthMetric,
BedType.BedTypeName
FROM
Import_Values
INNER JOIN BaseVehicle
ON Import_Values.base_vehicle_id=BaseVehicle.BaseVehicleID
INNER JOIN Parts
ON Import_Values.part_type_id=Parts.PartTerminologyID
INNER JOIN Make
ON BaseVehicle.MakeID=Make.MakeID
INNER JOIN Model
ON BaseVehicle.ModelID=Model.ModelID
INNER JOIN Vehicle
ON Import_Values.base_vehicle_id=Vehicle.BaseVehicleID
INNER JOIN SubModel
ON Vehicle.SubModelID=SubModel.SubModelID
INNER JOIN VehicleConfig
ON Vehicle.VehicleID=VehicleConfig.VehicleID
INNER JOIN EngineConfig
ON VehicleConfig.EngineConfigID=EngineConfig.EngineConfigID
INNER JOIN EngineDesignation
ON EngineConfig.EngineDesignationID=EngineDesignation.EngineDesignationID
INNER JOIN EngineVIN
ON EngineConfig.EngineVINID=EngineVIN.EngineVINID
INNER JOIN EngineBase
ON EngineConfig.EngineBaseID=EngineBase.EngineBaseID
INNER JOIN FuelDeliveryConfig
ON EngineConfig.FuelDeliveryConfigID=FuelDeliveryConfig.FuelDeliveryConfigID
INNER JOIN FuelDeliveryType
ON FuelDeliveryConfig.FuelDeliveryTypeID=FuelDeliveryType.FuelDeliveryTypeID
INNER JOIN FuelDeliverySubType
ON FuelDeliveryConfig.FuelDeliverySubTypeID=FuelDeliverySubType.FuelDeliverySubTypeID
INNER JOIN FuelSystemControlType
ON FuelDeliveryConfig.FuelSystemControlTypeID=FuelSystemControlType.FuelSystemControlTypeID
INNER JOIN FuelSystemDesign
ON FuelDeliveryConfig.FuelSystemDesignID=FuelSystemDesign.FuelSystemDesignID
INNER JOIN Aspiration
ON EngineConfig.AspirationID=Aspiration.AspirationID
INNER JOIN CylinderHeadType
ON EngineConfig.CylinderHeadTypeID=CylinderHeadType.CylinderHeadTypeID
INNER JOIN FuelType
ON EngineConfig.FuelTypeID=FuelType.FuelTypeID
INNER JOIN IgnitionSystemType
ON EngineConfig.IgnitionSystemTypeID=IgnitionSystemType.IgnitionSystemTypeID
INNER JOIN Mfr
ON EngineConfig.EngineMfrID=Mfr.MfrID
INNER JOIN EngineVersion
ON EngineConfig.EngineVersionID=EngineVersion.EngineVersionID
INNER JOIN Valves
ON EngineConfig.ValvesID=Valves.Valvesid
INNER JOIN BedConfig
ON VehicleConfig.BedConfigID=BedConfig.BedConfigID
INNER JOIN BedLength
ON BedConfig.BedLengthID=BedLength.BedLengthID
INNER JOIN BedType
ON BedConfig.BedTypeID=BedType.BedTypeID
And the results...
+----+-------------+-----------------------+--------+-----------------------+---------+---------+-----------------------------------------------------------------+--------+-------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------------------+--------+-----------------------+---------+---------+-----------------------------------------------------------------+--------+-------------------+
| 1 | SIMPLE | VehicleConfig | ALL | NULL | NULL | NULL | NULL | 171375 | |
| 1 | SIMPLE | Import_Values | ALL | base_vehicle_id | NULL | NULL | NULL | 18933 | Using join buffer |
| 1 | SIMPLE | BedConfig | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.VehicleConfig.BedConfigID | 1 | |
| 1 | SIMPLE | BedType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.BedConfig.BedTypeID | 1 | |
| 1 | SIMPLE | BedLength | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.BedConfig.BedLengthID | 1 | |
| 1 | SIMPLE | EngineConfig | eq_ref | PRIMARY,EngineBaseID | PRIMARY | 4 | icarcare_importfeeds.VehicleConfig.EngineConfigID | 1 | |
| 1 | SIMPLE | FuelType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.FuelTypeID | 1 | |
| 1 | SIMPLE | Valves | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.ValvesID | 1 | |
| 1 | SIMPLE | EngineVIN | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineVINID | 1 | |
| 1 | SIMPLE | EngineVersion | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineVersionID | 1 | |
| 1 | SIMPLE | IgnitionSystemType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.IgnitionSystemTypeID | 1 | |
| 1 | SIMPLE | Mfr | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineMfrID | 1 | |
| 1 | SIMPLE | Aspiration | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.AspirationID | 1 | |
| 1 | SIMPLE | FuelDeliveryConfig | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.FuelDeliveryConfigID | 1 | |
| 1 | SIMPLE | FuelSystemDesign | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.FuelDeliveryConfig.FuelSystemDesignID | 1 | |
| 1 | SIMPLE | FuelDeliverySubType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.FuelDeliveryConfig.FuelDeliverySubTypeID | 1 | |
| 1 | SIMPLE | EngineDesignation | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineDesignationID | 1 | |
| 1 | SIMPLE | EngineBase | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.EngineBaseID | 1 | |
| 1 | SIMPLE | CylinderHeadType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.EngineConfig.CylinderHeadTypeID | 1 | |
| 1 | SIMPLE | Parts | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.Import_Values.part_type_id | 1 | Using where |
| 1 | SIMPLE | FuelSystemControlType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.FuelDeliveryConfig.FuelSystemControlTypeID | 1 | |
| 1 | SIMPLE | BaseVehicle | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.Import_Values.base_vehicle_id | 1 | Using where |
| 1 | SIMPLE | Make | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.BaseVehicle.MakeID | 1 | |
| 1 | SIMPLE | Model | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.BaseVehicle.ModelID | 1 | |
| 1 | SIMPLE | Vehicle | eq_ref | PRIMARY,BaseVehicleID | PRIMARY | 4 | icarcare_importfeeds.VehicleConfig.VehicleID | 1 | Using where |
| 1 | SIMPLE | SubModel | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.Vehicle.SubModelID | 1 | |
| 1 | SIMPLE | FuelDeliveryType | eq_ref | PRIMARY | PRIMARY | 4 | icarcare_importfeeds.FuelDeliveryConfig.FuelDeliveryTypeID | 1 | |
+----+-------------+-----------------------+--------+-----------------------+---------+---------+-----------------------------------------------------------------+--------+-------------------+
27 rows in set (2 hours 39 min 30.51 sec)
Is there anything I can do? I've tried analyzing the tables, optimizing them, etc. It just seems there must be more to do than let it run for 2 hours, lol.
Are you sure that your query is logically correct? I know nothing about your database so I can't tell what's going on, but here is a tiny bit of the query that seems to go against the pattern set down by the rest of the lookups in the query.
Here it is:
INNER JOIN Vehicle
ON Import_Values.base_vehicle_id=Vehicle.BaseVehicleID
This looks strange, given the rest of your query. I would have expected the ON condition to be looking up one entry in Vehicle based on Vehicle.VehicleID which is probably the indexed field in that table. Instead, it's looking it up based on Vehicle.BaseVehicleID which is probably an unindexed field.
It's entirely possible that this is logically correct. But if it's logically incorrect, it could explain the long delay. If this turns out to be the case, then your query will deliver wrong results in addition to taking too long to complete.
Edit:
What is the relationship between ImportValues, Vehicle, and VehicleConfig?
Are there many VehicleConfigs for one Vehicle? Or is it vice versa?
It looks like The explain plan ended up using a plan that amounts to a Cartesian Join between ImportValues and VehicleConfig. It's very unusual for a Cartesian join to produce the desired results. In any event, you can expect a Cartesian Join to take a long, long time. Even if the query doesn't yield a Cartesian join, if the plan uses the same algorithm, it's still going to take a long, long time.
Second Edit:
There is another join condition that looks anomalous. It's the join condition under VehicleConfig.
INNER JOIN VehicleConfig
ON Vehicle.VehicleID=VehicleConfig.VehicleID
This is a pretty good join condition, but not for the VehicleConfig table. What it says, in plain English, is that VehicleConfig determines Vehicle. Not terribly surprising given the names of the tables and columns.
But what determines VehicleConfig?
If the answer is "nothing", then it's no wonder that poor old MySQL is doing a full scan on the table, in addition to a full scan on Import_Values.
My conclusion: fix your query so that 26 of the 27 tables are determined by an FK that references the table's PK. 24 of your joins already meet this criterion. When you are finally done, and you do an explain plan, you should see only one table doing a full table scan, namely the Import_Values table.
It should run faster, and it's more likely to produce correct results.
In looking at your explain plan, it looks like it's fine for everything except for Import_Values and VehicleConfig. It doesn't even look like it is considering any keys for VehicleConfig and just going straight into a full table scan which is bad.
Try creating a composite index for VehicleConfig on VehicleID and BedConfigID and an index for Import_Values on base_vehicle_id and part_type_id. If they already have indexes, it may just be a matter of updating statistics which you can do with the following commands. Hope that helps.
ANALYZE TABLE VehicleConfig
ANALYZE TABLE Import_Values
Please try applying an Execution Plan when executing this Query and see if doesn't suggest any missing keys/ indexes.
Try to set forceplan on and run the query. Another suggesstion is to
break the query into multiple queries with 6-7 table joins in each
query.

MySQL left join performance issues

I have been having issues with MySQL (version 5.5) left join performance on a number of queries. In all cases I have been able to work around the issue by restructuring the queries with unions and subselects (I saw some examples of this in the book High Performance MySQL). The problem is this this leads to very messy queries.
Below is an example of two queries that produce the exact same results. The first query is roughly two orders of magnitude slower than the second. The second query is much less readable than the first.
As far as I can tell these sorts of queries are not performing poorly because of bad indexing. In all cases when I restructure the query it runs just fine. I have also tried carefully looking at the indexes and using hints to no avail.
Has anyone else run into similar issues with MySQL? Are there any server parameters I should try tweaking? Has anyone found a cleaner way to work around this sort of issue?
Query 1
select
i.id,
sum(vp.measurement * pol.quantity_ordered) measurement_on_order
from items i
left join (vendor_products vp, purchase_order_lines pol, purchase_orders po) on
vp.item_id = i.id and
pol.vendor_product_id = vp.id and
pol.purchase_order_id = po.id and
po.received_at is null and
po.closed_at is null
group by i.id
explain:
+----+-------------+-------+--------+-------------------------------+-------------------+---------+-------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-------------------------------+-------------------+---------+-------------------------------------+------+-------------+
| 1 | SIMPLE | i | index | NULL | PRIMARY | 4 | NULL | 241 | Using index |
| 1 | SIMPLE | po | ref | PRIMARY,received_at,closed_at | received_at | 9 | const | 2 | |
| 1 | SIMPLE | pol | ref | purchase_order_id | purchase_order_id | 4 | nutkernel_dev.po.id | 7 | |
| 1 | SIMPLE | vp | eq_ref | PRIMARY,item_id | PRIMARY | 4 | nutkernel_dev.pol.vendor_product_id | 1 | |
+----+-------------+-------+--------+-------------------------------+-------------------+---------+-------------------------------------+------+-------------+
Query 2
select
i.id,
sum(on_order.measurement_on_order) measurement_on_order
from (
(
select
i.id item_id,
sum(vp.measurement * pol.quantity_ordered) measurement_on_order
from purchase_orders po
join purchase_order_lines pol on pol.purchase_order_id = po.id
join vendor_products vp on pol.vendor_product_id = vp.id
join items i on vp.item_id = i.id
where
po.received_at is null and po.closed_at is null
group by i.id
)
union all
(select id, 0 from items)
) on_order
join items i on on_order.item_id = i.id
group by i.id
explain:
+------+--------------+------------+--------+-------------------------------+--------------------------------+---------+-------------------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+--------------+------------+--------+-------------------------------+--------------------------------+---------+-------------------------------------+------+----------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 3793 | Using temporary; Using filesort |
| 1 | PRIMARY | i | eq_ref | PRIMARY | PRIMARY | 4 | on_order.item_id | 1 | Using index |
| 2 | DERIVED | po | ALL | PRIMARY,received_at,closed_at | NULL | NULL | NULL | 20 | Using where; Using temporary; Using filesort |
| 2 | DERIVED | pol | ref | purchase_order_id | purchase_order_id | 4 | nutkernel_dev.po.id | 7 | |
| 2 | DERIVED | vp | eq_ref | PRIMARY,item_id | PRIMARY | 4 | nutkernel_dev.pol.vendor_product_id | 1 | |
| 2 | DERIVED | i | eq_ref | PRIMARY | PRIMARY | 4 | nutkernel_dev.vp.item_id | 1 | Using index |
| 3 | UNION | items | index | NULL | index_new_items_on_external_id | 257 | NULL | 3380 | Using index |
| NULL | UNION RESULT | <union2,3> | ALL | NULL | NULL | NULL | NULL | NULL | |
+------+--------------+------------+--------+-------------------------------+--------------------------------+---------+-------------------------------------+------+----------------------------------------------+

slow mysql count because of subselect

how to make this select statement more faster?
the first left join with the subselect is making it slower...
mysql> SELECT COUNT(DISTINCT w1.id) AS AMOUNT FROM tblWerbemittel w1
JOIN tblVorgang v1 ON w1.object_group = v1.werbemittel_id
INNER JOIN ( SELECT wmax.object_group, MAX( wmax.object_revision ) wmaxobjrev FROM tblWerbemittel wmax GROUP BY wmax.object_group ) AS wmaxselect ON w1.object_group = wmaxselect.object_group AND w1.object_revision = wmaxselect.wmaxobjrev
LEFT JOIN ( SELECT vmax.object_group, MAX( vmax.object_revision ) vmaxobjrev FROM tblVorgang vmax GROUP BY vmax.object_group ) AS vmaxselect ON v1.object_group = vmaxselect.object_group AND v1.object_revision = vmaxselect.vmaxobjrev
LEFT JOIN tblWerbemittel_has_tblAngebot wha ON wha.werbemittel_id = w1.object_group
LEFT JOIN tblAngebot ta ON ta.id = wha.angebot_id
LEFT JOIN tblLieferanten tl ON tl.id = ta.lieferant_id AND wha.zuschlag = (SELECT MAX(zuschlag) FROM tblWerbemittel_has_tblAngebot WHERE werbemittel_id = w1.object_group)
WHERE w1.flags =0 AND v1.flags=0;
+--------+
| AMOUNT |
+--------+
| 1982 |
+--------+
1 row in set (1.30 sec)
Some indexes has been already set and as EXPLAIN shows they were used.
+----+--------------------+-------------------------------+--------+----------------------------------------+----------------------+---------+-----------------------------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------------------------------+--------+----------------------------------------+----------------------+---------+-----------------------------------------------+------+----------------------------------------------+
| 1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 2072 | |
| 1 | PRIMARY | v1 | ref | werbemittel_group,werbemittel_id_index | werbemittel_group | 4 | wmaxselect.object_group | 2 | Using where |
| 1 | PRIMARY | <derived3> | ALL | NULL | NULL | NULL | NULL | 3376 | |
| 1 | PRIMARY | w1 | eq_ref | object_revision,or_og_index | object_revision | 8 | wmaxselect.wmaxobjrev,wmaxselect.object_group | 1 | Using where |
| 1 | PRIMARY | wha | ref | PRIMARY,werbemittel_id_index | werbemittel_id_index | 4 | dpd.w1.object_group | 1 | |
| 1 | PRIMARY | ta | eq_ref | PRIMARY | PRIMARY | 4 | dpd.wha.angebot_id | 1 | |
| 1 | PRIMARY | tl | eq_ref | PRIMARY | PRIMARY | 4 | dpd.ta.lieferant_id | 1 | Using index |
| 4 | DEPENDENT SUBQUERY | tblWerbemittel_has_tblAngebot | ref | PRIMARY,werbemittel_id_index | werbemittel_id_index | 4 | dpd.w1.object_group | 1 | |
| 3 | DERIVED | vmax | index | NULL | object_revision_uq | 8 | NULL | 4668 | Using index; Using temporary; Using filesort |
| 2 | DERIVED | wmax | range | NULL | or_og_index | 4 | NULL | 2168 | Using index for group-by |
+----+--------------------+-------------------------------+--------+----------------------------------------+----------------------+---------+-----------------------------------------------+------+----------------------------------------------+
10 rows in set (0.01 sec)
The main problem while the statement above takes about 2 seconds seems to be the subselect where no index can be used.
How to write the statement even more faster?
Thanks for help. MT
Do you have the following indexes?
for tblWerbemittel - object_group, object_revision
for tblVorgang - object_group, object_revision
for tblWerbemittel_has_tblAngebot - werbemittel_id, zuschlag
Let me know if that helps, there are a few more that I can see might help but try those first.
EDIT
Can you try these two queries and see if they run fast?
SELECT w1.id AS AMOUNT
FROM tblWerbemittel w1 INNER JOIN
(SELECT wmax.object_group,
MAX( wmax.object_revision ) AS wmaxobjrev
FROM tblWerbemittel AS wmax
GROUP BY wmax.object_group ) AS wmaxselect ON w1.object_group = wmaxselect.object_group AND
w1.object_revision = wmaxselect.wmaxobjrev
WHERE w1.flags = 0
SELECT v1.werbemittel_id
FROM tblVorgang v1 LEFT JOIN
(SELECT vmax.object_group,
MAX( vmax.object_revision ) AS vmaxobjrev
FROM tblVorgang AS vmax
GROUP BY vmax.object_group ) AS vmaxselect ON v1.object_group = vmaxselect.object_group AND
v1.object_revision = vmaxselect.vmaxobjrev LEFT JOIN
WHERE v1.flags = 0
While I consider I don't have sufficient data to provide a 100% correct answer, but I can throw in a handful of tips.
Forst of all, MYSQL is stupid. Bear that in mind and always rearrange your queries so that the most data is excluded at the beginning. For instance, if the last join reduced the number of results from 10k to 2k while the others don't, try swapping their positions so that each subsequent join operates on the smallest subset of data possible.
Same applies to the WHERE clause.
Also, joins tend to be slower than subqueries. I don't know if that's a rule or just something that I'm observing in my case, but you can always try to substitute a join or two with a subquery.
While I suppose this doesn't really answer your question, I hope it at least gives you an idea about where to start looking for optimisations.