tables:
employee
employee_orgn
Joint primary key(employee_id,orgn_id)
two index：key1:employee_id,index2:orgn_id
orgn
Some employee have no organization.
sql:
explain SELECT DISTINCT
e.*
FROM
employee e
LEFT JOIN
employee_orgn eo ON eo.employee_id = e.id
LEFT JOIN
orgn o ON o.id = eo.orgn_id
WHERE
e.state != 'deleted'
AND e.state != 'hidden'
AND (o.state != 'hidden' OR o.state IS NULL)
ORDER BY e.id DESC
explain:
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
| 1 | SIMPLE | e |all | NULL | NULL |NULL | NULL | 12792 |Using where;USing tempory;Using filesort |
| 1 | SIMPLE | eo |index | PRIMARY | idx_orgn_id |8 | NULL | 13226 |Using index:Distinct |
| 1 | SIMPLE | o |eq_ref | PRIMARY | PRIMARY |8 | eo.orgn_id | 1 |Using where:Distinct |
Q：
Here left join, mysql nested loop query 10 orders of magnitude 8?
Why are there temporary tables, and why sorting is file sorting?
Why the second line is the overlay index
I hope someone will explain this explain result and optimize the analysis.
Thanks in advance.
You shouldn't put conditions on tables that are left joined in the WHERE clause. Instead, put them in the ON clause. Then you don't need to use OR o.state IS NULL, which causes optimizer problems.
SELECT DISTINCT
e.*
FROM
employee e
LEFT JOIN
employee_orgn eo ON eo.employee_id = e.id
LEFT JOIN
orgn o ON o.id = eo.orgn_id AND o.state != hidden
WHERE
e.state NOT IN ('deleted', 'hidden')
ORDER BY e.id DESC
I would recommend re-writing the query -- to get rid of the select distinct. I think this is the logic you want:
SELECT e.*
FROM employee e
WHERE e.state not in ('deleted', 'hidden')
NOT EXISTS (SELECT 1
FROM employee_orgn eo JOIN
orgn o
ON o.id = eo.orgn_id AND o.state = 'hidden'
WHERE eo.employee_id = e.id
)
ORDER BY e.id DESC;
For this query, you want an index on employee_orgn(employee_id, orgn_id) and orgn(id, state).
Related
I have tables departments, employees, and emails in MySQL 5.6.17 (for a Rails app). Each department has many employees, and both departments and employees have many emails. I want to sort departments by the number of emails to the entire department and individual employees within the department. My attempt:
SELECT departments.*, COUNT(DISTINCT employees.id) AS employees_count, COUNT(DISTINCT emails.id) AS emails_count
FROM departments
LEFT OUTER JOIN employees
ON employees.department_id = departments.id AND employees.is_employed = true
LEFT OUTER JOIN emails
ON (emails.emailable_id = departments.id AND emails.emailable_type = 'department')
OR (emails.emailable_id = employees.id AND emails.emailable_type = 'employee')
GROUP BY departments.id
ORDER BY emails_count DESC
LIMIT 20;
Unfortunately, this query takes over 3 minutes to complete. Since this query will be used in a web interface, that's not a workable timeframe. An EXPLAIN gives:
+----+-------------+-------------+-------+-------------------------------------------------+----------------------------------+---------+-------------------------------+-------+------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------+-------+-------------------------------------------------+----------------------------------+---------+-------------------------------+-------+------------------------------------------------+
| 1 | SIMPLE | departments | index | PRIMARY | PRIMARY | 4 | NULL | 37468 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | employees | ref | index_employees_on_department_id | index_employees_on_department_id | 5 | development_db.departments.id | 5 | Using where |
| 1 | SIMPLE | emails | ALL | index_emails_on_emailable_id_and_emailable_type | NULL | NULL | NULL | 10278 | Range checked for each record (index map: 0x2) |
+----+-------------+-------------+-------+-------------------------------------------------+----------------------------------+---------+-------------------------------+-------+------------------------------------------------+
The index on emails is, then, not being used. This index is used successfully when I join emails only to departments or only to employees, but not with both at once.
Why is this? What can I do about this? Is there a more efficient way to query the desired data?
It might help to do the aggregation first before the joins:
SELECT d.*, e.employees_count, em.emails_count
FROM d LEFT OUTER JOIN
(SELECT e.department_id, count(*) as employees_count
FROM employees e
WHERE e.is_employed = true
GROUP BY e.department_id
) e
ON e.department_id = d.id LEFT OUTER JOIN
(SELECT department_id, count(distinct id) as emails_count
FROM (SELECT em.emailable_id as department_id, em.id
FROM emails em
WHERE em.emailable_type = 'department'
UNION ALL
SELECT e.department_id, em.id
FROM emails em JOIN
employees e
ON em.emailable_id = e.id AND em.emailable_type = 'employee'
) ee
GROUP BY department_id
) em
ON em.department_id = d.id LEFT OUTER JOIN
ORDER BY emails_count DESC
LIMIT 20;
You also want an index on emails(emailable_id, emailable_type, id) and on emails(emailable_type, emailable_id, id).
I am designing an open schema database with the following table definitions
mysql> desc orders;
+-------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| json | text | NO | | NULL | |
+-------+---------+------+-----+---------+----------------+
mysql> desc ordersnames;
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(330) | NO | UNI | NULL | |
+-------+--------------+------+-----+---------+----------------+
with an index on name
mysql> desc orderskeys;
+-----------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+----------------+
| ID | int(11) | NO | PRI | NULL | auto_increment |
| reference | int(11) | NO | MUL | NULL | |
| nameref | int(11) | NO | MUL | NULL | |
| value | varchar(330) | NO | | NULL | |
+-----------+--------------+------+-----+---------+----------------+
with indices on:
reference,nameref,value
nameref,value
reference
All json fields (1 dimension only) have entry in the orderskeys table per existing field, whereby nameref is a reference to the field name as defined in ordersname.
I would typically query like this:
SELECT
orderskeysdeliveryPostcode.value deliveryPostcode,
orders.ID,
orderskeysCN.value CN
FROM
orders
JOIN ordersnames as ordersnamesCN
on ordersnamesCN.name = 'CN'
JOIN orderskeys as orderskeysCN
on orderskeysCN.nameref = ordersnamesCN.ID
and orderskeysCN.reference = orders.ID
and orderskeysCN.value = '10094'
JOIN ordersnames as ordersnamesdeliveryPostcode
on ordersnamesdeliveryPostcode.name = 'deliveryPostcode'
JOIN orderskeys as orderskeysdeliveryPostcode
on orderskeysdeliveryPostcode.nameref = ordersnamesdeliveryPostcode.ID
and orderskeysdeliveryPostcode.reference = orders.ID
order by deliveryPostcode
limit 0,1000
yielding a result set like this
+------------------+--------+-------+
| deliveryPostcode | ID | CN |
+------------------+--------+-------+
| NULL | 251018 | 10094 |
| NULL | 157153 | 10094 |
| NULL | 95419 | 10094 |
| B-5030 | 172944 | 10094 |
+------------------+--------+-------+
-> lightning fast even with 400k + orders records
However, not all record do contain all fields, so the above query will not yield the records that do not have a 'deliveryPostcode field', so I have to query like this
SELECT
orderskeysdeliveryPostcode.value deliveryPostcode,
orders.ID,
orderskeysCN.value CN
FROM
orders
JOIN ordersnames as ordersnamesCN
on ordersnamesCN.name = 'CN'
JOIN orderskeys as orderskeysCN
on orderskeysCN.nameref = ordersnamesCN.ID
and orderskeysCN.reference = orders.ID
and orderskeysCN.value = '10094'
JOIN ordersnames as ordersnamesdeliveryPostcode
on ordersnamesdeliveryPostcode.name = 'deliveryPostcode'
LEFT JOIN orderskeys as orderskeysdeliveryPostcode
on orderskeysdeliveryPostcode.nameref = ordersnamesdeliveryPostcode.ID
and orderskeysdeliveryPostcode.reference = orders.ID
limit 0,1000
-> equally fast, but as soon as I add an ORDER BY clause on the key value from a left joined table, mysql wants to do the sorting externally (temporary, filesort) instead of using an existing index.
SELECT
orderskeysdeliveryPostcode.value deliveryPostcode,
orders.ID,
orderskeysCN.value CN
FROM
orders
JOIN ordersnames as ordersnamesCN
on ordersnamesCN.name = 'CN'
JOIN orderskeys as orderskeysCN
on orderskeysCN.nameref = ordersnamesCN.ID
and orderskeysCN.reference = orders.ID
and orderskeysCN.value = '10094'
JOIN ordersnames as ordersnamesdeliveryPostcode
on ordersnamesdeliveryPostcode.name = 'deliveryPostcode'
LEFT JOIN orderskeys as orderskeysdeliveryPostcode
on orderskeysdeliveryPostcode.nameref = ordersnamesdeliveryPostcode.ID
and orderskeysdeliveryPostcode.reference = orders.ID
ORDER BY deliveryPostCode
limit 0,1000
-> very slow ...
In fact the sorting operation itself is not much different , as all NULL values for column deliveryPostcode would be at the beginning (ASC) or the end (DESC) while the rest of the dataset would have the same order as with JOIN instead of LEFT JOIN.
How can I query (and order) such tables efficiently? Do I need different relations or indices ?
Much obliged ...
With INNER JOINs, to reduce the number of lookups, MySQL is going to start with the table with the fewest rows (see the EXPLAIN result to see which table MySQL starts with).
If you order by anything other than a column in that first table, or there is no index to satisfy the ORDER BY clause on that first table, MySQL is going to have to do a filesort.
The use of a temporary table is much more likely when text columns are involved, and not just an in-memory temporary table, but a dreadful on-disk temporary table.
Use STRAIGHT_JOIN to force the order that MySQL performs inner joins.
I am not sure what logic do you have in some parts of your query.
I think it still can be optimized.
But just to resolve the issue you have, try just switch it to RIGHT JOIN for now:
SELECT
orderskeysdeliveryPostcode.value deliveryPostcode,
o.id,
o.CN
FROM orderskeys as orderskeysdeliveryPostcode
INNER JOIN ordersnames as ord_n
on ord_n.id = orderskeysdeliveryPostcode.nameref
AND ord_n.name = 'deliveryPostcode'
RIGHT JOIN (
SELECT
orders.ID,
orderskeysCN.CN
FROM
orders
LEFT JOIN
(SELECT
orderskeys.value as CN,
orderskeys.reference
FROM
orderskeys
INNER JOIN ordersnames as ordersnamesCN
ON ordersnamesCN.id = orderskeys.nameref
AND ordersnamesCN.name = 'CN'
WHERE orderskeys.value = '12209'
) as orderskeysCN
ON
orderskeysCN.reference = orders.ID
limit 0,1000
) as o
on
orderskeysdeliveryPostcode.reference = o.ID
ORDER BY deliveryPostCode;
and here is sqlfiddle we can play with. Just need you to add data inserts there.
I am trying to perform a count to get the total number of results in a pagination but the query is too slow 2.12s
+-------+
| size |
+-------+
| 50000 |
+-------+
1 row in set (2.12 sec)
my count query
select count(appeloffre0_.ID_APPEL_OFFRE) as size
from ao.appel_offre appeloffre0_
inner join ao.acheteur acheteur1_
on appeloffre0_.ID_ACHETEUR=acheteur1_.ID_ACHETEUR
where
(exists (select 1 from ao.lot lot2_ where lot2_.ID_APPEL_OFFRE=appeloffre0_.ID_APPEL_OFFRE and lot2_.ESTIMATION_COUT>=1))
and (exists (select 1 from ao.lieu_execution lieuexecut3_ where lieuexecut3_.appel_offre=appeloffre0_.ID_APPEL_OFFRE and lieuexecut3_.region=1))
and (exists (select 1 from ao.ao_activite aoactivite4_ where aoactivite4_.ID_APPEL_OFFRE=appeloffre0_.ID_APPEL_OFFRE and (aoactivite4_.ID_ACTIVITE=1)))
and appeloffre0_.DATE_OUVERTURE_PLIS>'2015-01-01'
and (appeloffre0_.CATEGORIE='fournitures' or appeloffre0_.CATEGORIE='travaux' or appeloffre0_.CATEGORIE='services')
and acheteur1_.ID_ENTITE_MERE=2
explain cmd :
+----+--------------------+--------------+------+---------------------------------------------+--------------------+---------+--------------------------------+-------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+--------------+------+---------------------------------------------+--------------------+---------+--------------------------------+-------+--------------------------+
| 1 | PRIMARY | acheteur1_ | ref | PRIMARY,acheteur_ibfk_1 | acheteur_ibfk_1 | 5 | const | 3 | Using where; Using index |
| 1 | PRIMARY | appeloffre0_ | ref | appel_offre_ibfk_2 | appel_offre_ibfk_2 | 4 | ao.acheteur1_.ID_ACHETEUR | 31061 | Using where |
| 4 | DEPENDENT SUBQUERY | aoactivite4_ | ref | ao_activites_activite_fk,ao_activites_ao_fk | ao_activites_ao_fk | 4 | ao.appeloffre0_.ID_APPEL_OFFRE | 3 | Using where |
| 3 | DEPENDENT SUBQUERY | lieuexecut3_ | ref | fk_ao_lieuex,fk_region_lieuex | fk_ao_lieuex | 4 | ao.appeloffre0_.ID_APPEL_OFFRE | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | lot2_ | ref | FK_LOT_AO | FK_LOT_AO | 4 | ao.appeloffre0_.ID_APPEL_OFFRE | 5 | Using where |
+----+--------------------+--------------+------+---------------------------------------------+--------------------+---------+--------------------------------+-------+--------------------------+
the index acheteur_ibfk_1 is a FK references table ENTITE_MERE because i have and acheteur1_.ID_ENTITE_MERE=2 in where clause.
You can have multiple conditions on your joins by using ON condition1 AND condition2 etc.
SELECT COUNT(appeloffre0_.ID_APPEL_OFFRE) as size
FROM ao.appel_offre appeloffre0_
JOIN ao.acheteur acheteur1_ ON appeloffre0_.ID_ACHETEUR=acheteur1_.ID_ACHETEUR
JOIN ao.lot lot2_ ON appeloffre0_.ID_APPEL_OFFRE=lot2_.ID_APPEL_OFFRE AND lot2_.ESTIMATION_COUT>=1
JOIN ao.lieu_execution lieuexecut3_ ON appeloffre0_.ID_APPEL_OFFRE=lieuexecut3_.ID_APPEL_OFFRE AND lieuexecut3_.ID_ACTIVITE=1
JOIN ao.ao_activite aoactivite4_ ON appeloffre0_.ID_APPEL_OFFRE=aoactivite4_.ID_APPEL_OFFRE AND aoactivite4_.ID_ACTIVITE=1
WHERE appeloffre0_.DATE_OUVERTURE_PLIS>'2015-01-01'
AND (appeloffre0_.CATEGORIE='fournitures' OR appeloffre0_.CATEGORIE='travaux' OR appeloffre0_.CATEGORIE='services')
AND acheteur1_.ID_ENTITE_MERE=2;
You can try:
select count(aa.ID_APPEL_OFFRE) as size
from (
select ID_APPEL_OFFRE, ID_ACHETEUR from ao.appel_offre appeloffre0_
inner join ao.acheteur acheteur1_
on appeloffre0_.ID_ACHETEUR=acheteur1_.ID_ACHETEUR
where appeloffre0_.DATE_OUVERTURE_PLIS>'2015-01-01'
and (appeloffre0_.CATEGORIE in ('fournitures','travaux','services'))
and (acheteur1_.ID_ENTITE_MERE=2)) aa
inner join ao.lot lot2_ on lot2_.ID_APPEL_OFFRE=aa.ID_APPEL_OFFRE
inner join ao.lieu_execution lieuexecut3_ on lieuexecut3_.appel_offre=aa.ID_APPEL_OFFRE
inner join ao.ao_activite aoactivite4_ on aoactivite4_.ID_APPEL_OFFRE=aa.ID_APPEL_OFFRE
where
aoactivite4_.ID_ACTIVITE=1
and lot2_.ESTIMATION_COUT>=1
and lieuexecut3_.region=1;
But I haven't seen your tables so I am not 100% sure that you won't get duplicates because of joins.
A couple of low-hanging fruits might also be found by ensuring that your appeloffre0_.CATEGORIE and appeloffre0_.DATE_OUVERTURE_PLIS have indexes on them.
Other fields which should have indexes on them are ao.lot.ID_APPEL_OFFRE, ao.lieu_execution.ID_APPEL_OFFRE and ao.ao_activite.ID_APPEL_OFFRE, and ao.appel_offre.ID_ACHETEUR (all the joined fields).
I would have the following indexes on your tables if not already... These are covering indexes for your query meaning the index has the applicable column to get your results without having to go to the actual raw data pages.
table index
appel_offre ( DATE_OUVERTURE_PLIS, CATEGORIE, ID_APPEL_OFFRE, ID_ACHETEUR )
lot ( ID_APPEL_OFFRE, ESTIMATION_COUT )
lieu_execution ( appel_offre, region )
ao_activite ( ID_APPEL_OFFRE, ID_ACTIVITE )
Having indexes on just individual columns won't really help optimize what you are looking for. Also, I am doing count of DISTINCT ID_APPEL_OFFRE's in case any of the JOINed tables have more than 1 record, it does not create a Cartesian result count for you
select
count(distinct AOF.ID_APPEL_OFFRE) as size
from
ao.appel_offre AOF
JOIN ao.acheteur ACH
on AOF.ID_ACHETEUR = ACH.ID_ACHETEUR
and ACH.ID_ENTITE_MERE = 2
JOIN ao.lot
ON AOF.ID_APPEL_OFFRE = lot.ID_APPEL_OFFRE
and lot.ESTIMATION_COUT >= 1
JOIN ao.lieu_execution EX
ON AOF.ID_APPEL_OFFRE = EX.appel_offre
and EX.region = 1
JOIN ao.ao_activite ACT
ON AOF.ID_APPEL_OFFRE = ACT.ID_APPEL_OFFRE
and ACT.ID_ACTIVITE = 1
where
AOF.DATE_OUVERTURE_PLIS > '2015-01-01'
and ( AOF.CATEGORIE = 'fournitures'
or AOF.CATEGORIE = 'travaux'
or AOF.CATEGORIE = 'services')
Like #FuzzyTree said in his comment exists is faster than an inner join if it's not a 1:1 relationship because it terminates as soon as it finds 1 whereas the join will get every matching row.
But the solution is that We add in and not exists :
where ( appeloffre0_.ID_APPEL_OFFRE IN (select lot2_.ID_APPEL_OFFRE from ao.lot lot2_
where lot2_.ESTIMATION_COUT>=1)
)
So the query run very fast than exists or joins .
select count(appeloffre0_.ID_APPEL_OFFRE) as size
from ao.appel_offre appeloffre0_
inner join ao.acheteur acheteur1_
on appeloffre0_.ID_ACHETEUR=acheteur1_.ID_ACHETEUR
where
( appeloffre0_.ID_APPEL_OFFRE IN (select lot2_.ID_APPEL_OFFRE from ao.lot lot2_ where lot2_.ESTIMATION_COUT>=1))
and (appeloffre0_.ID_APPEL_OFFRE IN (select lieuexecut3_.appel_offre from ao.lieu_execution lieuexecut3_ where lieuexecut3_.region=1))
and (appeloffre0_.ID_APPEL_OFFRE IN (select aoactivite4_.ID_APPEL_OFFRE from ao.ao_activite aoactivite4_ where aoactivite4_.ID_ACTIVITE=1 ))
and appeloffre0_.DATE_OUVERTURE_PLIS>'2015-01-01'
and (appeloffre0_.CATEGORIE='fournitures' or appeloffre0_.CATEGORIE='travaux' or appeloffre0_.CATEGORIE='services')
and acheteur1_.ID_ENTITE_MERE=2
This query runs more than 12 seconds, even though all tables are relatively small - about 2 thousands rows.
SELECT attr_73206_ AS attr_73270_
FROM object_73130_ f1
LEFT OUTER JOIN (
SELECT id_field, attr_73206_ FROM (
SELECT m.id_field, t0.attr_73102_ AS attr_73206_ FROM object_73200_ o
INNER JOIN master_slave m ON (m.id_object = 73130 OR m.id_object = 73290) AND (m.id_master = 73200 OR m.id_master = 73354) AND m.id_slave_field = o.id
INNER JOIN object_73101_ t0 ON t0.id = o.attr_73206_
ORDER BY o.id_order
) AS o GROUP BY o.id_field
) AS o ON f1.id = o.id_field
Both tables have fields id as primary keys. Besides, id_field, id_order,attr_73206_ and all fields in master_slave are indexed. As for the logic of this query, on the whole it's of master-detail kind. Table object_73130_ is a master-table, table object_73200_ is a detail-table. They are linked by a master_slave table. object_73101_ is an ad-hoc table used to get a real value for the field attr_73206_ by its id. For each row in the master table the query returns a field from the very first row of its detail table. Firstly, the query had another look, but here at stackoverflow I was advised to use this more optimized structure (instead of a subquery which was used previously, and, by the way, the query started to run much faster). I observe that the subquery in the first JOIN block runs very fast but returns a number of rows comparable to the number of rows in the main master-table. In any way, I do not know how to optimize it. I just wonder why a simple fast-running join causes so much trouble. Oh, the main observation is that if I remove an ad-hoc object_73101_ from the query to return just an id, but not a real value, then the query runs as quick as a flash. So, all attention should be focused on this part of the query
INNER JOIN object_73101_ t0 ON t0.id = o.attr_73206_
Why does it slow down the whole query so terribly?
EDIT
In this way it runs super-fast
SELECT t0.attr_73102_ AS attr_73270_
FROM object_73130_ f1
LEFT OUTER JOIN (
SELECT id_field, attr_73206_ FROM (
SELECT m.id_field, attr_73206_ FROM object_73200_ o
INNER JOIN master_slave m ON (m.id_object = 73130 OR m.id_object = 73290) AND (m.id_master = 73200 OR m.id_master = 73354) AND m.id_slave_field = o.id
ORDER BY o.id_order
) AS o GROUP BY o.id_field
) AS o ON f1.id = o.id_field
LEFT JOIN object_73101_ t0 ON t0.id = o.attr_73206_
So, you can see, that I just put the add-hoc join outside of the subquery. But, the problem is, that subquery is automatically created and I have an access to that part of algo which creates it and I can modify this algo, and I do not have access to the part of algo which builds the whole query, so the only thing I can do is just to fix the subquery somehow. Anyway, I still can't understand why INNER JOIN inside a subquery can slow down the whole query hundreds of times.
EDIT
A new version of query with different aliases for each table. This has no effect on the performance:
SELECT attr_73206_ AS attr_73270_
FROM object_73130_ f1
LEFT OUTER JOIN (
SELECT id_field, attr_73206_ FROM (
SELECT m.id_field, t0.attr_73102_ AS attr_73206_ FROM object_73200_ a
INNER JOIN master_slave m ON (m.id_object = 73130 OR m.id_object = 73290) AND (m.id_master = 73200 OR m.id_master = 73354) AND m.id_slave_field = a.id
INNER JOIN object_73101_ t0 ON t0.id = a.attr_73206_
ORDER BY a.id_order
) AS b GROUP BY b.id_field
) AS c ON f1.id = c.id_field
EDIT
This is the result of EXPLAIN command:
| id | select_type | TABLE | TYPE | possible_keys | KEY | key_len | ROWS | Extra |
| 1 | PRIMARY | f1 | INDEX | NULL | PRIMARY | 4 | 1570 | USING INDEX
| 1 | PRIMARY | derived2| ALL | NULL | NULL | NULL | 1564 |
| 2 | DERIVED | derived3| ALL | NULL | NULL | NULL | 1575 | USING TEMPORARY; USING filesort
| 3 | DERIVED | m | RANGE | id_object,id_master,..| id_object | 4 | 1356 | USING WHERE; USING TEMPORARY; USING filesort
| 3 | DERIVED | a | eq_ref | PRIMARY,attr_73206_ | PRIMARY | 4 | 1 |
| 3 | DERIVED | t0 | eq_ref | PRIMARY | PRIMARY | 4 | 1 |
What is wrong with that?
EDIT
Here is the result of EXPLAIN command for the "super-fast" query
| id | select_type | TABLE | TYPE | possible_keys | KEY | key_len | ROWS | Extra
| 1 | PRIMARY | f1 | INDEX | NULL | PRIMARY | 4 | 1570 | USING INDEX
| 1 | PRIMARY | derived2| ALL | NULL | NULL | NULL | 1570 |
| 1 | PRIMARY | t0 | eq_ref| PRIMARY | PRIMARY | 4 | 1 |
| 2 | DERIVED | derived3| ALL | NULL | NULL | NULL | 1581 | USING TEMPORARY; USING filesort
| 3 | DERIVED | m | RANGE | id_object,id_master,| id_bject | 4 | 1356 | USING WHERE; USING TEMPORARY; USING filesort
| 3 | DERIVED | a | eq_ref | PRIMARY | PRIMARY | 4 | 1 |
CLOSED
I will use my own "super-fast" query, which I presented above. I think it is impossible to optimize it anymore.
Without knowing the exact nature of the data/query, there are a couple things that I'm seeing:
MySQL is notoriously bad at handling sub-selects, as it requires the creation of derived tables. In fact, some versions of MySQL also ignore indexes when using sub-selects. Typically, it's better to use JOINs instead of sub-selects, but if you need to use sub-selects, it's best to make that sub-select as lean as possible.
Unless you have a very specific reason for putting the ORDER BY in the sub-select, it may be a good idea to move it to the "main" query portion because the result set may be smaller (allowing for quicker sorting).
So all that being said, I tried to re-write your query using JOIN logic, but I was wondering What table the final value (attr_73102_) is coming from? Is it the result of the sub-select, or is it coming from table object_73130_? If it's coming from the sub-select, then I don't see why you're bothering with the original LEFT JOIN, as you will only be returning the list of values from the sub-select, and NULL for any non-matching rows from object_73130_.
Regardless, not knowing this answer, I think the query below MAY be syntactically equivalent:
SELECT t0.attr_73102_ AS attr_73270_
FROM object_73130_ f1
LEFT JOIN (object_73200_ o
INNER JOIN master_slave m ON m.id_slave_field = o.id
INNER JOIN object_73101_ t0 ON t0.id = o.attr_73206_)
ON f1.id = o.id_field
WHERE m.id_object IN (73130,73290)
AND m.id_master IN (73200,73354)
GROUP BY o.id_field
ORDER BY o.id_order;
I have three tables: categories, articles, and article_events, with the following structure
categories: id, name (100,000 rows)
articles: id, category_id (6000 rows)
article_events: id, article_id, status_id (20,000 rows)
The highest article_events.id for each article row describes the current status of each article.
I'm returning a table of categories and how many articles are in them with a most-recent-event status_id of '1'.
What I have so far works, but is fairly slow (10 seconds) with the size of my tables. Wondering if there's a way to make this faster. All the tables have proper indexes as far as I know.
SELECT c.id,
c.name,
SUM(CASE WHEN e.status_id = 1 THEN 1 ELSE 0 END) article_count
FROM categories c
LEFT JOIN articles a ON a.category_id = c.id
LEFT JOIN (
SELECT article_id, MAX(id) event_id
FROM article_events
GROUP BY article_id
) most_recent ON most_recent.article_id = a.id
LEFT JOIN article_events e ON most_recent.event_id = e.id
GROUP BY c.id
Basically I have to join to the events table twice, since asking for the status_id along with the MAX(id) just returns the first status_id it finds, and not the one associated with the MAX(id) row.
Any way to make this better? or do I just have to live with 10 seconds? Thanks!
Edit:
Here's my EXPLAIN for the query:
ID | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra
---------------------------------------------------------------------------------------------------------------------------
1 | PRIMARY | c | index | NULL | PRIMARY | 4 | NULL | 124044 | Using index; Using temporary; Using filesort
1 | PRIMARY | a | ref | category_id | category_id | 4 | c.id | 3 |
1 | PRIMARY | <derived2> | ALL | NULL | NULL | NULL | NULL | 6351 |
1 | PRIMARY | e | eq_ref | PRIMARY | PRIMARY | 4 | most_recent.event_id | 1 |
2 | DERIVED | article_events | ALL | NULL | NULL | NULL | NULL | 19743 | Using temporary; Using filesort
If you can eliminate subqueries with JOINs, it often performs better because derived tables can't use indexes. Here's your query without subqueries:
SELECT c.id,
c.name,
COUNT(a1.article_id) AS article_count
FROM categories c
LEFT JOIN articles a ON a.category_id = c.id
LEFT JOIN article_events ae1
ON ae1.article_id = a.id
LEFT JOIN article_events ae2
ON ae2.article_id = a.id
AND ae2.id > a1.id
WHERE ae2.id IS NULL
GROUP BY c.id
You'll want to experiment with the indexes and use EXPLAIN to test, but here's my guess (I'm assuming id fields are primary keys and you are using InnoDB):
categories: `name`
articles: `category_id`
article_events: (`article_id`, `id`)
Didn't try it, but I'm thinking this will save a bit of work for the database:
SELECT ae.article_id AS ref_article_id,
MAX(ae.id) event_id,
ae.status_id,
(select a.category_id from articles a where a.id = ref_article_id) AS cat_id,
(select c.name from categories c where c.id = cat_id) AS cat_name
FROM article_events
GROUP BY ae.article_id
Hope that helps
EDIT:
By the way... Keep in mind that joins have to go through each row, so you should start your selection from the small end and work your way up, if you can help it. In this case, the query has to run through 100,000 records, and join each one, then join those 100,000 again, and again, and again, even if values are null, it still has to go through those.
Hope this all helps...
I don't like that index on categories.id is used, as you're selecting the whole table.
Try running:
ANALYZE TABLE categories;
ANALYZE TABLE article_events;
and re-run the query.