Index performance on multiple tables - mysql

Question

I'm experiencing some troubles into making my query faster for production.
The query I want to execute currently takes 12 sec to show the resultset, and it crashes the production server which is ressource restricted.
The point is that I need to get all the enregistrement records when they are the last of the given periode (which is a date as YYYYMM).
After getting these records, I want to sum one of the fields given into I.sum_field as a total field.
When I comment the CASE part, the query takes approx 5sec (+/- 500ms).
Here is the query :
SELECT
I.libelle,
E1.periode,
E1.created_at,
CASE WHEN I.sum_field = 'fat' THEN SUM(E1.Fat)
WHEN I.sum_field = 'etp' THEN SUM(E1.Etp)
WHEN I.sum_field = 'nb_ident' THEN COUNT(*)
WHEN I.sum_field = 'cdi_actif' THEN SUM(E1.cdi_actif)
END AS total
FROM
indicateur_motif IM
INNER JOIN indicateur I
ON IM.indicateur_id = I.id
INNER JOIN `position` P
ON IM.motif_id = P.id
INNER JOIN enregistrement E1
ON P.id = E1.position_id
INNER JOIN
( SELECT
MAX(id) AS id,
MAX(created_at) AS created_at
FROM
enregistrement
WHERE
(etat_mouvement_id IN (1,3,4))
AND (periode >= '201410' AND periode <= '201512')
AND created_at <= DATE_FORMAT('2015-02-03', '%Y-%m-%d %H:%i:%s')
GROUP BY
salarie_id,
periode ) E2
ON E1.id = E2.id
AND E1.created_at = E2.created_at
WHERE
I.formule_id = 1
GROUP BY
I.id,
E1.periode
ORDER BY
I.position,
E1.periode
Here is the EXPLAIN result :
id select_type table type possible_keys key key_len ref rows Extra
------ ----------- -------------- ------ ---------------------------------------------- ---------------------------------------------- ------- ------------------ ------ ----------------------------------------------------
1 PRIMARY I ALL PRIMARY (NULL) (NULL) (NULL) 21 Using where; Using temporary; Using filesort
1 PRIMARY IM ref indicateur_motif_indicateur_id_motif_id_unique indicateur_motif_indicateur_id_motif_id_unique 4 orhase.I.id 2 Using index
1 PRIMARY P eq_ref PRIMARY PRIMARY 4 orhase.IM.motif_id 1 Using index
1 PRIMARY <derived2> ALL (NULL) (NULL) (NULL) (NULL) 165352 Using where; Using join buffer (Block Nested Loop)
1 PRIMARY e1 eq_ref PRIMARY PRIMARY 4 e2.id 1 Using where
2 DERIVED enregistrement index sp sp 771 (NULL) 165352 Using where
Here is a sample of the resultset :
libelle periode created_at total
------------------------------------------ ------- ------------------- ---------
CDI actifs fin de période 201410 2014-10-01 00:00:00 4689
CDI actifs fin de période 201411 2015-01-29 08:12:03 4674
CDI actifs fin de période 201412 2015-01-29 08:12:03 4660
CDI actifs fin de période 201501 2015-01-29 08:12:04 4444
CDI actifs fin de période 201502 2015-01-29 08:12:04 4222
CDI actifs fin de période 201503 2015-01-29 08:12:04 4195
CDI actifs fin de période 201504 2015-01-29 08:12:04 4176
CDI actifs fin de période 201505 2015-01-29 08:12:04 4155
CDI actifs fin de période 201506 2015-01-29 08:12:04 4136
CDI actifs fin de période 201507 2015-01-29 08:12:04 4121
CDI actifs fin de période 201508 2015-01-29 08:12:04 4080
CDI actifs fin de période 201509 2015-01-29 08:12:04 4061
CDI actifs fin de période 201510 2015-01-29 08:12:04 4036
CDI actifs fin de période 201511 2015-01-29 08:12:04 4001
CDI actifs fin de période 201512 2015-01-29 08:12:04 3976
ETP fin de période CDI stock 201410 2014-10-01 00:00:00 4259.16
ETP fin de période CDI stock 201411 2015-01-29 08:12:03 4241.91
ETP fin de période CDI stock 201412 2015-01-29 08:12:03 4222.12
ETP fin de période CDI stock 201501 2015-01-29 08:12:04 4028.07
I just have no idea where to put a new index to avoid this execution time... I've already put one on enregistrement, called sp :
ALTER TABLE enregistrement ADD INDEX sp(salarie_id, periode);
This one makes me get an execution time from 16sec to 12s.
Any ideas ?
Thanks.

Answer 1

Don't know if this will help, but what is your case doing... You are summing totally different fields and counting another into a "Total". I would suspect you might actually want these as their own columns.
However, that being said, what do you have for indexes... Your explain shows some, but I would try to include the following if they are NOT available...
table index
indicateur ( formule_id, id, position )
indicateur_motif ( indicateur_id, motif_id )
`position` ( id )
enregistrement ( position_id, id, created_at ) <-- for the JOIN portion
enregistrement ( etat_mouvement_id, periode, created_at, salarie_id, id ) <-- for sub-select query
Also, from your joins, you are not really using anything from the 'Position' table. Yes, you join from motif to position, position to enreg, but since
IM.motif_id = P.id and P.id = E1.position_id
then you could jump directly
IM.motif_id = E1.position_id
and remove the 'position' table from the query. Here is a slightly revised query to what you started. I removed the position reference, and also changed the "group by" of the inner query so that it might be better performance matching the available index for columns periode, and salarie_id.
SELECT
I.libelle,
E1.periode,
E1.created_at,
CASE WHEN I.sum_field = 'fat' THEN SUM(E1.Fat)
WHEN I.sum_field = 'etp' THEN SUM(E1.Etp)
WHEN I.sum_field = 'nb_ident' THEN COUNT(*)
WHEN I.sum_field = 'cdi_actif' THEN SUM(E1.cdi_actif)
END AS total
FROM
indicateur I
JOIN indicateur_motif IM
ON I.id = IM.indicateur_id
INNER JOIN enregistrement E1
ON IM.motif_id = E1.position_id
INNER JOIN
( SELECT
MAX(id) AS id,
MAX(created_at) AS created_at
FROM
enregistrement
WHERE
etat_mouvement_id IN (1,3,4)
AND periode >= '201410'
AND periode <= '201512'
AND created_at <= '2015-02-03'
GROUP BY
periode,
salarie_id ) E2
ON E1.id = E2.id
AND E1.created_at = E2.created_at
WHERE
I.formule_id = 1
GROUP BY
I.id,
E1.periode
ORDER BY
I.position,
E1.periode

Answer 2

I dont know what your tables look like, but this query:
SELECT MAX(id) AS id, MAX(created_at) AS created_at
FROM enregistrement
WHERE (etat_mouvement_id IN (1,3,4))
AND (periode >= '201410' AND periode <= '201512')
AND created_at <= DATE_FORMAT('2015-02-03', '%Y-%m-%d %H:%i:%s')
GROUP BY salarie_id, periode
is very expensive. If you want to try to fix this solely through indexes, adding indexes to the id and created_at columns might be a good start. The other suggestion I might make is to run this query in a separate transaction, and insert the results into a temp table. That should at least free up some of the required resources by turning it into a simple join rather than a very complex search operation in the middle of your query. If that doesnt work, you could also try running all of the selects and joins without the sums, inserting those results into a temp table, and then selecting and summing the results from there.
That said, without seeing your tables, the number of rows in each and all of the data in each column, what kind of hardware youre running, or having any idea what your prod environment looks like in regards to usage, its really hard to say exactly where the problem might be. I'm pretty sure there is no built-in function in MySQL yet, but profiling the query using something like Jet Profiler might be worthwhile if this is business critical. Seeing exactly where the resource pressure is coming from would be the first thing I would want to do if I were writing a query that is crashing production servers.

Answer 3

your slowness is coming from your sub-select on enregistrement. they are both seem to be table scanning what looks all the records. The IN is also not helping.
try creating indexes on the following table fields and let me know.
enregistrement.etat_mouvement_id
enregistrement.periode
enregistrement.created_at

Answer 4

Here it is. I reduced the execution time from 12s to 6.8s with this query :
SELECT I.libelle, e1.periode,
CASE WHEN I.sum_field = 'fat' THEN SUM(E1.Fat)
WHEN I.sum_field = 'etp' THEN SUM(E1.Etp)
WHEN I.sum_field = 'nb_ident' THEN COUNT(*)
WHEN I.sum_field = 'cdi_actif' THEN SUM(E1.cdi_actif) END AS 'total'
FROM indicateur_motif IM
INNER JOIN indicateur I ON IM.indicateur_id = I.id
INNER JOIN enregistrement e1 ON IM.motif_id = e1.position_id
INNER JOIN
(
SELECT MAX(created_at) AS createdat, salarie_id, periode
FROM enregistrement
WHERE (etat_mouvement_id IN (1,3,4))
AND (periode >= '201410' AND periode <= '201512')
AND created_at <= DATE_FORMAT('2015-02-03', '%Y-%m-%d %H:%i:%s')
GROUP BY salarie_id, periode
) e2 ON (e1.created_at = e2.createdat AND e1.salarie_id = e2.salarie_id AND e1.periode = e2.periode)
WHERE I.formule_id = 1
GROUP BY I.id, e1.periode
ORDER BY I.position, e1.periode
Just for information, this subquery :
SELECT MAX(created_at) AS createdat, salarie_id, periode
FROM enregistrement
WHERE (etat_mouvement_id IN (1,3,4))
AND (periode >= '201410' AND periode <= '201512')
AND created_at <= DATE_FORMAT('2015-02-03', '%Y-%m-%d %H:%i:%s')
GROUP BY salarie_id, periode
Only takes 0.003s to execute, thanks to my sp index :
ALTER TABLE enregistrement ADD INDEX sp(salarie_id, periode);
#DRapp : You were right on my JOINS, I removed position from the joins and corrected the query. On total field, I do want to get the values on a single column, to not to do conditions on my code logic.
I tried #DRapp indexes and query proposition, they just slowed or changed nothing to my query.
id select_type table type possible_keys key key_len ref rows Extra
------ ----------- -------------- ------ ---------------------------------------------- ---------------------------------------------- ------- --------------------------------- ------ ----------------------------------------------------
1 PRIMARY <derived2> ALL (NULL) (NULL) (NULL) (NULL) 165352 Using temporary; Using filesort
1 PRIMARY e1 ref sp sp 771 e2.salarie_id,e2.periode 1 Using where
1 PRIMARY I ALL PRIMARY (NULL) (NULL) (NULL) 21 Using where; Using join buffer (Block Nested Loop)
1 PRIMARY IM eq_ref indicateur_motif_indicateur_id_motif_id_unique indicateur_motif_indicateur_id_motif_id_unique 8 orhase.I.id,orhase.e1.position_id 1 Using index
2 DERIVED enregistrement index sp sp 771 (NULL) 165352 Using where
With this EXPLAIN result, I want to resolve the first line, which describes Using temporary; Using filesort. The solution would be to index the GROUP BY columns, but I dont know if it's possible to create a composite index on these two fields, because they come from different tables. What would be a better or an alternative solution ?
Thanks all for your answers :)