What can cause mysql performance degradation after move? - mysql

Question

I recently started moving my application from one host to another. From my home computer, to a virtual machine in the cloud. When testing the performance on the new node I noticed severe degradation. Comparing the results of the same query, with the same data, with the same version of mysql.
On my home computer:
mysql> SELECT id FROM events WHERE id in (SELECT distinct event AS id FROM results WHERE status='Inactive') AND (DATEDIFF(NOW(), startdate) < 30) AND (DATEDIFF(NOW(), startdate) > -1) AND status <> 10 AND (form = 'IndSingleDay' OR form = 'IndMultiDay');
+------+
| id |
+------+
| 8238 |
| 8369 |
+------+
2 rows in set (0,57 sec)
and on the new machine:
mysql> SELECT id FROM events WHERE id in (SELECT distinct event AS id FROM results WHERE status='Inactive') AND (DATEDIFF(NOW(), startdate) < 30) AND (DATEDIFF(NOW(), startdate) > -1) AND status <> 10 AND (form = 'IndSingleDay' OR form = 'IndMultiDay');
+------+
| id |
+------+
| 8369 |
+------+
1 row in set (26.70 sec)
Which means 46 times slower. That is not okay. I tried to get an explanation to why it was so slow. For my home computer:
mysql> explain SELECT id FROM events WHERE id in (SELECT distinct event AS id FROM results WHERE status='Inactive') AND (DATEDIFF(NOW(), startdate) < 30) AND (DATEDIFF(NOW(), startdate) > -1) AND status <> 10 AND (form = 'IndSingleDay' OR form = 'IndMultiDay');
+----+--------------+-------------+--------+---------------+------------+---------+-------------------+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------+-------------+--------+---------------+------------+---------+-------------------+---------+-------------+
| 1 | SIMPLE | events | ALL | PRIMARY | NULL | NULL | NULL | 5370 | Using where |
| 1 | SIMPLE | <subquery2> | eq_ref | <auto_key> | <auto_key> | 5 | eventor.events.id | 1 | NULL |
| 2 | MATERIALIZED | results | ALL | idx_event | NULL | NULL | NULL | 1319428 | Using where |
+----+--------------+-------------+--------+---------------+------------+---------+-------------------+---------+-------------+
3 rows in set (0,00 sec)
And for my virtual node:
mysql> explain SELECT id FROM events WHERE id in (SELECT distinct event AS id FROM results WHERE status='Inactive') AND (DATEDIFF(NOW(), startdate) < 30) AND (DATEDIFF(NOW(), startdate) > -1) AND status <> 10 AND (form = 'IndSingleDay' OR form = 'IndMultiDay');
+----+--------------------+---------+----------------+---------------+-----------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+---------+----------------+---------------+-----------+---------+------+------+-------------+
| 1 | PRIMARY | events | ALL | NULL | NULL | NULL | NULL | 7297 | Using where |
| 2 | DEPENDENT SUBQUERY | results | index_subquery | idx_event | idx_event | 5 | func | 199 | Using where |
+----+--------------------+---------+----------------+---------------+-----------+---------+------+------+-------------+
2 rows in set (0.00 sec)
As you can see the results differ. I have not been able to figure out what the difference is. From all other point of views, the two system setups look similar.

Answer 1

In this case, the most likely problem is the processing of the subquery. This changed between some recent versions of MySQL (older versions do a poor job of optimizing the subqueries, the newest version does a better job).
One simple solution is to replace the in with exists and a correlated subquery:
SELECT id
FROM events
WHERE exists (SELECT 1
FROM results
WHERE status='Inactive' and results.event = events.id
) AND
(DATEDIFF(NOW(), startdate) < 30) AND (DATEDIFF(NOW(), startdate) > -1) AND status <> 10 AND (form = 'IndSingleDay' OR form = 'IndMultiDay');
This should work well in both versions, especially if you have an index on results(status, event).

Answer 2

The difference between 5.5 and 5.6 because of the new optimizations for handling subqueries explains (as discussed in comments) the difference in performance, but this conclusion also masks the fact that the original query is not written optimally to begin with. There does not seem to be a need for a subquery here at all.
The "events" table needs an index on (status,form,startdate) and the "results" table needs an index on (status) and another index on (event).
SELECT DISTINCT e.id
FROM events e
JOIN results r ON r.event = e.id AND r.status = 'Inactive'
WHERE (e.form = 'IndSingleDay' OR e.form = 'IndMultiDay')
AND e.status != 10
AND start_date > DATE_SUB(DATE(NOW()), INTERVAL 30 DAY)
AND start_date < DATE_SUB(DATE(NOW()), INTERVAL 2 DAY);
You might have to tweak the values "30" and "2" to get precisely the same logic, but the important principle here is that you never want to use a column as an argument to a function in the WHERE clause if it can be avoided by rewriting the expression another way, because the optimizer can't look "backwards" through the function to discover the actual range of raw values that you are wanting it to find. Instead, it has to evaluate the function against all of the possible data that it can't otherwise eliminate.
Using functions to derive constant values for comparison to the column, as shown above, allows the optimizer to realize that it's actually looking for a range of start_date values, and narrow down the possible rows accordingly, assuming an index exists on the values in question.
If I've decoded your query correctly, this version should be faster than any subquery if the indexes are in place.