Why is this group by causing a filesort? - mysql

Question

I have a highly relational set of tables that I've flattened into two InnoDB tables:
mysql> desc WPropertyCube;
+--------------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| lineOfBusinessId | bigint(20) unsigned | NO | MUL | NULL | |
| txtProperty1 | varchar(125) | YES | | NULL | |
| txtProperty2 | varchar(125) | YES | | NULL | |
| txtProperty3 | varchar(125) | YES | | NULL | |
...
| txtProperty20 | varchar(125) | YES | | NULL | |
| lookupPropertyId1 | bigint(20) unsigned | YES | | NULL | |
| lookupPropertyId2 | bigint(20) unsigned | YES | | NULL | |
| lookupPropertyId3 | bigint(20) unsigned | YES | | NULL | |
...
| lookupPropertyId30 | bigint(20) unsigned | YES | | NULL | |
+--------------------+---------------------+------+-----+---------+----------------+
mysql> show indexes from WPropertyCube;
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| WPropertyCube | 0 | PRIMARY | 1 | id | A | 379383 | NULL | NULL | | BTREE | | |
| WPropertyCube | 1 | WPropertyCube_idx01 | 1 | lineOfBusinessId | A | 204 | NULL | NULL | | BTREE | | |
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> desc WMeasureCube;
+----------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+---------------------+------+-----+---------+-------+
| propertyCubeId | bigint(20) unsigned | NO | PRI | NULL | |
| actionDate | datetime | NO | PRI | NULL | |
| measure1 | decimal(15,6) | YES | | NULL | |
| measure2 | decimal(15,6) | YES | | NULL | |
| measure3 | decimal(15,6) | YES | | NULL | |
...
| measure30 | decimal(15,6) | YES | | NULL | |
+----------------+---------------------+------+-----+---------+-------+
mysql> show indexes from WMeasureCube;
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| WMeasureCube | 0 | PRIMARY | 1 | propertyCubeId | A | 18 | NULL | NULL | | BTREE | | |
| WMeasureCube | 0 | PRIMARY | 2 | actionDate | A | 81680372 | NULL | NULL | | BTREE | | |
| WMeasureCube | 1 | WMeasureCube_idx1 | 1 | actionDate | A | 18 | NULL | NULL | | BTREE | | |
| WMeasureCube | 1 | WMeasureCube_idx1 | 2 | propertyCubeId | A | 81680372 | NULL | NULL | | BTREE | | |
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
You can see I've been playing around with indexes but haven't hit upon the magic combination yet. WPropertyCube has ~100K records, and WMeasureCube has ~80M. My query is this:
mysql> explain SELECT wmc.actionDate,
-> SUM(wmc.measure7),
-> SUM(wmc.measure8),
-> SUM(wmc.measure9)
-> FROM WMeasureCube wmc
-> INNER JOIN WPropertyCube wpc
-> ON wmc.propertyCubeId = wpc.id
-> WHERE wpc.lineOfBusinessId IN ( 1, 2, 3, 4 )
-> AND wmc.actionDate BETWEEN '2010-06-28' AND '2010-09-26'
-> GROUP BY wmc.actionDate
-> ORDER BY wmc.actionDate;
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+
| 1 | SIMPLE | wmc | ALL | PRIMARY,WMeasureCube_idx1 | NULL | NULL | NULL | 81680372 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | wpc | eq_ref | PRIMARY,WPropertyCube_idx01 | PRIMARY | 8 | db.wmc.propertyCubeId | 1 | Using where |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+
This query is aggregating ~30K records into ~1,000 records. Notice that no indices are used to access WMeasureCube, and worse, there's a filesort which is slooooowww. Query time ranges between 30 - 150 seconds. Confoundingly, if I remove the GROUP BY:
mysql> explain SELECT wmc.actionDate,
-> SUM(wmc.measure7),
-> SUM(wmc.measure8),
-> SUM(wmc.measure9)
-> FROM WMeasureCube wmc
-> INNER JOIN WPropertyCube wpc
-> ON wmc.propertyCubeId = wpc.id
-> WHERE wpc.lineOfBusinessId IN ( 1, 2, 3, 4 )
-> AND wmc.actionDate BETWEEN '2010-06-28' AND '2010-09-26'
-> ORDER BY wmc.actionDate;
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+
| 1 | SIMPLE | wmc | ALL | PRIMARY,WMeasureCube_idx1 | NULL | NULL | NULL | 81680372 | Using where |
| 1 | SIMPLE | wpc | eq_ref | PRIMARY,WPropertyCube_idx01 | PRIMARY | 8 | db.wmc.propertyCubeId | 1 | Using where |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+
Still no index use, but at least there's no filesort. This query runs consistently in less than a second. This is especially weird in that ordering by a set of columns should be congruent, in terms of execution time, to grouping by the same set.
What can I do to speed up my query?
I've tried to add all the server variables, but they take up too much space. So, I've added the ones I think will help the most.
mysql> show variables;
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+
| big_tables | OFF |
| binlog_cache_size | 4194304 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_format | STATEMENT |
| bulk_insert_buffer_size | 8388608 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| default_storage_engine | InnoDB |
| innodb_adaptive_flushing | ON |
| innodb_adaptive_hash_index | ON |
| innodb_additional_mem_pool_size | 16777216 |
| innodb_autoextend_increment | 8 |
| innodb_autoinc_lock_mode | 1 |
| innodb_buffer_pool_instances | 1 |
| innodb_buffer_pool_size | 6442450944 |
| innodb_change_buffering | all |
| innodb_checksums | ON |
| innodb_commit_concurrency | 0 |
| innodb_concurrency_tickets | 500 |
| innodb_data_file_path | ibdata1:10M:autoextend |
| innodb_data_home_dir | |
| innodb_doublewrite | ON |
| innodb_fast_shutdown | 1 |
| innodb_file_format | Barracuda |
| innodb_file_format_check | ON |
| innodb_file_format_max | Barracuda |
| innodb_file_per_table | ON |
| innodb_flush_log_at_trx_commit | 0 |
| innodb_flush_method | |
| innodb_force_recovery | 0 |
| innodb_io_capacity | 200 |
| innodb_lock_wait_timeout | 120 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_buffer_size | 16777216 |
| innodb_log_file_size | 268435456 |
| innodb_log_files_in_group | 3 |
| innodb_log_group_home_dir | ./ |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_old_blocks_pct | 37 |
| innodb_old_blocks_time | 0 |
| innodb_open_files | 300 |
| innodb_purge_batch_size | 20 |
| innodb_purge_threads | 0 |
| innodb_read_ahead_threshold | 56 |
| innodb_read_io_threads | 8 |
| innodb_replication_delay | 0 |
| innodb_rollback_on_timeout | OFF |
| innodb_spin_wait_delay | 6 |
| innodb_stats_on_metadata | ON |
| innodb_stats_sample_pages | 8 |
| innodb_strict_mode | OFF |
| innodb_support_xa | OFF |
| innodb_sync_spin_loops | 30 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 0 |
| innodb_thread_sleep_delay | 10000 |
| innodb_use_native_aio | ON |
| innodb_use_sys_malloc | ON |
| innodb_version | 1.1.2 |
| innodb_write_io_threads | 8 |
| join_buffer_size | 8388608 |
| key_buffer_size | 8388608 |
| large_files_support | ON |
| large_page_size | 0 |
| large_pages | OFF |
| last_insert_id | 0 |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| max_heap_table_size | 536870912 |
| max_join_size | 18446744073709551615 |
| max_length_for_sort_data | 1024 |
| max_sort_length | 1024 |
| max_sp_recursion_depth | 0 |
| max_tmp_tables | 32 |
| optimizer_prune_level | 1 |
| optimizer_search_depth | 62 |
| optimizer_switch | index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on |
| preload_buffer_size | 32768 |
| profiling | OFF |
| profiling_history_size | 15 |
| protocol_version | 10 |
| pseudo_thread_id | 13 |
| query_alloc_block_size | 8192 |
| query_cache_limit | 8388608 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 134217728 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 8192 |
| rand_seed1 | 0 |
| rand_seed2 | 0 |
| range_alloc_block_size | 4096 |
| read_buffer_size | 8388608 |
| sort_buffer_size | 16777216 |
| sql_big_selects | ON |
| sql_big_tables | OFF |
| sql_max_join_size | 18446744073709551615 |
| storage_engine | InnoDB |
| tmp_table_size | 536870912 |
| tmpdir | /dev/shm |
| tx_isolation | REPEATABLE-READ |
| version | 5.5.6-rc-log |
| version_comment | MySQL Community Server (GPL) |
| version_compile_machine | x86_64 |
| version_compile_os | linux2.6 |
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+
[EDIT] Thanks to a suggestion by Daniel, I've changed the DATETIME field to DATE. Happily, the index is now being used. Unhappily, we're still filesorting.
+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+
| 1 | SIMPLE | wpc | index | PRIMARY,WPropertyCube_idx01 | WPropertyCube_idx01 | 8 | NULL | 377500 | Using where; Using index; Using temporary; Using filesort |
| 1 | SIMPLE | wmc | ref | PRIMARY,WMeasureCube_idx1 | PRIMARY | 8 | db.wpc.id | 49 | Using where |
+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+

Answer 1

i think your problem is the datetime column actiondate. try to set this column to date column if you dont need the time information! then grouping should be faster and your index should work.