'top' gives:
top - 13:07:35 up 827 days, 8:18, 2 users, load average: 5,26, 5,38, 5,45
KiB Mem: 32848512 total, 32265240 used, 583272 free, 261908 buffers
KiB Swap: 1046520 total, 313784 used, 732736 free, 28190744 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23927 mysql 20 0 917m 243m 8152 S 553,7 0,8 65:09.27 mysqld
mysql> SHOW FULL PROCESSLIST;
+------+--------+-----------------+-------------+---------+------+--------------+--------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+--------+-----------------+-------------+---------+------+--------------+--------------------------------------------------------------------+
| 600 | oneill | localhost | NULL | Query | 0 | NULL | SHOW FULL PROCESSLIST |
| 956 | oneill | 127.0.0.1:42219 | FoxPoker | Sleep | 0 | | NULL |
| 1318 | oneill | 127.0.0.1:44207 | FoxPoker | Sleep | 0 | | NULL |
| 1319 | oneill | 127.0.0.1:44210 | FoxPoker | Sleep | 0 | | NULL |
| 1321 | oneill | 127.0.0.1:44215 | FoxPoker | Sleep | 0 | | NULL |
| 1489 | oneill | 127.0.0.1:45218 | FoxPoker | Sleep | 0 | | NULL |
| 1490 | oneill | 127.0.0.1:45217 | FoxPoker | Sleep | 0 | | NULL |
| 1700 | oneill | 127.0.0.1:46246 | FoxPoker | Sleep | 0 | | NULL |
| 1701 | oneill | 127.0.0.1:46248 | FoxPoker | Sleep | 0 | | NULL |
| 2203 | oneill | 127.0.0.1:49218 | FoxPoker | Sleep | 0 | | NULL |
| 3262 | oneill | 127.0.0.1:54855 | FoxPoker | Sleep | 0 | | NULL |
| 3263 | oneill | 127.0.0.1:54858 | FoxPokerDev | Sleep | 0 | | NULL |
| 3462 | oneill | 127.0.0.1:56029 | FoxPoker | Sleep | 0 | | NULL |
| 4269 | oneill | 127.0.0.1:60551 | FoxPoker | Sleep | 0 | | NULL |
| 4945 | oneill | 127.0.0.1:36032 | FoxPoker | Query | 3 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="pMGmpY" |
| 4950 | oneill | 127.0.0.1:36073 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="tAf994" |
| 4952 | oneill | 127.0.0.1:36083 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="ea6Kv2" |
| 4954 | oneill | 127.0.0.1:36097 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="yFYvur" |
| 4955 | oneill | 127.0.0.1:36098 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="73qTcv" |
| 4956 | oneill | 127.0.0.1:36099 | FoxPoker | Query | 1 | Sending data | SELECT COUNT(deviceId) FROM devices WHERE BINARY deviceId="q64MfG" |
+------+--------+-----------------+-------------+---------+------+--------------+--------------------------------------------------------------------+
mysql> show status like '%onn%';
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| Aborted_connects | 1 |
| Connections | 5528 |
| Max_used_connections | 25 |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_finished_connects | 0 |
| Threads_connected | 19 |
+--------------------------+-------+
7 rows in set (0.00 sec)
(Connections keep on growing.)
mysqladmin status -h localhost -u -p
Uptime: 924 Threads: 21 Questions: 92803 Slow queries: 1677 Opens: 1025 Flush tables: 1 Open tables: 265 Queries per second avg: 100.436
I restared mysql, optimized all tables and nothing changed.
Any idea ?
Login to MySQL and check for expensive queries using:
SHOW FULL PROCESSLIST;
Find the offending sql that's causing the 1600 slow queries and performance tune it.
You can also look to enable slow query logging via the below documentation.
https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html
Related
Can you please explain MySQL 8
performance_schema events_statements_summary_by_digest count_start
?
+------------+----------------+
| COUNT_STAR | SUM_TIMER_WAIT |
+------------+----------------+
| 21562 | 1422617337000 |
| 5134 | 231538954000 |
| 5134 | 42625981791000 |
| 184 | 6224664000 |
| 65 | 67034144000 |
| 48 | 80661283000 |
| 39 | 25631638000 |
| 32 | 1131746000 |
| 32 | 1206939000 |
| 32 | 5997462000 |
| 20 | 2349761000 |
| 8 | 284036000 |
| 8 | 2976134000 |
| 7 | 1254130000 |
| 6 | 1792915000 |
| 6 | 784386000 |
| 6 | 821875000 |
| 6 | 1102248000 |
| 6 | 1079227000 |
| 5 | 11042289000 |
| 5 | 4207319000 |
| 5 | 4012671000 |
| 5 | 6844824000 |
+------------+----------------+
I found this in documentation https://dev.mysql.com/doc/mysql-perfschema-excerpt/8.0/en/wait-summary-tables.html but it's hard to understand for me.
COUNT_STAR
The number of summarized events. This value includes all events, whether timed or nontimed.
Thanks!
MySQL 5.7.17 MGR deploy in single-primary mode, 3 node all one one machine, the same configuration.
And then we test insert on the primary node and observe that the slave nodes have high delay with the write node, even primary node finish the insert test, slave node data is incresing!
Why MySQL Group Replication slave nodes have high delay with write node?
here is the my.cnf:
[mysqld]
datadir=/dba/mysql/data/s1
basedir=/dba/mysql/mysql-5.7/
port=24801
socket=/dba/mysql/data/s1/s1.sock
server_id=1
gtid_mode=ON
enforce_gtid_consistency=ON
master_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE
log_slave_updates=ON
log_bin=binlog
binlog_format=ROW
transaction_write_set_extraction=XXHASH64
loose-group_replication_group_name="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
loose-group_replication_start_on_boot=off
loose-group_replication_local_address= "127.0.0.1:24901"
loose-group_replication_group_seeds= "127.0.0.1:24901,127.0.0.1:24902,127.0.0.1:24903"
loose-group_replication_bootstrap_group= off
loose-group_replication_single_primary_mode=true
loose-group_replication_enforce_update_everywhere_checks=false
slave_parallel_type=LOGICAL_CLOCK
slave_preserve_commit_order=1
slave_parallel_workers=4
and the MGR config :
mysql> show variables like '%group_replication%';
+----------------------------------------------------+-------------------------------------------------+
| Variable_name | Value |
+----------------------------------------------------+-------------------------------------------------+
| group_replication_allow_local_disjoint_gtids_join | OFF |
| group_replication_allow_local_lower_version_join | OFF |
| group_replication_auto_increment_increment | 7 |
| group_replication_bootstrap_group | OFF |
| group_replication_components_stop_timeout | 31536000 |
| group_replication_compression_threshold | 1000000 |
| group_replication_enforce_update_everywhere_checks | OFF |
| group_replication_flow_control_applier_threshold | 25000 |
| group_replication_flow_control_certifier_threshold | 25000 |
| group_replication_flow_control_mode | QUOTA |
| group_replication_force_members | |
| group_replication_group_name | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa |
| group_replication_group_seeds | 127.0.0.1:24901,127.0.0.1:24902,127.0.0.1:24903 |
| group_replication_gtid_assignment_block_size | 1000000 |
| group_replication_ip_whitelist | AUTOMATIC |
| group_replication_local_address | 127.0.0.1:24901 |
| group_replication_poll_spin_loops | 0 |
| group_replication_recovery_complete_at | TRANSACTIONS_APPLIED |
| group_replication_recovery_reconnect_interval | 60 |
| group_replication_recovery_retry_count | 10 |
| group_replication_recovery_ssl_ca | |
| group_replication_recovery_ssl_capath | |
| group_replication_recovery_ssl_cert | |
| group_replication_recovery_ssl_cipher | |
| group_replication_recovery_ssl_crl | |
| group_replication_recovery_ssl_crlpath | |
| group_replication_recovery_ssl_key | |
| group_replication_recovery_ssl_verify_server_cert | OFF |
| group_replication_recovery_use_ssl | OFF |
| group_replication_single_primary_mode | ON |
| group_replication_ssl_mode | DISABLED |
| group_replication_start_on_boot | OFF |
+----------------------------------------------------+-------------------------------------------------+
32 rows in set (0.01 sec)
I have deployed MariaDB Cluster before, and this problem only comes out recently (I don't have this problem before and I don't know why).
I have server 1, 2 and 3. I executed an INSERT command at server 3, however, the tables at server 1 and 2 remains unchanged.
3 servers are at different parts of the world. After the INSERT command, the state uuid remains the same.
Here is the status of server 1:
MariaDB [mysql]> show status like 'wsrep_%';
+------------------------------+----------------------------------------------------------+
| Variable_name | Value |
+------------------------------+----------------------------------------------------------+
| wsrep_local_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_protocol_version | 7 |
| wsrep_last_committed | 205 |
| wsrep_replicated | 170 |
| wsrep_replicated_bytes | 160481 |
| wsrep_repl_keys | 664 |
| wsrep_repl_keys_bytes | 9222 |
| wsrep_repl_data_bytes | 140379 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 46 |
| wsrep_received_bytes | 26150 |
| wsrep_local_commits | 170 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 1 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_cached_downto | 1 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 7.482927 |
| wsrep_apply_oooe | 0.009756 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.009756 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 28 |
| wsrep_causal_reads | 0 |
| wsrep_cert_interval | 0.009756 |
| wsrep_incoming_addresses | server1:3306,server2:3306,server3:3306 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0.200155/0.201113/0.201752/0.000614937/4 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_gcomm_uuid | c4f91b4f-fee1-11e5-8c4f-6e451c332f79 |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 6 |
| wsrep_local_index | 0 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info#codership.com> |
| wsrep_provider_version | 25.3.14(r3560) |
| wsrep_ready | ON |
| wsrep_thread_count | 2 |
+------------------------------+----------------------------------------------------------+
Status of server 2:
MariaDB [(none)]> show status like 'wsrep_%';
+------------------------------+----------------------------------------------------------+
| Variable_name | Value |
+------------------------------+----------------------------------------------------------+
| wsrep_local_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_protocol_version | 7 |
| wsrep_last_committed | 225 |
| wsrep_replicated | 35 |
| wsrep_replicated_bytes | 25700 |
| wsrep_repl_keys | 119 |
| wsrep_repl_keys_bytes | 1757 |
| wsrep_repl_data_bytes | 21703 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 187 |
| wsrep_received_bytes | 177793 |
| wsrep_local_commits | 35 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 1 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_max | 4 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_recv_queue_avg | 0.032086 |
| wsrep_local_cached_downto | 9 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 7.193548 |
| wsrep_apply_oooe | 0.004630 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.004630 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 28 |
| wsrep_causal_reads | 0 |
| wsrep_cert_interval | 0.009217 |
| wsrep_incoming_addresses | server1:3306,server2:3306,server3:3306 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0.200138/0.201917/0.203696/0.00177914/2 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_gcomm_uuid | d562e272-fee1-11e5-b2a2-d3a6b5579aab |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 1 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info#codership.com> |
| wsrep_provider_version | 25.3.14(r3560) |
| wsrep_ready | ON |
| wsrep_thread_count | 2 |
+------------------------------+----------------------------------------------------------+
57 rows in set (0.01 sec)
Status of server3 (As you can see, the latency shows all 0 but I don't know why)
MariaDB [(none)]> show status like 'wsrep_%';
+------------------------------+----------------------------------------------------------+
| Variable_name | Value |
+------------------------------+----------------------------------------------------------+
| wsrep_local_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_protocol_version | 7 |
| wsrep_last_committed | 245 |
| wsrep_replicated | 5 |
| wsrep_replicated_bytes | 4350 |
| wsrep_repl_keys | 11 |
| wsrep_repl_keys_bytes | 203 |
| wsrep_repl_data_bytes | 3827 |
| wsrep_repl_other_bytes | 0 |
| wsrep_received | 226 |
| wsrep_received_bytes | 208559 |
| wsrep_local_commits | 1 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_cached_downto | 19 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 7.022026 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 28 |
| wsrep_causal_reads | 0 |
| wsrep_cert_interval | 0.008811 |
| wsrep_incoming_addresses | server1:3306,server2:3306,server3:3306 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0/0/0/0/0 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_gcomm_uuid | fd022144-fee1-11e5-a7a3-f23274fef9c3 |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | c4f9e2e2-fee1-11e5-8648-a22b867b5a6e |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 2 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info#codership.com> |
| wsrep_provider_version | 25.3.14(r3560) |
| wsrep_ready | ON |
| wsrep_thread_count | 2 |
+------------------------------+----------------------------------------------------------+
57 rows in set (0.00 sec)
iptables at all three servers are set to ACCEPT all input and output traffics.
The log shows that all servers have joined and synced with the cluster.
Does anyone know why? Thanks.
I finally found that is is the app's problem that use MyISAM as storage engine, which causes the error. There is no error after change back to InnoDB.
I suggest you to check the table engines, Because the Galera cluster support InnoDB engine not MyISAM.
So here is a easy way how to migrate Mysql database with MyISAM tables into Galera and InnoDB:
Make sure your db schema doesn't contain FULLTEXT indexes or any other constructions, which are not supported by InnoDB engine
Dump schema of your database
replace in the dump string "MYISAM" with "INNODB"
Dump data
Prepare db user in Galera Cluster (mysql.user table is not replicated across cluster, so you have to insert db user into each of your mariadb servers)
Import schema (with innodb engine)
Import data
Cleanup dump files
Thanks from https://support.qualityunit.com/718375-Migrate-MySQL-Database-with-Myisam-engine-to-MariaDB-Galera-Cluster
I have an issue with a rails 4 application that leaves idle mysql connections. While we look at the code to determine the cause, I at least want to be able to automatically eliminate the offending processes. The mysql documentation says setting wait_timeout and interactive_timeout will close inactive connections so I've set them both to 300. However, as can be seen below, there are various processes that exceed the 300 limit. I've confirmed that the settings are indeed 300. Why isn't mysql eliminating these processes?
mysql> show processlist;
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
| 143 | application | localhost | application | Sleep | 1500 | | NULL |
| 144 | application | localhost | application | Sleep | 1499 | | NULL |
| 145 | application | localhost | application | Sleep | 0 | | NULL |
| 146 | application | localhost | application | Sleep | 780 | | NULL |
| 147 | application | localhost | application | Sleep | 783 | | NULL |
| 148 | application | localhost | application | Sleep | 0 | | NULL |
| 149 | application | localhost | application | Sleep | 7 | | NULL |
| 150 | application | localhost | application | Sleep | 3 | | NULL |
| 151 | application | localhost | application | Sleep | 8 | | NULL |
| 152 | application | localhost | application | Sleep | 3 | | NULL |
| 155 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+-----+--------------+-----------+--------------+---------+------+-------+------------------+
11 rows in set (0.00 sec)
try run mysql query kill 143;
this works for me
I have a highly relational set of tables that I've flattened into two InnoDB tables:
mysql> desc WPropertyCube;
+--------------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| lineOfBusinessId | bigint(20) unsigned | NO | MUL | NULL | |
| txtProperty1 | varchar(125) | YES | | NULL | |
| txtProperty2 | varchar(125) | YES | | NULL | |
| txtProperty3 | varchar(125) | YES | | NULL | |
...
| txtProperty20 | varchar(125) | YES | | NULL | |
| lookupPropertyId1 | bigint(20) unsigned | YES | | NULL | |
| lookupPropertyId2 | bigint(20) unsigned | YES | | NULL | |
| lookupPropertyId3 | bigint(20) unsigned | YES | | NULL | |
...
| lookupPropertyId30 | bigint(20) unsigned | YES | | NULL | |
+--------------------+---------------------+------+-----+---------+----------------+
mysql> show indexes from WPropertyCube;
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| WPropertyCube | 0 | PRIMARY | 1 | id | A | 379383 | NULL | NULL | | BTREE | | |
| WPropertyCube | 1 | WPropertyCube_idx01 | 1 | lineOfBusinessId | A | 204 | NULL | NULL | | BTREE | | |
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> desc WMeasureCube;
+----------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+---------------------+------+-----+---------+-------+
| propertyCubeId | bigint(20) unsigned | NO | PRI | NULL | |
| actionDate | datetime | NO | PRI | NULL | |
| measure1 | decimal(15,6) | YES | | NULL | |
| measure2 | decimal(15,6) | YES | | NULL | |
| measure3 | decimal(15,6) | YES | | NULL | |
...
| measure30 | decimal(15,6) | YES | | NULL | |
+----------------+---------------------+------+-----+---------+-------+
mysql> show indexes from WMeasureCube;
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| WMeasureCube | 0 | PRIMARY | 1 | propertyCubeId | A | 18 | NULL | NULL | | BTREE | | |
| WMeasureCube | 0 | PRIMARY | 2 | actionDate | A | 81680372 | NULL | NULL | | BTREE | | |
| WMeasureCube | 1 | WMeasureCube_idx1 | 1 | actionDate | A | 18 | NULL | NULL | | BTREE | | |
| WMeasureCube | 1 | WMeasureCube_idx1 | 2 | propertyCubeId | A | 81680372 | NULL | NULL | | BTREE | | |
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
You can see I've been playing around with indexes but haven't hit upon the magic combination yet. WPropertyCube has ~100K records, and WMeasureCube has ~80M. My query is this:
mysql> explain SELECT wmc.actionDate,
-> SUM(wmc.measure7),
-> SUM(wmc.measure8),
-> SUM(wmc.measure9)
-> FROM WMeasureCube wmc
-> INNER JOIN WPropertyCube wpc
-> ON wmc.propertyCubeId = wpc.id
-> WHERE wpc.lineOfBusinessId IN ( 1, 2, 3, 4 )
-> AND wmc.actionDate BETWEEN '2010-06-28' AND '2010-09-26'
-> GROUP BY wmc.actionDate
-> ORDER BY wmc.actionDate;
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+
| 1 | SIMPLE | wmc | ALL | PRIMARY,WMeasureCube_idx1 | NULL | NULL | NULL | 81680372 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | wpc | eq_ref | PRIMARY,WPropertyCube_idx01 | PRIMARY | 8 | db.wmc.propertyCubeId | 1 | Using where |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+
This query is aggregating ~30K records into ~1,000 records. Notice that no indices are used to access WMeasureCube, and worse, there's a filesort which is slooooowww. Query time ranges between 30 - 150 seconds. Confoundingly, if I remove the GROUP BY:
mysql> explain SELECT wmc.actionDate,
-> SUM(wmc.measure7),
-> SUM(wmc.measure8),
-> SUM(wmc.measure9)
-> FROM WMeasureCube wmc
-> INNER JOIN WPropertyCube wpc
-> ON wmc.propertyCubeId = wpc.id
-> WHERE wpc.lineOfBusinessId IN ( 1, 2, 3, 4 )
-> AND wmc.actionDate BETWEEN '2010-06-28' AND '2010-09-26'
-> ORDER BY wmc.actionDate;
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+
| 1 | SIMPLE | wmc | ALL | PRIMARY,WMeasureCube_idx1 | NULL | NULL | NULL | 81680372 | Using where |
| 1 | SIMPLE | wpc | eq_ref | PRIMARY,WPropertyCube_idx01 | PRIMARY | 8 | db.wmc.propertyCubeId | 1 | Using where |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+
Still no index use, but at least there's no filesort. This query runs consistently in less than a second. This is especially weird in that ordering by a set of columns should be congruent, in terms of execution time, to grouping by the same set.
What can I do to speed up my query?
I've tried to add all the server variables, but they take up too much space. So, I've added the ones I think will help the most.
mysql> show variables;
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+
| big_tables | OFF |
| binlog_cache_size | 4194304 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_format | STATEMENT |
| bulk_insert_buffer_size | 8388608 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| default_storage_engine | InnoDB |
| innodb_adaptive_flushing | ON |
| innodb_adaptive_hash_index | ON |
| innodb_additional_mem_pool_size | 16777216 |
| innodb_autoextend_increment | 8 |
| innodb_autoinc_lock_mode | 1 |
| innodb_buffer_pool_instances | 1 |
| innodb_buffer_pool_size | 6442450944 |
| innodb_change_buffering | all |
| innodb_checksums | ON |
| innodb_commit_concurrency | 0 |
| innodb_concurrency_tickets | 500 |
| innodb_data_file_path | ibdata1:10M:autoextend |
| innodb_data_home_dir | |
| innodb_doublewrite | ON |
| innodb_fast_shutdown | 1 |
| innodb_file_format | Barracuda |
| innodb_file_format_check | ON |
| innodb_file_format_max | Barracuda |
| innodb_file_per_table | ON |
| innodb_flush_log_at_trx_commit | 0 |
| innodb_flush_method | |
| innodb_force_recovery | 0 |
| innodb_io_capacity | 200 |
| innodb_lock_wait_timeout | 120 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_buffer_size | 16777216 |
| innodb_log_file_size | 268435456 |
| innodb_log_files_in_group | 3 |
| innodb_log_group_home_dir | ./ |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_old_blocks_pct | 37 |
| innodb_old_blocks_time | 0 |
| innodb_open_files | 300 |
| innodb_purge_batch_size | 20 |
| innodb_purge_threads | 0 |
| innodb_read_ahead_threshold | 56 |
| innodb_read_io_threads | 8 |
| innodb_replication_delay | 0 |
| innodb_rollback_on_timeout | OFF |
| innodb_spin_wait_delay | 6 |
| innodb_stats_on_metadata | ON |
| innodb_stats_sample_pages | 8 |
| innodb_strict_mode | OFF |
| innodb_support_xa | OFF |
| innodb_sync_spin_loops | 30 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 0 |
| innodb_thread_sleep_delay | 10000 |
| innodb_use_native_aio | ON |
| innodb_use_sys_malloc | ON |
| innodb_version | 1.1.2 |
| innodb_write_io_threads | 8 |
| join_buffer_size | 8388608 |
| key_buffer_size | 8388608 |
| large_files_support | ON |
| large_page_size | 0 |
| large_pages | OFF |
| last_insert_id | 0 |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| max_heap_table_size | 536870912 |
| max_join_size | 18446744073709551615 |
| max_length_for_sort_data | 1024 |
| max_sort_length | 1024 |
| max_sp_recursion_depth | 0 |
| max_tmp_tables | 32 |
| optimizer_prune_level | 1 |
| optimizer_search_depth | 62 |
| optimizer_switch | index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on |
| preload_buffer_size | 32768 |
| profiling | OFF |
| profiling_history_size | 15 |
| protocol_version | 10 |
| pseudo_thread_id | 13 |
| query_alloc_block_size | 8192 |
| query_cache_limit | 8388608 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 134217728 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 8192 |
| rand_seed1 | 0 |
| rand_seed2 | 0 |
| range_alloc_block_size | 4096 |
| read_buffer_size | 8388608 |
| sort_buffer_size | 16777216 |
| sql_big_selects | ON |
| sql_big_tables | OFF |
| sql_max_join_size | 18446744073709551615 |
| storage_engine | InnoDB |
| tmp_table_size | 536870912 |
| tmpdir | /dev/shm |
| tx_isolation | REPEATABLE-READ |
| version | 5.5.6-rc-log |
| version_comment | MySQL Community Server (GPL) |
| version_compile_machine | x86_64 |
| version_compile_os | linux2.6 |
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+
[EDIT] Thanks to a suggestion by Daniel, I've changed the DATETIME field to DATE. Happily, the index is now being used. Unhappily, we're still filesorting.
+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+
| 1 | SIMPLE | wpc | index | PRIMARY,WPropertyCube_idx01 | WPropertyCube_idx01 | 8 | NULL | 377500 | Using where; Using index; Using temporary; Using filesort |
| 1 | SIMPLE | wmc | ref | PRIMARY,WMeasureCube_idx1 | PRIMARY | 8 | db.wpc.id | 49 | Using where |
+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+
i think your problem is the datetime column actiondate. try to set this column to date column if you dont need the time information! then grouping should be faster and your index should work.