Performance issues on heavy loading with MySQL - mysql

Question

Basically, we have a big production issue where our sql queries takes more than 300 seconds before ending.
We are using Mysql as SQL server, and our tables are with MyISAM.
The problematic queries are basic queries between two tables, and with a sort and group by. The issue is that each table has more than 30 millions rows.
Here is a fake example of this query :
SELECT
i.id_stat as idStat,
SUM(srv0.duree_st) AS dureeAppelTotale,
_date as dateI,
DATE_FORMAT(srv0._date,'%d/%m') AS datekey,
DATE_FORMAT(_date,'%d%m%Y') AS datekeyGroup,
heure,
SUM(srv0.nombre_st) AS nbAppel
FROM ids_stat i , SRV201606 srv0
WHERE i.id_rubrique = srv0.id_rubrique
AND i.id_appli = 24071
AND srv0._date >= '2016-06-01 00:00:00'
AND srv0._date <= '2016-07-01 00:00:00'
GROUP BY id_stat, datekeyGroup, heure
ORDER BY _date
Indexes on the ids_stat table are :
show index from ids_stat;
+----------+------------+---------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------+------------+---------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| ids_stat | 0 | PRIMARY | 1 | id_rubrique | A | 7435537 | NULL | NULL | | BTREE | |
| ids_stat | 1 | id_appli_index | 1 | id_appli | A | 13593 | NULL | NULL | | BTREE | |
| ids_stat | 1 | date_creation_index | 1 | date_creation | A | 2478512 | NULL | NULL | | BTREE | |
on the SRV0 table, indexes are :
show index from SRV201606;
+-----------+------------+----------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-----------+------------+----------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+
| SRV201606 | 0 | PRIMARY | 1 | _date | A | NULL | NULL | NULL | | BTREE | |
| SRV201606 | 0 | PRIMARY | 2 | heure | A | NULL | NULL | NULL | | BTREE | |
| SRV201606 | 0 | PRIMARY | 3 | minute | A | NULL | NULL | NULL | | BTREE | |
| SRV201606 | 0 | PRIMARY | 4 | id_plaque | A | NULL | NULL | NULL | | BTREE | |
| SRV201606 | 0 | PRIMARY | 5 | id_rubrique | A | NULL | NULL | NULL | | BTREE | |
| SRV201606 | 0 | PRIMARY | 6 | id_combinaison | A | 27472765 | NULL | NULL | | BTREE | |
| SRV201606 | 1 | id_rubrique | 1 | id_rubrique | A | NULL | NULL | NULL | | BTREE | |
| SRV201606 | 1 | id_combinaison | 1 | id_combinaison | A | NULL | NULL | NULL | | BTREE | |
Also, here is an explain of the query :
+----+-------------+-------+------+------------------------+----------------+---------+-----------------------------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------------+----------------+---------+-----------------------------+------+---------------------------------+
| 1 | SIMPLE | i | ref | PRIMARY,id_appli_index | id_appli_index | 2 | const | 153 | Using temporary; Using filesort |
| 1 | SIMPLE | srv0 | ref | PRIMARY,id_rubrique | id_rubrique | 8 | StatVoxProdV2.i.id_rubrique | 16 | Using where |
We also read that these old join made with where were harmless as they are transcripted as an inner join by the sql engine.
Can you help us fiding ways to tune our mysql Server so it can be boosted ? Is there any configuration that could help us out ?
Many thanks
[EDIT]
As asked in the comments, here is he version :
SELECT VERSION();
+------------+
| VERSION() |
+------------+
| 5.0.45-log |
Server variables :
show Variables;
+---------------------------------+----------------------------------+
| Variable_name | Value |
+---------------------------------+----------------------------------+
| auto_increment_increment | 1 |
| auto_increment_offset | 1 |
| automatic_sp_privileges | ON |
| back_log | 50 |
| basedir | /usr/local/ |
| binlog_cache_size | 32768 |
| bulk_insert_buffer_size | 8388608 |
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/local/share/mysql/charsets/ |
| collation_connection | utf8_general_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
| completion_type | 0 |
| concurrent_insert | 1 |
| connect_timeout | 5 |
| datadir | THEDATADIR |
| date_format | %Y-%m-%d |
| datetime_format | %Y-%m-%d %H:%i:%s |
| default_week_format | 0 |
| delay_key_write | ON |
| delayed_insert_limit | 100 |
| delayed_insert_timeout | 300 |
| delayed_queue_size | 1000 |
| div_precision_increment | 4 |
| engine_condition_pushdown | OFF |
| expire_logs_days | 32 |
| flush | OFF |
| flush_time | 0 |
| ft_boolean_syntax | + -><()~*:""&| |
| ft_max_word_len | 84 |
| ft_min_word_len | 4 |
| ft_query_expansion_limit | 20 |
| ft_stopword_file | (built-in) |
| group_concat_max_len | 1024 |
| have_archive | NO |
| have_bdb | NO |
| have_blackhole_engine | NO |
| have_compress | YES |
| have_crypt | YES |
| have_csv | NO |
| have_dynamic_loading | YES |
| have_example_engine | NO |
| have_federated_engine | NO |
| have_geometry | YES |
| have_innodb | DISABLED |
| have_isam | NO |
| have_merge_engine | YES |
| have_ndbcluster | NO |
| have_openssl | NO |
| have_ssl | NO |
| have_query_cache | YES |
| have_raid | NO |
| have_rtree_keys | YES |
| have_symlink | YES |
| hostname | THEHOSTNAME |
| init_connect | |
| init_file | |
| init_slave | |
| innodb_additional_mem_pool_size | 1048576 |
| innodb_autoextend_increment | 8 |
| innodb_buffer_pool_awe_mem_mb | 0 |
| innodb_buffer_pool_size | 1048576 |
| innodb_checksums | ON |
| innodb_commit_concurrency | 0 |
| innodb_concurrency_tickets | 500 |
| innodb_data_file_path | |
| innodb_data_home_dir | |
| innodb_doublewrite | ON |
| innodb_fast_shutdown | 1 |
| innodb_file_io_threads | 4 |
| innodb_file_per_table | OFF |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_flush_method | |
| innodb_force_recovery | 0 |
| innodb_lock_wait_timeout | 50 |
| innodb_locks_unsafe_for_binlog | OFF |
| innodb_log_arch_dir | |
| innodb_log_archive | OFF |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 |
| innodb_log_group_home_dir | |
| innodb_max_dirty_pages_pct | 90 |
| innodb_max_purge_lag | 0 |
| innodb_mirrored_log_groups | 1 |
| innodb_open_files | 300 |
| innodb_rollback_on_timeout | OFF |
| innodb_support_xa | ON |
| innodb_sync_spin_loops | 20 |
| innodb_table_locks | ON |
| innodb_thread_concurrency | 8 |
| innodb_thread_sleep_delay | 10000 |
| interactive_timeout | 28800 |
| join_buffer_size | 131072 |
| key_buffer_size | 402653184 |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| language | /usr/local/share/mysql/english/ |
| large_files_support | ON |
| large_page_size | 0 |
| large_pages | OFF |
| lc_time_names | en_US |
| license | GPL |
| local_infile | ON |
| locked_in_memory | OFF |
| log | OFF |
| log_bin | ON |
| log_bin_trust_function_creators | OFF |
| log_error | |
| log_queries_not_using_indexes | OFF |
| log_slave_updates | OFF |
| log_slow_queries | ON |
| log_warnings | 1 |
| long_query_time | 10 |
| low_priority_updates | OFF |
| lower_case_file_system | OFF |
| lower_case_table_names | 0 |
| max_allowed_packet | 1047552 |
| max_binlog_cache_size | 4294967295 |
| max_binlog_size | 104857600 |
| max_connect_errors | 10 |
| max_connections | 100 |
| max_delayed_threads | 20 |
| max_error_count | 64 |
| max_heap_table_size | 134217728 |
| max_insert_delayed_threads | 20 |
| max_join_size | 4294967295 |
| max_length_for_sort_data | 1024 |
| max_prepared_stmt_count | 16382 |
| max_relay_log_size | 0 |
| max_seeks_for_key | 4294967295 |
| max_sort_length | 1024 |
| max_sp_recursion_depth | 0 |
| max_tmp_tables | 32 |
| max_user_connections | 0 |
| max_write_lock_count | 4294967295 |
| multi_range_count | 256 |
| myisam_data_pointer_size | 6 |
| myisam_max_sort_file_size | 2147483647 |
| myisam_recover_options | OFF |
| myisam_repair_threads | 1 |
| myisam_sort_buffer_size | 67108864 |
| myisam_stats_method | nulls_unequal |
| net_buffer_length | 16384 |
| net_read_timeout | 30 |
| net_retry_count | 10 |
| net_write_timeout | 60 |
| new | OFF |
| old_passwords | OFF |
| open_files_limit | 4096 |
| optimizer_prune_level | 1 |
| optimizer_search_depth | 62 |
| pid_file | /usr2/mysql/var/run/mysqld.pid |
| port | 3306 |
| preload_buffer_size | 32768 |
| profiling | OFF |
| profiling_history_size | 15 |
| protocol_version | 10 |
| query_alloc_block_size | 8192 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 33554432 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
| query_prealloc_size | 8192 |
| range_alloc_block_size | 2048 |
| read_buffer_size | 2093056 |
| read_only | OFF |
| read_rnd_buffer_size | 8384512 |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
| rpl_recovery_rank | 0 |
| secure_auth | OFF |
| secure_file_priv | |
| server_id | 2 |
| skip_external_locking | ON |
| skip_networking | OFF |
| skip_show_database | OFF |
| slave_compressed_protocol | OFF |
| slave_load_tmpdir | /usr2/mysql/tmp/ |
| slave_net_timeout | 3600 |
| slave_skip_errors | ALL |
| slave_transaction_retries | 10 |
| slow_launch_time | 2 |
| socket | /usr2/mysql/var/run/mysqld.sock |
| sort_buffer_size | 2097144 |
| sql_big_selects | ON |
| sql_mode | |
| sql_notes | ON |
| sql_warnings | OFF |
| ssl_ca | |
| ssl_capath | |
| ssl_cert | |
| ssl_cipher | |
| ssl_key | |
| storage_engine | MyISAM |
| sync_binlog | 0 |
| sync_frm | ON |
| system_time_zone | CET |
| table_cache | 512 |
| table_lock_wait_timeout | 50 |
| table_type | MyISAM |
| thread_cache_size | 8 |
| thread_stack | 196608 |
| time_format | %H:%i:%s |
| time_zone | SYSTEM |
| timed_mutexes | OFF |
| tmp_table_size | 134217728 |
| tmpdir | /usr2/mysql/tmp/ |
| transaction_alloc_block_size | 8192 |
| transaction_prealloc_size | 4096 |
| tx_isolation | REPEATABLE-READ |
| updatable_views_with_limit | YES |
| version | 5.0.45-log |
| version_comment | Source distribution |
| version_compile_machine | i686 |
| version_compile_os | pc-linux-gnu |
| wait_timeout | 28800 |
ids_stat schema :
SHOW CREATE TABLE ids_stat;
CREATE TABLE `ids_stat` (
`id_rubrique` bigint(20) NOT NULL default '0',
`id_stat` int(10) default NULL,
`id_appli` smallint(5) unsigned NOT NULL default '0',
`nom_rubrique` varchar(240) NOT NULL,
`date_creation` timestamp NOT NULL default CURRENT_TIMESTAMP,
PRIMARY KEY (`id_rubrique`),
KEY `id_appli_index` (`id_appli`),
KEY `date_creation_index` (`date_creation`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
And the SRV0 example schema :
CREATE TABLE `SRV201606` (
`id_rubrique` bigint(20) NOT NULL default '0',
`id_plaque` tinyint(3) unsigned NOT NULL default '0',
`id_combinaison` smallint(5) unsigned NOT NULL default '0',
`_date` date NOT NULL default '0000-00-00',
`heure` tinyint(3) NOT NULL default '0',
`minute` tinyint(3) NOT NULL default '0',
`nombre_entree` mediumint(8) unsigned default '0',
`nombre_st` mediumint(8) unsigned default NULL,
`duree_st` mediumint(8) unsigned default NULL,
`flag` tinyint(3) unsigned NOT NULL default '0',
PRIMARY KEY (`_date`,`heure`,`minute`,`id_plaque`,`id_rubrique`,`id_combinaison`),
KEY `id_rubrique` (`id_rubrique`),
KEY `id_combinaison` (`id_combinaison`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

Answer 1

This is not my finished Answer!,
but can you only run this query on your big tables and give me the execution time and the EXPLAIN of it.
ALTER TABLE SRV201606 ADD KEY idx_rubrique_date (_date,id_rubrique);
SELECT
i.*,
SUM(srv0.duree_st) AS dureeAppelTotale,
srv0._date AS dateI,
DATE_FORMAT(srv0._date,'%d/%m') AS datekey,
DATE_FORMAT(_date,'%d%m%Y') AS datekeyGroup,
heure,
SUM(srv0.nombre_st) AS nbAppel
FROM (
SELECT
id_rubrique
, id_stat
FROM ids_stat
WHERE
id_appli = 24071
GROUP BY id_stat
) AS i
LEFT JOIN SRV201606 srv0 ON i.id_rubrique = srv0.id_rubrique
WHERE
srv0._date >= '2016-06-01'
AND
srv0._date <= '2016-07-01'
GROUP BY id_stat, _date, heure
ORDER BY _date;

Answer 2

You could try adding _date to
CREATE TABLE `SRV201606` (
...
KEY `id_rubrique` (`id_rubrique`),
so
KEY `id_rubrique` (`_date`,`id_rubrique`),
or alternatively
KEY `date-id_rubrique` (`_date`,`id_rubrique`),

Answer 3

Can you try removing the times from the dates and post the EXPLAIN, something like this:-
SELECT
i.id_stat as idStat,
SUM(srv0.duree_st) AS dureeAppelTotale,
_date as dateI,
DATE_FORMAT(srv0._date,'%d/%m') AS datekey,
DATE_FORMAT(_date,'%d%m%Y') AS datekeyGroup,
heure,
SUM(srv0.nombre_st) AS nbAppel
FROM ids_stat i
INNER JOIN SRV201606 srv0
ON i.id_rubrique = srv0.id_rubrique
WHERE i.id_appli = 24071
AND srv0._date BETWEEN '2016-06-01' AND '2016-07-01'
GROUP BY id_stat,
datekeyGroup,
heure
ORDER BY _date
I suspect that MySQL is trying to do a date conversion to a date / time for each row returned.

Answer 4

WHERE i.id_rubrique = srv0.id_rubrique
AND i.id_appli = 24071
AND srv0._date >= '2016-06-01 00:00:00'
AND srv0._date <= '2016-07-01 00:00:00'
Did you want both midnights? Suggest doing this instead:
AND srv0._date >= '2016-06-01'
AND srv0._date < '2016-06-01' + INTERVAL 1 MONTH
Need these indexes to facilitate starting with i:
i: (id_appli, id_rubrique, id_stat) -- in this order; and is 'covering'
srv0: (id_rubrique, _date) -- '=' part first, then 'range'
Need these indexes to facilitate starting with srv0:
i: (id_rubrique, id_appli, id_stat) -- in this order; and is 'covering'
srv0: (_date) -- adding anything after `_date` is useless
Adding all those indexes will give the optimizer the option to start with either table and do it efficiently.