MySQL "sending data" activity take longer time - mysql

Question

I recently upgraded MySQL 5.1 to 5.7.8rc.
I have a unique issue with "sending data" status during profiling. It takes more time than expected for every complex or union queries (some time for simple queries also). I tried with best optimized as well as original query. Googled and tried all the possible configurations but no luck.
Query performs supper fast in MySQL 5.1, but not in 5.7.
Also tried with table optimization, analyze, repair etc..
some details for reference:
OS: Centos 6.9 64 bit
MySQL: 5.7.8 rc
CPU: 4
RAM: 64 GB
Data volume: 450 GB
Type: Dedicated VM
Query Profiling:
Status Duration
starting 0.000515
checking permissions 0.000023
checking permissions 0.000016
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000014
checking permissions 0.000016
checking permissions 0.000014
checking permissions 0.000019
Opening tables 0.000079
init 0.000325
System lock 0.000068
optimizing 0.000079
statistics 0.001888
preparing 0.000151
Creating tmp table 0.000128
Sorting result 0.000027
optimizing 0.000034
statistics 0.000064
preparing 0.000047
Creating tmp table 0.000047
Sorting for group 0.000030
optimizing 0.000018
statistics 0.000022
preparing 0.000022
Creating tmp table 0.000043
Sorting for order 0.000023
executing 0.000016
Sending data 4.015596
Creating sort index 0.004766
executing 0.000010
Sending data 0.000159
executing 0.000008
Sending data 0.000025
Creating sort index 0.000349
end 0.000010
query end 0.000024
removing tmp table 0.000013
query end 0.000011
removing tmp table 0.000008
query end 0.000009
removing tmp table 0.000011
query end 0.000009
removing tmp table 0.000007
query end 0.000010
removing tmp table 0.000008
query end 0.000006
closing tables 0.000026
freeing items 0.000039
removing tmp table 0.000009
freeing items 0.000067
logging slow query 0.000047
cleaning up 0.000042
Execution Plan:
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
| 1 | PRIMARY | TIGM_GRP | NULL | index | PRIMARY,FLD_PARENT_GROUP_ID | PRIMARY | 2 | NULL | 87 | 0.33 | Using where |
| 1 | PRIMARY | TIPLD | NULL | range | FLD_PRICE_LEVEL_ID | FLD_PRICE_LEVEL_ID | 3 | NULL | 26 | 10.00 | Using index condition; Using where |
| 1 | PRIMARY | TIPLM | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIPLD.FLD_PRICE_LEVEL_ID | 1 | 10.00 | Using where |
| 1 | PRIMARY | TIGL | NULL | ref | FLD_GROUP_ID,fld_item_id | FLD_GROUP_ID | 2 | TIGM_GRP.FLD_GROUP_ID | 404 | 100.00 | NULL |
| 1 | PRIMARY | TIM | NULL | eq_ref | PRIMARY,FLD_ITEM_TYPE,FLD_ADDON_ID,FLD_PRODUCT_TYPE | PRIMARY | 4 | TIGL.FLD_ITEM_ID | 1 | 50.00 | Using where |
| 1 | PRIMARY | TSIAM | NULL | eq_ref | PRIMARY | PRIMARY | 4 | TIM.FLD_ADDON_ID | 1 | 10.00 | Using where |
| 1 | PRIMARY | TSIARM | NULL | ref | FLD_ADDON_ID | FLD_ADDON_ID | 5 | TIM.FLD_ADDON_ID | 1 | 10.00 | Using index condition; Using where |
| 1 | PRIMARY | TIPD | NULL | ref | FLD_ITEM_ID,FLD_PRICE_LEVEL_ID | FLD_ITEM_ID | 2 | TIM.FLD_ITEM_ID | 3 | 1.35 | Using index condition; Using where |
| 1 | PRIMARY | TIGM_PARENT | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIGM_GRP.FLD_PARENT_GROUP_ID | 1 | 100.00 | Using index |
| 2 | UNION | TIGM_GRP | NULL | index | PRIMARY,FLD_PARENT_GROUP_ID | PRIMARY | 2 | NULL | 87 | 0.06 | Using where |
| 2 | UNION | TIGM_PARENT | NULL | eq_ref | PRIMARY | PRIMARY | 2 | TIGM_GRP.FLD_PARENT_GROUP_ID | 1 | 100.00 | Using index |
|NULL| UNION RESULT | <union1,2> | NULL | ALL | NULL | NULL | NULL | NULL | NULL | NULL | Using temporary; Using filesort |
+----+--------------+-------------+------------+--------+-----------------------------------------------------+--------------------+---------+--------------------------------------+------+----------+------------------------------------+
my.cnf:
[mysql]
# CLIENT #######################################################################
port = 3306
socket = /var/lib/mysql/mysql.sock
[mysqld]
# GENERAL ######################################################################
user = mysql
port = 3306
socket = /var/lib/mysql/mysql.sock
server_id = 32108
default_storage_engine = InnoDB
pid_file = /var/run/mysqld/mysqld.pid
optimizer_prune_level = 0
optimizer_search_depth = 0
max_length_for_sort_data = 8388608 #New
net_buffer_length = 1048576 #New
back_log = 80
symbolic-links = 0
log_bin_trust_function_creators = 1
net_read_timeout = 10 #90
net_write_timeout = 10 #120
net_retry_count = 30
thread_stack = 512K #192K
long_query_time = 10
tmpdir = /tmp
# MyISAM #######################################################################
key_buffer_size = 64M
read_buffer_size = 32M
read_rnd_buffer_size = 32M
bulk_insert_buffer_size = 16M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 1G
myisam_repair_threads = 1
memlock
max_allowed_packet = 32M
max_connect_errors = 100
sql_mode = STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
sysdate-is-now = 1
explicit_defaults_for_timestamp = 1
innodb = FORCE
# Password policy disabled as per communication 05-Sep2017
validate_password = OFF
# DATA STORAGE ##################################################################
datadir = /var/lib/mysql/
# BINARY LOGGING ################################################################
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 14
sync-binlog = 1
# REPLICATION ###################################################################
skip-slave-start = 1
relay-log = /var/log/mysql-realy-logs/relay-bin
slave-skip-errors = 1062 #,1053,1032,1237,1146
slave-net-timeout = 60
relay_log_purge = 1
# CACHES AND LIMITS #############################################################
tmp-table-size = 256M
max-heap-table-size = 256M
query_cache_min_res_unit = 12288 #8192 #New
query-cache-type = 1
query-cache-size = 32M #102400 #0 #256M
max-connections = 150
thread-cache-size = 10 #-1 #Auto resized
open-files-limit = 65535
table_definition_cache = 2000
table_open_cache = 4096 #3092
table_open_cache_instances = 4
sort_buffer_size = 128M
join_buffer_size = 512M
binlog_cache_size = 16M
query_cache_limit = 4M
# INNODB ########################################################################
innodb_fast_shutdown = 1
innodb_flush_method = O_DIRECT
innodb_log_group_home_dir = /mysql_redo_logs/mysql_redo_logs
innodb_log_files_in_group = 2
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 2
innodb_file_per_table = 1 #ON
innodb_buffer_pool_size = 32G #20G
innodb_buffer_pool_instances = 32
innodb_log_buffer_size = 64M
innodb_adaptive_hash_index = 1 #ON
innodb_thread_concurrency = 300 # "0" is default and means infinite (as and when needed). #250 #32
innodb_thread_sleep_delay = 1
innodb_flush_neighbors = 1
innodb_sync_array_size = 832 # Default is 768
skip-innodb_doublewrite #New
innodb_page_cleaners = 32 # Must be =innodb_buffer_pool_instances
innodb_sort_buffer_size = 512M
innodb_read_io_threads = 64
innodb_write_io_threads = 16
#innodb_concurrency_tickets = 429496729
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 10 #80
innodb_compression_level = 0 #New
innodb_lru_scan_depth = 512 #1024 #Default
# LOGGING #######################################################################
log_error = /var/lib/mysql/mysql-error.log
log_queries_not_using_indexes = 1
slow_query_log = 1
log_error_verbosity = 3
[mysqld_safe]
open-files-limit = 4096
malloc-lib = /usr/lib64/libtcmalloc_minimal.so.4
[mysqldump]
quick
max_allowed_packet = 64M
So far I have tried most of the options, currently I started the service in caching mode for quick response.
Can you please help me fix this "sending data" delay issue

Answer 1

Everyone
I think I figured out the issue.
1) 5.6 the optimization engine and plan generated is a result mixture of meta-data and user query (non relational joins). engine trusts user and get best plan.
But 5.7 does not trust soft relations between the tables though they are indexed.
2) 5.7 expects explicit data integrity and do not believe on user.
POC:
I created the same tables involved with hard FK references, wala ! found a miraculous response. the same query which executed without FK took 6.085 Sec (sending data),
and the query with data integrity enforced took 0.05 Sec (with no change in indexing).
So I think 5.7 expects strong data integrity constraints explicitly.

Answer 2

A lot of Optimizer changes happened between 5.1 and 5.7. For one thing, ICP is new; I see it several times in the EXPLAIN. We need to see the query to discuss further. If possible, get the EXPLAIN from 5.1.
Meanwhile, here are some minor comments on my.cnf:
thread_stack = 512K #192K
Generally, it is not useful to increase this setting.
optimizer_prune_level = 0
optimizer_search_depth = 0
What prompted you to set to 0?
innodb = FORCE
deprecated in 5.7.5; suggest removing before it becomes an error.
slave-skip-errors = 1062 #,1053,1032,1237,1146
Sweeping gremlins under the rug -- you will stub your toes on they lump.
innodb_buffer_pool_size = 32G #20G
Unless you have some large apps on the same machine, 44G might be a little better.
innodb_page_cleaners = 32 # Must be =innodb_buffer_pool_instances
re the comment -- Technically, it is auto-limited to pool_instances; see https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_page_cleaners (Thanks, WilsonHauck)
log_queries_not_using_indexes = 1
In my opinion, this clutters the slowlog. The interesting entries are those that exceed long_query_time. Do you have any interesting queries in the slowlog?