Mysql CPU usage during update operation - mysql

Question

I am facing an issue with high mysql load in particularly whole of the CPU gets consumed. We have 16core cpu's. A job runs which imports certain data in csv format to database. When this starts the CPU goes high and mysql gets hung. Below
INNODB ENGINE STATUS
mysql> show engine innodb status\G;
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
180614 21:27:28 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 27 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1113759 1_second, 1113758 sleeps, 111273 10_second, 1177 background, 1177 flush
srv_master_thread log flush and writes: 1120485
----------
SpgaPHORES
----------
OS WAIT ARRAY INFO: reservation count 6324138, signal count 163497140
Mutex spin waits 3563712357, rounds 2684759736, OS waits 3547098
RW-shared spins 28930883, rounds 72633117, OS waits 977277
RW-excl spins 8651762, rounds 274697595, OS waits 1300580
Spin rounds per wait: 0.75 mutex, 2.51 RW-shared, 31.75 RW-excl
------------------------
LATEST DETECTED DEADLOCK
------------------------
180609 5:00:01
*** (1) TRANSACTION:
TRANSACTION 192A64A2D3, ACTIVE 1 sec starting index read
UPDATE products SET hjmpTS = 80060 ,modifiedTS='2018-06-09 05:00:01' WHERE GK = 8816144056321
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 971922 page no 31354 n bits 136 index `PRIMARY` of table `pgagmcb_prd`.`products` trx id 192A64ACB2 lock_mode X locks rec but not gap
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 19C4CE52B2
Purge done for trx's n:o < 19C4CE1594 undo n:o < 0
History list length 821
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 392730, OS thread handle 0x7f09af186700, query id 5877276140 localhost root
show engine innodb status
---TRANSACTION 19C4CE4A6D, not started
MySQL thread id 392695, OS thread handle 0x7f09c5bef700, query id 5877273919 prgkptomcat02 118.21.1.11 pgag_oauth_prd
---TRANSACTION 19C4CA0181, not started
MySQL thread id 392677, OS thread handle 0x7f09c8a5d700, query id 5876980085 prgkptomcat04 118.21.1.13 pgag_oauth_prd
---TRANSACTION 19C4CE52B1, not started
MySQL thread id 392667, OS thread handle 0x7f09aebf0700, query id 5877276137 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4CDC68D, not started
MySQL thread id 392642, OS thread handle 0x7f11a4161700, query id 5877238586 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4CDBF35, not started
MySQL thread id 392641, OS thread handle 0x7f09af0c3700, query id 5877236562 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4C412C3, not started
MySQL thread id 392586, OS thread handle 0x7f09c5a28700, query id 5876568407 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4C70266, not started
MySQL thread id 392584, OS thread handle 0x7f09ae966700, query id 5876773008 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4CAFCC7, not started
MySQL thread id 392525, OS thread handle 0x7f09c43d0700, query id 5877046769 prgkptomcat02 118.21.1.11 pgag_oauth_prd
---TRANSACTION 19C4CAFD1F, not started
MySQL thread id 392523, OS thread handle 0x7f09c8daa700, query id 5877046862 prgkptomcat02 118.21.1.11 pgag_oauth_prd
---TRANSACTION 19C4CAFD07, not started
MySQL thread id 392524, OS thread handle 0x7f09c5820700, query id 5877046837 prgkptomcat02 118.21.1.11 pgag_oauth_prd
---TRANSACTION 19C4CDE3C3, not started
MySQL thread id 392490, OS thread handle 0x7f09c5d75700, query id 5877246447 118.21.1.14 pgag_oauth_prd
---TRANSACTION 19C4CDCB36, not started
MySQL thread id 392485, OS thread handle 0x7f09aefbf700, query id 5877239893 prgkptomcat03 118.21.1.12 pgagmcb_prd
---TRANSACTION 19C4CE52A3, not started
MySQL thread id 392451, OS thread handle 0x7f09af71c700, query id 5877276127 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4C92CF9, not started
MySQL thread id 392449, OS thread handle 0x7f09c5f7d700, query id 5876922937 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4CE52A6, not started
MySQL thread id 392442, OS thread handle 0x7f09af38e700, query id 5877276129 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4CE4F03, not started
MySQL thread id 392421, OS thread handle 0x7f09af30c700, query id 5877275162 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4C9B259, not started
MySQL thread id 392415, OS thread handle 0x7f09c4597700, query id 5876958837 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4CAE196, not started
MySQL thread id 392402, OS thread handle 0x7f09c8c65700, query id 5877039508 prgkptomcat01 118.21.1.10 pgagmcb_prd
---TRANSACTION 19C4CE52A1, not started
MySQL thread id 392383, OS thread handle 0x7f09aeaab700, query id 5877276126 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4CE52AA, not started
MySQL thread id 392382, OS thread handle 0x7f09c4c72700, query id 5877276132 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4CDBDE9, not started
MySQL thread id 392352, OS thread handle 0x7f09c5082700, query id 5877236203 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4CE5279, not started
MySQL thread id 392350, OS thread handle 0x7f09c4db7700, query id 5877276084 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4CDEAEF, not started
MySQL thread id 392323, OS thread handle 0x7f09c48e4700, query id 5877248380 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4CE52A5, not started
MySQL thread id 392322, OS thread handle 0x7f09c56db700, query id 5877276112 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4CE4BD9, not started
MySQL thread id 392319, OS thread handle 0x7f09c6523700, query id 5877274310 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4CC77FB, not started
MySQL thread id 392297, OS thread handle 0x7f09c538e700, query id 5877148868 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4C70074, not started
MySQL thread id 392296, OS thread handle 0x7f09c8a9e700, query id 5876772482 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4CD042D, not started
MySQL thread id 392295, OS thread handle 0x7f09ae5d8700, query id 5877186230 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4CE2C0A, not started
MySQL thread id 392293, OS thread handle 0x7f09c5aeb700, query id 5877265779 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4CE3571, not started
MySQL thread id 392283, OS thread handle 0x7f09c6627700, query id 5877268313 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4CC96A6, not started
MySQL thread id 392282, OS thread handle 0x7f09c40c4700, query id 5877156935 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4C930C7, not started
MySQL thread id 392281, OS thread handle 0x7f09c8e2c700, query id 5876923943 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4C6F0A5, not started
MySQL thread id 392278, OS thread handle 0x7f09c569a700, query id 5876768228 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4CC399F, not started
MySQL thread id 392276, OS thread handle 0x7f09c471d700, query id 5877132036 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4CE52A0, not started
MySQL thread id 392234, OS thread handle 0x7f09c5249700, query id 5877276138 118.21.1.14 pgagmcb_prd Opening tables
SELECT item_t0.GK FROM cartentries item_t0 WHERE ( item_t0.p_order =8995720658987) AND (item_t0.TypeGKString=8796094496850 ) order by item_t0.EntryNumber ASC, item_t0.createdTS ASC
---TRANSACTION 19C4CE369D, not started
MySQL thread id 392232, OS thread handle 0x7f09c8be3700, query id 5877268619 prgkptomcat04 118.21.1.13 pgagmcb_prd
---TRANSACTION 19C4CAE1B0, not started
MySQL thread id 392136, OS thread handle 0x7f09c4d76700, query id 5877039534 prgkptomcat01 118.21.1.10 pgagmcb_prd
---TRANSACTION 19C4CAE1AA, not started
MySQL thread id 392135, OS thread handle 0x7f09c4fbf700, query id 5877039527 prgkptomcat01 118.21.1.10 pgagmcb_prd
---TRANSACTION 19C4CE5272, not started
MySQL thread id 392114, OS thread handle 0x7f09c50c3700, query id 5877276077 prgkptomcat01 118.21.1.10 pgagmcb_prd
---TRANSACTION 19C4CE32D8, not started
MySQL thread id 392027, OS thread handle 0x7f11a409e700, query id 5877267604 prgkptomcat03 118.21.1.12 pgagmcb_prd
---TRANSACTION 19C4CDBDEE, not started
MySQL thread id 391944, OS thread handle 0x7f11a4224700, query id 5877236208 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4CA0332, not started
MySQL thread id 391874, OS thread handle 0x7f09c5aaa700, query id 5876980541 prgkptomcat04 118.21.1.13 pgag_oauth_prd
---TRANSACTION 19C4CE50B6, not started
MySQL thread id 391814, OS thread handle 0x7f09c5eba700, query id 5877275617 prgkptomcat02 118.21.1.11 pgagmcb_prd
---TRANSACTION 19C4CE1588, not started
MySQL thread id 391776, OS thread handle 0x7f09c5d34700, query id 5877259853 118.21.1.14 pgagmcb_prd
---TRANSACTION 19C4CE52AF, not started
mysql tables in use 1, locked 1
MySQL thread id 391692, OS thread handle 0x7f11a40df700, query id 5877276141 118.21.1.14 pgagmcb_prd init
UPDATE cronjobs SET hjmpTS = 8629763 ,modifiedTS='2018-06-14 21:27:28',p_endtime='2018-06-14 21:27:28' WHERE GK = 8811756159477
Trx read view will not see trx with id >= 19C4CE527F, sees < 19C4CE4C4D
---TRANSACTION 19C4CE524E, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 0
MySQL thread id 392064, OS thread handle 0x7f09afb2c700, query id 5877276038 prgkptomcat01 118.21.1.10 pgagmcb_prd Sending data
MySQL thread id 392062, OS thread handle 0x7f09c55d7700, query id 5877275109 prgkptomcat01 118.21.1.10 pgagmcb_prd Sending data
SELECT item_t0.GK FROM addresses item_t0 WHERE ( item_t0.p_pkid ='29001499' AND item_t0.p_billingaddress =0 AND item_t0.p_shippingaddress =1) AND (item_t0.TypeGKString=8796094103634 ) UNION ALL SELECT item_t0.GK FROM pointofserviceaddress item_t0 WHERE ( item_t0.p_pkid ='29001499' AND item_t0.p_billingaddress =0 AND item_t0.p_shippingaddress =1) AND (item_t0.TypeGKString=8797338533970 )
Trx read view will not see trx with id >= 19C4CE4ECF, sees < 19C4CE4A1C
---TRANSACTION 19C4CE4EC4, ACTIVE 0 sec
mysql tables in use 2, locked 0
MySQL thread id 392067, OS thread handle 0x7f09affff700, query id 5877275099 prgkptomcat01 118.21.1.10 pgagmcb_prd Sending data
h id >= 19C4CE4C62, sees < 19C4CE48E5
---TRANSACTION 19C4CE4C4D, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 0
MySQL thread id 392059, OS thread handle 0x7f09c430d700, query id 5877274440 prgkptomcat01 118.21.1.10 emagmcb_prd Sending data
SELECT item_t0.GK FROM addresses item_t0 WHERE ( item_t0.p_pkid ='20087352' AND item_t0.p_billingaddress =0 AND item_t0.p_shippingaddress =1) AND (item_t0.TypeGKString=8796094103634 ) UNION ALL SELECT item_t0.GK FROM pointofserviceaddress item_t0 WHERE ( item_t0.p_pkid ='20087352' AND item_t0.p_billingaddress =0 AND item_t0.p_shippingaddress =1) AND (item_t0.TypeGKString=8797338533970 )
Trx read view will not see trx with id >= 19C4CE4C4E, sees < 19C4CE48E5
--------
FILE I/O
--------
I/O thread 0 state: waiting for i/o request (insert buffer thread)
I/O thread 1 state: waiting for i/o request (log thread)
I/O thread 2 state: waiting for i/o request (read thread)
I/O thread 3 state: waiting for i/o request (read thread)
I/O thread 4 state: waiting for i/o request (read thread)
I/O thread 5 state: waiting for i/o request (read thread)
I/O thread 6 state: waiting for i/o request (write thread)
I/O thread 7 state: waiting for i/o request (write thread)
I/O thread 8 state: waiting for i/o request (write thread)
I/O thread 9 state: waiting for i/o request (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
689458 OS file reads, 21884164 OS file writes, 5163807 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 22.07 writes/s, 5.22 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 156723, seg size 156725, 39554 merges
merged operations:
insert 106864, delete mark 147895, delete 47175
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 63749393, node heap has 66107 buffer(s)
11918.00 hash searches/s, 20228.44 non-hash searches/s
---
LOG
---
Log sequence number 3872974342573
Log flushed up to 3872974307599
Last checkpoint at 3872972346157
0 pending log writes, 0 pending chkp writes
1520749 log i/o's done, 1.33 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 32967229440; in additional pool allocated 0
Dictionary memory allocated 5343676
Buffer pool size 1966080
Free buffers 901470
Database pages 998503
Old database pages 368568
Modified db pages 1090
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 4341, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 889753, created 108750, written 32815581
0.00 reads/s, 0.15 creates/s, 30.44 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 998503, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
22 read views open inside InnoDB
Main thread process no. 11805, id 139679969232640, state: sleeping
Number of rows inserted 4605702, updated 110022489, deleted 3897802, read 309526373829
0.81 inserts/s, 49.85 updates/s, 0.56 deletes/s, 1280085.96 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
[client]
port = 3306
socket = /AB/mysql/logs/mysql.sock
[mysqld]
port = 3306
bind-address = 0.0.0.0
socket = /AB/mysql/logs/mysql.sock
basedir = /AB/mysql
user = mysql
tmpdir = /AB/mysql_temp_files
character-set-server = utf8
datadir = /AB/mysql/data
default-storage-engine = InnoDB
expire_logs_days = 10
general_log = 0
general_log_file = /AB/mysql/logs/general-log.log
innodb_buffer_pool_size = 30G
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 0
key_buffer_size = 16M
log_error = /AB/mysql/logs/mysqld-error.log
log_queries_not_using_indexes = 1
long_query_time = 5
log-bin = /AB/mysql/binlog/mysql-bin
max_allowed_packet = 35M
max_binlog_size = 100M
max_connect_errors = 100000
max_connections = 910
myisam_recover = BACKUP
query_cache_limit = 2M
query_cache_size = 0
query_cache_type = 0
skip-external-locking
slow_query_log = 0
slow_query_log_file = /AB/mysql/logs/mysqld-slow-queries.log
table_open_cache = 256
thread_cache_size = 8
thread_stack = 256K
transaction-isolation = READ-COMMITTED
############################
# MySql replication - Master
############################
server_id = 1
binlog_format = row
binlog-ignore-db=mysql # input the database that should be ignored for replication
binlog-ignore-db=performance_schema
binlog-ignore-db=pkgmcb_eldas
replicate-wild-ignore-table=mysql.% #disable slave replication for mysql db
relay-log = /AB/mysql/mysql-relay-bin
relay-log-index = /AB/mysql/mysql-relay-bin.index
master-info-file = /AB/mysql/mysql-master.info
relay-log-info-file = /AB/mysql/mysql-relay-log-info
[mysqld_safe]
log-error = /AB/mysql/logs/mysqld.log
socket = /AB/mysql/logs/mysql.sock
pid-file = /AB/mysql/logs/mysqld.pid
[mysqldump]
max_allowed_packet = 16M
quick
[mysql]
no_auto_rehash
mysql> show global status;
+------------------------------------------+---------------+
| Variable_name | Value |
+------------------------------------------+---------------+
| Aborted_clients | 13 |
| Aborted_connects | 3753 |
| Binlog_cache_disk_use | 32815 |
| Binlog_cache_use | 92017182 |
| Bytes_received | 1617086123131 |
| Bytes_sent | 2825568326969 |
| Com_admin_commands | 37462 |
| Com_change_db | 2 |
| Com_commit | 128435552 |
| Com_create_table | 18730 |
| Com_delete | 15234374 |
| Com_delete_multi | 37450 |
| Com_drop_table | 18725 |
| Com_insert | 4603989 |
| Com_insert_select | 18725 |
| Com_purge_before_date | 13 |
| Com_rollback | 15159 |
| Com_select | 5360296742 |
| Com_set_option | 257428949 |
| Com_show_charsets | 4 |
| Com_show_collations | 6371 |
| Com_show_engine_status | 4870 |
| Com_show_processlist | 1187 |
| Com_show_status | 95630 |
| Com_show_tables | 21 |
| Com_show_triggers | 0 |
| Com_show_variables | 21400 |
| Com_update | 110599528 |
| Compression | OFF |
| Connections | 392731 |
| Created_tmp_disk_tables | 77030 |
| Created_tmp_files | 445171 |
| Created_tmp_tables | 665507495 |
| Delayed_errors | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Flush_commands | 1 |
| Handler_commit | 5774785433 |
| Handler_delete | 3897802 |
| Handler_discover | 0 |
| Handler_prepare | 402020246 |
| Handler_read_first | 161347921 |
| Handler_read_key | 32213287627 |
| Handler_read_last | 2 |
| Handler_read_next | 47210732684 |
| Handler_read_prev | 32726 |
| Handler_read_rnd | 215354676 |
| Handler_read_rnd_next | 482467384071 |
| Handler_rollback | 170 |
| Handler_savepoint | 0 |
| Handler_savepoint_rollback | 0 |
| Handler_update | 221011669 |
| Handler_write | 21797614285 |
| Innodb_buffer_pool_pages_data | 998503 |
| Innodb_buffer_pool_bytes_data | 16359473152 |
| Innodb_buffer_pool_pages_dirty | 1029 |
| Innodb_buffer_pool_bytes_dirty | 16859136 |
| Innodb_buffer_pool_pages_flushed | 32815553 |
| Innodb_buffer_pool_pages_free | 901470 |
| Innodb_buffer_pool_pages_misc | 66107 |
| Innodb_buffer_pool_pages_total | 1966080 |
| Innodb_buffer_pool_read_ahead_rnd | 0 |
| Innodb_buffer_pool_read_ahead | 207758 |
| Innodb_buffer_pool_read_ahead_evicted | 0 |
| Innodb_buffer_pool_read_requests | 390826699904 |
| Innodb_buffer_pool_reads | 645677 |
| Innodb_buffer_pool_wait_free | 0 |
| Innodb_buffer_pool_write_requests | 887071687 |
| Innodb_data_fsyncs | 5163787 |
| Innodb_data_pending_fsyncs | 0 |
| Innodb_data_pending_reads | 0 |
| Innodb_data_pending_writes | 0 |
| Innodb_data_read | 14588825600 |
| Innodb_data_reads | 689458 |
| Innodb_data_writes | 21884132 |
| Innodb_data_written | 1155002425856 |
| Innodb_dblwr_pages_written | 32815553 |
| Innodb_dblwr_writes | 918315 |
| Innodb_have_atomic_builtins | ON |
| Innodb_log_waits | 0 |
| Innodb_log_write_requests | 234974909 |
| Innodb_log_writes | 1390975 |
| Innodb_os_log_fsyncs | 1505727 |
| Innodb_os_log_pending_fsyncs | 0 |
| Innodb_os_log_pending_writes | 0 |
| Innodb_os_log_written | 79636049408 |
| Innodb_page_size | 16384 |
| Innodb_pages_created | 108750 |
| Innodb_pages_read | 889753 |
| Innodb_pages_written | 32815553 |
| Innodb_row_lock_current_waits | 0 |
| Innodb_row_lock_time | 195660 |
| Innodb_row_lock_time_avg | 0 |
| Innodb_row_lock_time_max | 2243 |
| Innodb_row_lock_waits | 1286068 |
| Innodb_rows_deleted | 3897802 |
| Innodb_rows_inserted | 4605700 |
| Innodb_rows_read | 309520074518 |
| Innodb_rows_updated | 110022225 |
| Innodb_truncated_status_writes | 0 |
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 13396 |
| Key_blocks_used | 5921 |
| Key_read_requests | 1782635934 |
| Key_reads | 0 |
| Key_write_requests | 292962271 |
| Key_writes | 0 |
| Last_query_cost | 0.000000 |
| Max_used_connections | 266 |
| Not_flushed_delayed_rows | 0 |
| Open_files | 5 |
| Open_streams | 0 |
| Open_table_definitions | 400 |
| Open_tables | 256 |
| Opened_files | 959559 |
| Opened_table_definitions | 43623 |
| Opened_tables | 21935444 |
| Performance_schema_cond_classes_lost | 0 |
| Performance_schema_cond_instances_lost | 0 |
| Performance_schema_file_classes_lost | 0 |
| Performance_schema_file_handles_lost | 0 |
| Performance_schema_file_instances_lost | 0 |
| Performance_schema_locker_lost | 0 |
| Performance_schema_mutex_classes_lost | 0 |
| Performance_schema_mutex_instances_lost | 0 |
| Performance_schema_rwlock_classes_lost | 0 |
| Performance_schema_rwlock_instances_lost | 0 |
| Performance_schema_table_handles_lost | 0 |
| Performance_schema_table_instances_lost | 0 |
| Performance_schema_thread_classes_lost | 0 |
| Performance_schema_thread_instances_lost | 0 |
| Prepared_stmt_count | 0 |
| Qcache_free_blocks | 0 |
| Qcache_free_memory | 0 |
| Qcache_hits | 0 |
| Qcache_inserts | 0 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 0 |
| Qcache_queries_in_cache | 0 |
| Qcache_total_blocks | 0 |
| Queries | 5877263773 |
| Questions | 5877218009 |
| Rpl_status | AUTH_MASTER |
| Select_full_join | 1085194 |
| Select_full_range_join | 0 |
| Select_range | 106321329 |
| Select_range_check | 0 |
| Select_scan | 491596929 |
| Slave_heartbeat_period | 0.000 |
| Slave_open_temp_tables | 0 |
| Slave_received_heartbeats | 0 |
| Slave_retried_transactions | 0 |
| Slave_running | OFF |
| Slow_launch_threads | 0 |
| Slow_queries | 407546261 |
| Sort_merge_passes | 631000 |
| Sort_range | 4293576025 |
| Sort_rows | 17877617809 |
| Sort_scan | 13049200 |
| Table_locks_immediate | 7035027122 |
| Table_locks_waited | 0 |
| Tc_log_max_pages_used | 0 |
| Tc_log_page_size | 0 |
| Tc_log_page_waits | 7 |
| Threads_cached | 5 |
| Threads_connected | 102 |
| Threads_created | 7052 |
| Threads_running | 21 |
| Uptime | 1123494 |
| Uptime_since_flush_status | 1123494 |
+------------------------------------------+---------------+
289 rows in set (0.00 sec)
Any help is really appreciated. Thanks

Answer 1

Analysis of Global Status and Variables:
Observations:
Version: 5.5.47-log
30 GB of RAM
Uptime = 13d 00:04:54
You are not running on Windows.
Running 64-bit version
You appear to be running entirely (or mostly) InnoDB.
The More Important Issues:
What did you mean by "30G assigned for mysql"? I see that the buffer_pool is set to that. That is reasonable if you have more than 30G of RAM available for all of MySQL. If the 40GB machine is mostly dedicated to MySQL, then 30G for buffer_pool is good.
Time to upgrade to 5.6. The table_cache is being hit rather hard, but table_open_cache_hits and _misses are not available until 5.6. Meanwhile, see if this helps:
table_open_cache = 500
InnoDB's log is churning like mad. Recommend the following. Caution -- it is complex to change:
innodb_log_file_size = 128M
What are you doing to cause so many rows to be deleted? Perhaps you are "reloading" a table? There are faster ways than using DELETE. Tell us what is up so we can tailor a solution.
There are lots of clues of frequent and/or slow queries. Please turn on the slowlog and have the following
log_queries_not_using_indexes = OFF -- ON is just clutter
Don't decrease long_query_time until you have fixed some of the 362 slow queries/sec.
After a day, let's summarize and analyze what is in the slowlog.
Creating and dropping table once a minute? What's up?
tx_isolation = READ-COMMITTED -- What's up?
These will help some:
innodb_buffer_pool_instances = 16
thread_cache_size = 20
join_buffer_size = 256K
Details and other observations:
( Key_blocks_used * 1024 / key_buffer_size ) = 5,921 * 1024 / 16M = 36.1% -- Percent of key_buffer used. High-water-mark.
-- Lower key_buffer_size to avoid unnecessary memory usage.
( innodb_buffer_pool_size / _ram ) = 30720M / 30720M = 100.0% -- % of RAM used for InnoDB buffer_pool
( (key_buffer_size / 0.20 + innodb_buffer_pool_size / 0.70) / _ram ) = (16M / 0.20 + 30720M / 0.70) / 30720M = 143.1% -- Most of available ram should be made available for caching.
-- http://mysql.rjweb.org/doc.php/memory
( Opened_tables ) = 21,935,444 / 1123494 = 20 /sec -- Frequency of opening Tables
-- increase table_open_cache
( innodb_buffer_pool_size / innodb_buffer_pool_instances ) = 30720M / 1 = 30720MB -- Size of each buffer_pool instance.
-- An instance should be at least 1GB. In very large RAM, have 16 instances.
( Innodb_buffer_pool_pages_free / Innodb_buffer_pool_pages_total ) = 901,470 / 1966080 = 45.9% -- Pct of buffer_pool currently not in use
-- innodb_buffer_pool_size is bigger than necessary?
( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 79,636,049,408 / (1123494 / 3600) / 2 / 5M = 24.3 -- Ratio
-- (see minutes)
( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 1,123,494 / 60 * 5M / 79636049408 = 1.23 -- Minutes between InnoDB log rotations Beginning with 5.6.8, this can be changed dynamically; be sure to also change my.cnf.
-- (The recommendation of 60 minutes between rotations is somewhat arbitrary.) Adjust innodb_log_file_size. (Cannot change in AWS.)
( Innodb_rows_deleted / Innodb_rows_inserted ) = 3,897,802 / 4605700 = 0.846 -- Churn
-- "Don't queue it, just do it." (If MySQL is being used as a queue.)
( Innodb_row_lock_waits ) = 1,286,068 / 1123494 = 1.1 /sec -- How often there is a delay in getting a row lock.
-- May be caused by complex queries that could be optimized.
( innodb_stats_on_metadata ) = ON -- Re-analyze table when touching stats.
-- ON is likely to slow down certain SHOWs and information_schema accesses.
( innodb_thread_concurrency ) = 0 -- 0 = Let InnoDB decide the best for concurrency_tickets.
-- Set to 0 or 64. This may cut back on CPU.
( innodb_print_all_deadlocks ) = OFF -- Whether to log all Deadlocks.
-- If you are plagued with Deadlocks, turn this on. Caution: If you have lots of deadlocks, this may write a lot to disk.
( join_buffer_size / _ram ) = 131,072 / 30720M = 0.00% -- 0-N per thread. May speed up JOINs (better to fix queries/indexes) (all engines) Used for index scan, range index scan, full table scan, each full JOIN, etc.
-- If large, decrease join_buffer_size to avoid memory pressure. Suggest less than 1% of RAM. If small, increase to 0.01% of RAM to improve some queries.
( query_prealloc_size / _ram ) = 8,192 / 30720M = 0.00% -- For parsing. Pct of RAM
( query_alloc_block_size / _ram ) = 8,192 / 30720M = 0.00% -- For parsing. Pct of RAM
( net_buffer_length / max_allowed_packet ) = 16,384 / 35M = 0.04%
( local_infile ) = ON
-- local_infile = ON is a potential security issue
( Questions ) = 5,877,218,009 / 1123494 = 5231 /sec -- Queries (outside SP) -- "qps"
-- >2000 may be stressing server
( Queries ) = 5,877,263,773 / 1123494 = 5231 /sec -- Queries (including inside SP)
-- >3000 may be stressing server
( Created_tmp_tables ) = 665,507,495 / 1123494 = 592 /sec -- Frequency of creating "temp" tables as part of complex SELECTs.
( Com_delete / Com_insert ) = 15,234,374 / 4603989 = 330.9% -- Deletes / Inserts (as a pct). (Ignores LOAD, REPLACE, etc.)
( Select_scan ) = 491,596,929 / 1123494 = 437 /sec -- full table scans
-- Add indexes / optimize queries (unless they are tiny tables)
( Select_scan / Com_select ) = 491,596,929 / 5360296742 = 9.2% -- % of selects doing full table scan. (May be fooled by Stored Routines.)
-- Add indexes / optimize queries
( Sort_merge_passes ) = 631,000 / 1123494 = 0.56 /sec -- Heafty sorts
-- Increase sort_buffer_size and/or optimize complex queries.
( innodb_autoinc_lock_mode ) = 1 -- Galera: desires 2 -- 2 = "interleaved"; 1 = "consecutive" is typical; 0 = "traditional".
( log_slow_queries ) = OFF -- Whether to log slow queries. (Before 5.1.29, 5.6.1)
( slow_query_log ) = OFF -- Whether to log slow queries. (5.1.12)
( long_query_time ) = 5.000000 = 5 -- Cutoff (Seconds) for defining a "slow" query.
-- Suggest 2
( Slow_queries ) = 407,546,261 / 1123494 = 362 /sec -- Frequency (Slow queries per sec)
-- Rework slow guys; improve indexes; watch disk space for slow log file
( Slow_queries / Questions ) = 407,546,261 / 5877218009 = 6.9% -- Frequency (% of all queries)
-- Find slow queries; check indexes.
( log_queries_not_using_indexes ) = ON -- Whether to include such in slowlog.
-- This clutters the slowlog; turn it off so you can see the real slow queries. And decrease long_query_time to catch most interesting queries.
( max_connect_errors ) = 100,000 -- A small protection against hackers.
-- Perhaps no more than 200.
( Threads_running - 1 ) = 21 - 1 = 20 -- Active threads (concurrency when data collected)
-- Optimize queries and/or schema
( Threads_created / Connections ) = 7,052 / 392731 = 1.8% -- Rapidity of process creation
-- Increase thread_cache_size (non-Windows)
( thread_cache_size / max_connections ) = 8 / 910 = 0.88%
-- (0 for Windows)
( Threads_running / thread_cache_size ) = 21 / 8 = 2.62 -- Threads: current / cached (Not relevant when using thread pooling)
-- Optimize queries
Abnormally large:
Binlog_cache_use = 82 /sec
Bytes_received = 1439336 /sec
Com_commit = 114 /sec
Com_create_table = 60 /HR
Com_delete = 14 /sec
Com_delete_multi = 0.033 /sec
Com_drop_table = 60 /HR
Com_purge_before_date = 0.042 /HR
Com_select = 4771 /sec
Com_set_option = 229 /sec
Com_update = 98 /sec
Created_tmp_files = 0.4 /sec
Handler_commit = 5140 /sec
Handler_prepare = 357 /sec
Handler_read_first = 143 /sec
Handler_write = 19401 /sec
Innodb_buffer_pool_pages_free = 901,470
Select_range = 95 /sec
Sort_range = 3821 /sec
Sort_rows = 15912 /sec
Tc_log_page_waits = 7
Threads_running = 21

Answer 2

Additional suggestions (Jun 21, 2018) to consider for your Linux and my.cnf-ini [mysqld] section
. for Linux since your limit is 1024, in Linux command,
ulimit -n 30000 # to increase Open Files handle count
. for your my.cnf
table_open_cache=10000 # from 256 to lower 20 opened_tables per SECOND
table_definition_cache=800 # from 400 to lower 43,623 opened_table_definitions
open_files_limit=13000 # from 4550 to lower opened_files of 959,559 in 13 days, 1 RPS.

Answer 3

Suggestions to consider for your my.cnf-ini [mysqld] section
max_connections=500 # from 910 only 266 max_used in 13 days
thread_cache_size=100 # from 8 to minimize thread churn and reduce threads_created
read_rnd_buffer_size=128K # from 256K to lower handler_read_rnd_next RPS
key_cache_age_threshold=64800 # from 300 seconds to read same data again
key_cache_division_limit=50 # from 100 for Hot/Warm cache
innodb_buffer_pool_size=24G # from 32G 17G was in use when GLOBAL STATUS recorded
. with your version 5.5.nn special handling required, see REFMAN for next 2.
innodb_log_buffer_size=128M # from 8M to support ~30 minutes
innodb_log_file_size=512M # from 5M for rotation every couple hours
max_write_lock_count=16 # to allow RD after nn locks vs possible wait on 4Billion locks
sort_buffer_size=3M # from 2M to reduce sort_merge_passes
thread_concurrency=30 # from 10 for expedited completion
for additional assistance please find contact information in my profile, Network profile.

Answer 4

(Second answer -- addressing the queries.)
No PRIMARY KEY? Naughty. Does that table have a column (or combination of columns) that is 'naturally' Unique? If so consider making the PK from it/them.
INDEX(p_pkid, p_billingaddress, p_shippingaddress, TypeGKString), in any order, will help the SELECT performance significantly, thereby speeding things up and decreasing CPU consumption.
INDEX(GK) is needed for the UPDATE, (Where's the SHOW CREATE TABLE cronjobs?)
Don't use BIGINT (8 bytes) for columns that need it. (How many different "genders" are there??) (Consider CHAR(2) CHARACTER SET ascii for country.) See TINYINT (1 byte) and its friends.

Answer 5

(Yes, this in my 3rd answer. This time to spell out findings for newer SHOW GLOBAL STATUS.)
Hmmm... I used the old SHOW VARIABLEs, so some of the advice is based on values from that. If you have followed my previous advice, then I need a new copy of SHOW GLOBAL VARIABLES.
Observations:
Version: 5.5.47-log
40 GB of RAM
Uptime = 11d 23:29:02
You are not running on Windows.
Running 64-bit version
You appear to be running entirely (or mostly) InnoDB.
The More Important Issues:
See this about changing innodb_log_file_size. (Or see the 5.5 manual, etc)
innodb_log_file_size = 80M -- too much waiting for the log. See above
innodb_buffer_pool_instances = 16 -- minor contention relief
table_open_cache = 512 -- bumping entries out of this cache has some impact
thread_cache_size = 200 -- process creation is somewhat costly
log_queries_not_using_indexes = OFF -- otherwise clutter
See this for finding the 'worst' queries for us to help you optimize.
Yikes, 268 queries/sec taking more than 5 seconds.
Deleting a lot more often than inserting? What is going on?
9K SETs per second? What is going on?
Why did you choose READ-COMMITTED?
Hundreds of Threads_running usually implies that MySQL is stumbling over itself, and it might be better to throttle the number of connections in the client. Some benchmarks show that 5.6 and 5.7 actually handle this situation better; 5.5 tends to decline in throughtput (queries/sec) after a few dozen connections. Consider upgrading, also since you are 3 major revisions behind.
Details and other observations:
( Opened_tables ) = 13,001,730 / 1034942 = 13 /sec -- Frequency of opening Tables
-- increase table_open_cache
( innodb_buffer_pool_size / innodb_buffer_pool_instances ) = 30720M / 1 = 30720MB -- Size of each buffer_pool instance.
-- An instance should be at least 1GB. In very large RAM, have 16 instances.
( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 45,704,499,200 / (1034942 / 3600) / 2 / 5M = 15.2 -- Ratio
-- (see minutes)
( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 1,034,942 / 60 * 5M / 45704499200 = 1.98 -- Minutes between InnoDB log rotations Beginning with 5.6.8, this can be changed dynamically; be sure to also change my.cnf.
-- (The recommendation of 60 minutes between rotations is somewhat arbitrary.) Adjust innodb_log_file_size. (Cannot change in AWS.)
( Innodb_rows_deleted / Innodb_rows_inserted ) = 3,412,394 / 4031801 = 0.846 -- Churn
-- "Don't queue it, just do it." (If MySQL is being used as a queue.)
( Innodb_row_lock_waits ) = 256,120 / 1034942 = 0.25 /sec -- How often there is a delay in getting a row lock.
-- May be caused by complex queries that could be optimized.
( innodb_stats_on_metadata ) = innodb_stats_on_metadata = ON -- Re-analyze table when touching stats.
-- ON is likely to slow down certain SHOWs and information_schema accesses.
( innodb_thread_concurrency ) = 0 -- 0 = Let InnoDB decide the best for concurrency_tickets.
-- Set to 0 or 64. This may cut back on CPU.
( innodb_print_all_deadlocks ) = innodb_print_all_deadlocks = OFF -- Whether to log all Deadlocks.
-- If you are plagued with Deadlocks, turn this on. Caution: If you have lots of deadlocks, this may write a lot to disk.
( net_buffer_length / max_allowed_packet ) = 16,384 / 35M = 0.04%
( local_infile ) = local_infile = ON
-- local_infile = ON is a potential security issue
( bulk_insert_buffer_size / _ram ) = 8M / 40960M = 0.02% -- Buffer for multi-row INSERTs and LOAD DATA
-- Too big could threaten RAM size. Too small could hinder such operations.
( Questions ) = 15,894,065,519 / 1034942 = 15357 /sec -- Queries (outside SP) -- "qps"
-- >2000 may be stressing server
( Queries ) = 15,894,107,993 / 1034942 = 15357 /sec -- Queries (including inside SP)
-- >3000 may be stressing server
( Created_tmp_tables ) = 501,042,067 / 1034942 = 484 /sec -- Frequency of creating "temp" tables as part of complex SELECTs.
( (Com_insert + Com_update + Com_delete + Com_replace) / Com_commit ) = (4031334 + 54661875 + 13451091 + 0) / 4930449084 = 0.0146 -- Statements per Commit (assuming all InnoDB)
-- Low: Might help to group queries together in transactions; High: long transactions strain various things.
( Com_delete / Com_insert ) = 13,451,091 / 4031334 = 333.7% -- Deletes / Inserts (as a pct). (Ignores LOAD, REPLACE, etc.)
( Select_scan ) = 348,481,657 / 1034942 = 336 /sec -- full table scans
-- Add indexes / optimize queries (unless they are tiny tables)
( Select_scan / Com_select ) = 348,481,657 / 1029506656 = 33.8% -- % of selects doing full table scan. (May be fooled by Stored Routines.)
-- Add indexes / optimize queries
( Sort_merge_passes ) = 297,216 / 1034942 = 0.29 /sec -- Heafty sorts
-- Increase sort_buffer_size and/or optimize complex queries.
( Com_insert + Com_delete + Com_delete_multi + Com_replace + Com_update + Com_update_multi ) = (4031334 + 13451091 + 34494 + 0 + 54661875 + 0) / 1034942 = 70 /sec -- writes/sec
-- 50 writes/sec + log flushes will probably max out I/O write capacity of normal drives
( log_slow_queries ) = log_slow_queries = OFF -- Whether to log slow queries. (Before 5.1.29, 5.6.1)
( slow_query_log ) = slow_query_log = OFF -- Whether to log slow queries. (5.1.12)
( long_query_time ) = 5 -- Cutoff (Seconds) for defining a "slow" query.
-- Suggest 2
( Slow_queries ) = 277,531,287 / 1034942 = 268 /sec -- Frequency (Slow queries per sec)
-- Rework slow guys; improve indexes; watch disk space for slow log file
( Slow_queries / Questions ) = 277,531,287 / 15894065519 = 1.7% -- Frequency (% of all queries)
-- Find slow queries; check indexes.
( log_queries_not_using_indexes ) = log_queries_not_using_indexes = ON -- Whether to include such in slowlog.
-- This clutters the slowlog; turn it off so you can see the real slow queries. And decrease long_query_time to catch most interesting queries.
( max_connect_errors ) = 100,000 -- A small protection against hackers.
-- Perhaps no more than 200.
( Threads_running - 1 ) = 215 - 1 = 214 -- Active threads (concurrency when data collected)
-- Optimize queries and/or schema
( Threads_created / Connections ) = 6,390 / 361954 = 1.8% -- Rapidity of process creation
-- Increase thread_cache_size (non-Windows)
( thread_cache_size / max_connections ) = 8 / 910 = 0.88%
-- (0 for Windows)
( Threads_running / thread_cache_size ) = 215 / 8 = 26.9 -- Threads: current / cached (Not relevant when using thread pooling)
-- Optimize queries
36 issues flagged, out of 162 computed Variables/Status/Expressions
Abnormally large:
Bytes_received = 820009 /sec
Com_commit = 4763 /sec
Com_delete = 13 /sec
Com_delete_multi = 0.033 /sec
Com_drop_table = 60 /HR
Com_purge_before_date = 0.042 /HR
Com_set_option = 9528 /sec
Com_show_engine_status = 21 /HR
Created_tmp_files = 0.21 /sec
Handler_commit = 1207 /sec
Handler_read_first = 105 /sec
Handler_read_key = 72824 /sec
Handler_write = 25131 /sec
Innodb_buffer_pool_pages_misc = 100,245
Key_write_requests = 324 /sec
Select_range = 73 /sec
Sort_range = 221 /sec
Tc_log_page_waits = 1
Threads_running = 215
Abnormal strings:
innodb_fast_shutdown = 1
innodb_use_native_aio = OFF
relay_log_info_file = /PK/mysql/mysql-relay-log-info
time_zone = SYSTPK
tx_isolation = READ-COMMITTED