Insert/delete of row takes multiple seconds under heavy load in MySQL - mysql

Question

In the last few days we are experiencing some strange performance issues with MySQL (old version 5.1.69).
The slow log shows stuff like:
# User#Host: jboss[jboss] # localhost [127.0.0.1]
# Query_time: 46.595796 Lock_time: 0.000022 Rows_sent: 0 Rows_examined: 0
SET timestamp=1496127084;
insert into ACTIVE_RENT (LINE_ID, CHARGED_UP_TO, EVENT_ID, NUMBER, RENT_START, TYPE) values (149914, '2017-05-02 00:00:00', 625751, 'ABCD1234567', '2013-08-02 00:00:00', 'DETENTION');
# User#Host: jboss[jboss] # localhost [127.0.0.1]
# Query_time: 19.401896 Lock_time: 0.000041 Rows_sent: 0 Rows_examined: 0
SET timestamp=1496127084;
delete from ACTIVE_RENT where ACTIVE_RENT_ID=463965;
There is no cascade specified on delete or update and it only has one foreign key to another table and there are no triggers for this table. The table really is quite basic. And there are only about 14k rows.
Normally these insert or deletes are very quick but the last few days they can be very very slow in peak times.
We have increased the innodb_buffer_size to 20G but that didn't change much (with respect to this problem, other stuff is faster). The DB size is about 40GB at the moment.
Below is the my.conf. Any idea where the bottle neck could come from and what to do about it? We are planning on upgrading MySQL to 5.5 but that is not going to happen for a few weeks.
[client]
default-character-set=utf8
socket=/home/mysql-data/mysql/mysql.sock
[mysql]
default-character-set=utf8
max_allowed_packet=32M
[mysqld]
# these are the install settings
local-infile=0
open_files_limit=8192
query_cache_size = 256M
query_cache_type = 1
query_cache_limit = 6M
query_prealloc_size = 16384
query_alloc_block_size = 2048
tmp_table_size = 128M
max_heap_table_size = 256M
table_cache = 1024
thread_cache_size = 512
key_buffer = 256M
join_buffer_size = 8M
read_buffer_size = 8M
sort_buffer_size = 8M
innodb_lock_wait_timeout = 120
binlog_format=row
max_allowed_packet=32M
log-bin=mysql-bin
server-id=1
#Allow group_concat to return longer data types
group_concat_max_len=16384
default-character-set = utf8
skip-character-set-client-handshake
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
innodb_buffer_pool_size = 20GB

Answer 1

query_cache_size = 256M -- too big. Whenever a write occurs, all entries in the QC for that table must be purged. The time for such is roughly proportional to the size. Lower to 50M.
Are your tables InnoDB? If the server has 64GB and is used mostly for MySQL, then set the buffer_pool to 70% of that.
Is ACTIVE_RENT_ID the PRIMARY KEY?
A one-row INSERT into InnoDB will not take 46s unless a long-running ALTER is going on. Look around; something else is involved. Or maybe a long-running transaction that eventually timed out at 50 seconds? Are you using autocommit=0? (I hope not.) Are you using BEGIN...COMMIT?
How big is the table ACTIVE_RENT?