Ubuntu Server is randomly going down - mysql

Question

I am having some trouble with my Digital Ocean's server which has a blog running on Wordpress.
Randomly the server goes on 100% memory/processing and over 200MB/s Disk IO until the blog crashes.
If I restart apache the server comes back to normal.
This is happening over three times a week and it has nothing to do with many users using the site at the same time.
I have browsed the internet and used some solutions and nothing has really helped.
I am using the following:
- Wordpress version: 4.7.2
- Php version:
PHP 5.6.28-1+deb.sury.org~xenial+1 (cli)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
- Linux Version:
Linux 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 16.04 server
I've spent some time looking on logfiles and the dmesg, fail2ban, faillog, and syslog does not have anything unusual.
However the apache error.log and access.log has some interesting things.
On the error log I've seen a lot of lines like these ones:
[Wed Oct 25 16:43:31.234364 2017] [:error] [pid 23211] [client 172.68.26.39:37081] PHP Warning: Error while sending QUERY packet. PID=23211 in public_html/wp-includes/wp-db.php on line 1877, referer: https://sitename/blogpostname/comment-page-6/
[Wed Oct 25 16:43:31.615950 2017] [:error] [pid 23216] [client 172.68.27.196:15015] PHP Warning: Error while sending QUERY packet. PID=23216 in public_html/wp-includes/wp-db.php on line 1877, referer: https://www.google.com.br/
[Wed Oct 25 16:43:31.600365 2017] [:error] [pid 23224] [client 172.68.27.64:20351] PHP Warning: Error while sending QUERY packet. PID=23224 in public_html/wp-includes/wp-db.php on line 1877, referer: https://www.google.com.br/
[Wed Oct 25 16:43:32.736840 2017] [:error] [pid 22410] [client 162.158.74.163:17321] PHP Warning: Error while sending QUERY packet. PID=22410 in public_html/wp-includes/wp-db.php on line 1877
Went on the wp-db.php file on line 1877 and it is a random usual line on the file, nothing wrong with it.
After a google search I've seen a lot of people with the same problem changing some variables on the php and apache configuration files.
So I've increased the following variables:
max_execution_time
max_allowed_packet_size
max_user_connections
max_connections
And disabled the W3 Total Cache plugin
Still with no Luck.
Checked the cronjobs and it doesn't have any job which trigger at the same time that I had this problem.
I am putting the apache's error.log and access.log and mysql error.log on the above links, in case you need them.
Last time I had this issue was 25 OCT 16:40h until the restart of the apache service at 16:57h
Some lines on the MYSQL ERROR LOG FILE
2017-10-25T18:40:00.355353Z 11952 [Note] Aborted connection 11952 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:40:01.869160Z 12116 [Note] Aborted connection 12116 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:10.465788Z 12185 [Note] Aborted connection 12185 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:11.058421Z 12199 [Note] Aborted connection 12199 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:13.415412Z 12187 [Note] Aborted connection 12187 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:15.844868Z 12186 [Note] Aborted connection 12186 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:21.387382Z 12217 [Note] Aborted connection 12217 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:21.418078Z 12218 [Note] Aborted connection 12218 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:21.747628Z 12093 [Note] Aborted connection 12093 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:24.705706Z 12219 [Note] Aborted connection 12219 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:43:29.267584Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2017-10-25T18:43:29.268753Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2017-10-25T18:43:29.812542Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-10-25T18:43:29.874865Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.15-0ubuntu0.16.04.1) starting as process 23296 ...
2017-10-25T18:43:29.945767Z 0 [Note] InnoDB: PUNCH HOLE support available
2017-10-25T18:43:29.945855Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-10-25T18:43:29.945891Z 0 [Note] InnoDB: Uses event mutexes
2017-10-25T18:43:29.945902Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-10-25T18:43:29.945911Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-10-25T18:43:29.945922Z 0 [Note] InnoDB: Using Linux native AIO
2017-10-25T18:43:29.954531Z 0 [Note] InnoDB: Number of pools: 1
2017-10-25T18:43:29.957647Z 0 [Note] InnoDB: Using CPU crc32 instructions
2017-10-25T18:43:29.962895Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2017-10-25T18:43:30.069723Z 0 [Note] InnoDB: Completed initialization of buffer pool
2017-10-25T18:44:29.139834Z 0 [ERROR] InnoDB: Cannot allocate 1677722136 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
2017-10-25 16:44:29 0x7fec07b49740 InnoDB: Assertion failure in thread 140651718285120 in file ut0ut.cc line 931
InnoDB: Failing assertion: !m_fatal
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
18:44:29 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 76382 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xe7f80b]
/usr/sbin/mysqld(handle_fatal_signal+0x489)[0x784769]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x113d0)[0x7fec06ac03d0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fec05e7a418]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fec05e7c01a]
/usr/sbin/mysqld[0x75a5ee]
/usr/sbin/mysqld(_ZN2ib14fatal_or_errorD2Ev+0x2e8)[0x1116f58]
/usr/sbin/mysqld[0xfd53c5]
/usr/sbin/mysqld(_Z8log_initv+0x4a5)[0xfdba25]
/usr/sbin/mysqld(_Z34innobase_start_or_create_for_mysqlv+0x22ed)[0x10bcd3d]
/usr/sbin/mysqld[0xf81339]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x51)[0x7d1bc1]
/usr/sbin/mysqld[0xc665b5]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x608)[0xc6dc58]
/usr/sbin/mysqld[0x77cce4]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x7e4)[0x77e264]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fec05e65830]
/usr/sbin/mysqld(_start+0x29)[0x774c79]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2017-10-25T18:44:29.747705Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2017-10-25T18:44:29.747973Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2017-10-25T18:44:30.168154Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-10-25T18:44:30.175188Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.15-0ubuntu0.16.04.1) starting as process 23379 ...
2017-10-25T18:44:30.227947Z 0 [Note] InnoDB: PUNCH HOLE support available
2017-10-25T18:44:30.228037Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-10-25T18:44:30.228056Z 0 [Note] InnoDB: Uses event mutexes
2017-10-25T18:44:30.228067Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-10-25T18:44:30.228075Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-10-25T18:44:30.228083Z 0 [Note] InnoDB: Using Linux native AIO
2017-10-25T18:44:30.229957Z 0 [Note] InnoDB: Number of pools: 1
2017-10-25T18:44:30.231671Z 0 [Note] InnoDB: Using CPU crc32 instructions
2017-10-25T18:44:30.235573Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2017-10-25T18:44:30.262313Z 0 [Note] InnoDB: Completed initialization of buffer pool
2017-10-25T18:45:26.314564Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2017-10-25T18:45:26.351139Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2017-10-25T18:45:26.387449Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 162362845023
2017-10-25T18:45:26.387528Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 162362845032
2017-10-25T18:45:26.387864Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 162362845032
2017-10-25T18:45:26.387888Z 0 [Note] InnoDB: Database was not shutdown normally!
2017-10-25T18:45:26.387932Z 0 [Note] InnoDB: Starting crash recovery.
2017-10-25T18:45:27.653149Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2017-10-25T18:45:27.655693Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2017-10-25T18:45:27.655993Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2017-10-25T18:45:27.832424Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2017-10-25T18:45:27.839253Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2017-10-25T18:45:27.839314Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2017-10-25T18:45:27.841792Z 0 [Note] InnoDB: Waiting for purge to start
2017-10-25T18:45:27.897127Z 0 [Note] InnoDB: 5.7.15 started; log sequence number 162362845032
2017-10-25T18:45:27.913296Z 0 [Note] Plugin 'FEDERATED' is disabled.
2017-10-25T18:45:27.921641Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2017-10-25T18:45:28.017907Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2017-10-25T18:45:28.018041Z 0 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
2017-10-25T18:45:28.018712Z 0 [Note] - '127.0.0.1' resolves to '127.0.0.1';
2017-10-25T18:45:28.019306Z 0 [Note] Server socket created on IP: '127.0.0.1'.
2017-10-25T18:45:28.163336Z 0 [Note] Event Scheduler: Loaded 0 events
2017-10-25T18:45:28.166514Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.15-0ubuntu0.16.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
2017-10-25T18:45:30.429764Z 0 [Note] InnoDB: Buffer pool(s) load completed at 171025 16:45:30
2017-10-25T18:53:52.366879Z 159 [Note] Aborted connection 159 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:03.345944Z 200 [Note] Aborted connection 200 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:04.660063Z 199 [Note] Aborted connection 199 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:05.729653Z 177 [Note] Aborted connection 177 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:12.520890Z 205 [Note] Aborted connection 205 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:12.669290Z 155 [Note] Aborted connection 155 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:16.064185Z 175 [Note] Aborted connection 175 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:19.823038Z 51 [Note] Aborted connection 51 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:20.741167Z 206 [Note] Aborted connection 206 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:20.746381Z 42 [Note] Aborted connection 42 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:24.312023Z 122 [Note] Aborted connection 122 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:24.352842Z 186 [Note] Aborted connection 186 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:25.449125Z 148 [Note] Aborted connection 148 to db: '<Database name>' user: '<Database User>' host: 'localhost' (Got timeout reading communication packets)
2017-10-25T18:56:28.626509Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2017-10-25T18:56:28.626841Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2017-10-25T18:56:30.071894Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2017-10-25T18:56:30.105638Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.15-0ubuntu0.16.04.1) starting as process 23873 ...
2017-10-25T18:56:30.235898Z 0 [Note] InnoDB: PUNCH HOLE support available
2017-10-25T18:56:30.236056Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-10-25T18:56:30.236126Z 0 [Note] InnoDB: Uses event mutexes
2017-10-25T18:56:30.236150Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-10-25T18:56:30.236167Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-10-25T18:56:30.236182Z 0 [Note] InnoDB: Using Linux native AIO
2017-10-25T18:56:30.239770Z 0 [Note] InnoDB: Number of pools: 1
2017-10-25T18:56:30.243200Z 0 [Note] InnoDB: Using CPU crc32 instructions
2017-10-25T18:56:30.297785Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2017-10-25T18:56:30.471494Z 0 [Note] InnoDB: Completed initialization of buffer pool
2017-10-25T18:57:24.662239Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2017-10-25T18:57:24.855967Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2017-10-25T18:57:24.892537Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 162362990185
2017-10-25T18:57:24.892649Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 162362990194
2017-10-25T18:57:24.893393Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 162362990194
2017-10-25T18:57:24.893430Z 0 [Note] InnoDB: Database was not shutdown normally!
2017-10-25T18:57:24.893449Z 0 [Note] InnoDB: Starting crash recovery.
2017-10-25T18:57:25.486004Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2017-10-25T18:57:25.486109Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2017-10-25T18:57:25.486191Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2017-10-25T18:57:25.561775Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2017-10-25T18:57:25.565595Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2017-10-25T18:57:25.565662Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2017-10-25T18:57:25.583476Z 0 [Note] InnoDB: 5.7.15 started; log sequence number 162362990194
2017-10-25T18:57:25.585667Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2017-10-25T18:57:25.589295Z 0 [Note] Plugin 'FEDERATED' is disabled.
2017-10-25T18:57:25.661688Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2017-10-25T18:57:25.661775Z 0 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
2017-10-25T18:57:25.661845Z 0 [Note] - '127.0.0.1' resolves to '127.0.0.1';
2017-10-25T18:57:25.661935Z 0 [Note] Server socket created on IP: '127.0.0.1'.
2017-10-25T18:57:25.756913Z 0 [Note] Event Scheduler: Loaded 0 events
2017-10-25T18:57:25.758546Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.15-0ubuntu0.16.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
2017-10-25T18:57:25.958884Z 0 [Note] InnoDB: Buffer pool(s) load completed at 171025 16:57:25
Some lines of the Apache Error.log
[Wed Oct 25 06:25:48.823233 2017] [core:notice] [pid 17104] AH00094: Command line: '/usr/sbin/apache2'
[Wed Oct 25 16:40:06.602720 2017] [:error] [pid 22585] [client 172.68.25.164:33251] PHP Warning: Error while sending QUERY packet. PID=22585 in /var/www/<site name>/public_html/wp-includes/wp-db.php on line 1877, referer: https://<site name>/2015/05/27/o-gerente-responde-qual-a-melhor-opcao-para-amortizar-o-financiamento-imobiliario/
[Wed Oct 25 16:40:35.820625 2017] [:error] [pid 22700] [client 162.158.74.163:29857] PHP Warning: Error while sending QUERY packet. PID=22700 in /var/www/<site name>/public_html/wp-includes/wp-db.php on line 1877, referer: https://<site name>/amortizacao-prazo-ou-parcela/
[Wed Oct 25 16:41:30.171910 2017] [:error] [pid 22751] [client 172.68.254.178:19539] PHP Warning: Error while sending QUERY packet. PID=22751 in /var/www/<site name>/public_html/wp-includes/wp-db.php on line 1877
[Wed Oct 25 16:44:52.784567 2017] [:error] [pid 22463] [client 108.162.219.224:10437] PHP Warning: Error while sending QUERY packet. PID=22463 in /var/www/mkt.<site name>/public_html/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php on line 1176
[Wed Oct 25 16:44:53.006562 2017] [:error] [pid 22892] [client 172.68.26.63:12817] PHP Warning: Error while sending QUERY packet. PID=22892 in /var/www/<site name>/public_html/wp-includes/wp-db.php on line 1877, referer: http://www.google.com.br/
[Wed Oct 25 16:44:56.689245 2017] [:error] [pid 22862] [client 108.162.219.224:9729] PHP Warning: Error while sending QUERY packet. PID=22862 in /var/www/mkt.<site name>/public_html/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php on line 1176
[Wed Oct 25 16:45:25.240149 2017] [:error] [pid 23208] [client 172.68.25.68:29319] PHP Warning: Error while sending QUERY packet. PID=23208 in /var/www/<site name>/public_html/wp-includes/wp-db.php on line 1877, referer: https://<site name>/2015/04/08/maquinas-de-cartao-de-credito-um-comparativo/
[Wed Oct 25 16:45:25.336803 2017] [:error] [pid 23228] [client 172.68.25.68:24895] PHP Warning: Error while sending QUERY packet. PID=23228 in /var/www/<site name>/public_html/wp-includes/wp-db.php on line 1877, referer: https://<site name>/2015/04/08/maquinas-de-cartao-de-credito-um-comparativo/
[Wed Oct 25 16:45:47.340049 2017] [:error] [pid 22925] [client 172.68.25.26:15907] PHP Warning: Error while sending QUERY packet. PID=22925 in /var/www/<site name>/public_html/wp-includes/wp-db.php on line 1877
AH00045: child process 22899 still did not exit, sending a SIGTERM
[Wed Oct 25 16:57:24.320984 2017] [core:warn] [pid 17104] AH00045: child process 22911 still did not exit, sending a SIGTERM
[Wed Oct 25 16:57:24.321014 2017] [core:warn] [pid 17104] AH00045: child process 22923 still did not exit, sending a SIGTERM
[Wed Oct 25 16:57:25.323354 2017] [core:warn] [pid 17104] AH00045: child process 23840 still did not exit, sending a SIGTERM
[Wed Oct 25 16:57:25.323570 2017] [core:warn] [pid 17104] AH00045: child process 23899 still did not exit, sending a SIGTERM
[Wed Oct 25 16:57:25.323586 2017] [core:warn] [pid 17104] AH00045: child process 22649 still did not exit, sending a SIGTERM
[Wed Oct 25 16:57:26.325315 2017] [core:notice] [pid 17104]
AH00051: child pid 23899 exit signal Segmentation fault (11), possible coredump in /etc/apache2`
[Wed Oct 25 16:57:29.895519 2017] [core:notice] [pid 24045] AH00094: Command line: '/usr/sbin/apache2'
[Wed Oct 25 16:57:27.326904 2017] [core:error] [pid 17104] AH00046: child process 23840 still did not exit, sending a SIGKILL
[Wed Oct 25 16:57:27.329833 2017] [core:error] [pid 17104] AH00046: child process 22649 still did not exit, sending a SIGKILL
[Wed Oct 25 16:57:28.340357 2017] [mpm_prefork:notice] [pid 17104] AH00169: caught SIGTERM, shutting down
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20131226/newrelic.so' - /usr/lib/php/20131226/newrelic.so: cannot open shared object file: No such file or directory in Unknown on line 0
[Wed Oct 25 16:57:29.894473 2017] [mpm_prefork:notice] [pid 24045] AH00163: Apache/2.4.18 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.0.2g-fips configured -- resuming normal operations
My configuration files:
mysqld.cnf
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
bind-address = 127.0.0.1
key_buffer_size = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover-options = BACKUP
query_cache_limit = 1M
query_cache_size = 16M
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
my.cnf
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/
#innodb_log_file_size = 256M
[mysqld]
innodb_file_per_table
#innodb_file_format = Barracuda
max_allowed_packet = 1G
innodb_log_file_size = 500M
innodb_log_buffer_size = 800M
innodb_strict_mode = 0
wait_timeout = 100
php.ini
[PHP]
engine = On
short_open_tag = Off
asp_tags = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = On
max_execution_time = 100
max_input_time = 60
memory_limit = -1
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 8M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 24M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
[CLI Server]
cli_server.color = On
[Date]
[filter]
[iconv]
[intl]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQL]
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 60
mysql.trace_mode = Off
[MySQLi]]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[Sybase-CT]
sybct.allow_persistent = On
sybct.max_persistent = -1
sybct.max_links = -1
sybct.min_server_severity = 10
sybct.min_client_severity = 10
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 0
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[MSSQL]
mssql.allow_persistent = On
mssql.max_persistent = -1
mssql.max_links = -1
mssql.min_error_severity = 10
mssql.min_message_severity = 10
mssql.compatibility_mode = Off
mssql.secure_connection = Off
[Assertion]
[COM]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
The GLOBAL STATUS and GLOBAL VARIABLES are on dropbox files:
https://www.dropbox.com/s/zzquqp4jfrs6z43/SHOW%20GLOBAL%20STATUS.txt?dl=0
https://www.dropbox.com/s/5p0hwlq2ma3c6jd/SHOW%20GLOBAL%20VARIABLES.txt?dl=0
And the Engine InnoDb status
Type Name Status
InnoDB "=====================================
2017-10-27 ..."

Answer 1

The following suggestions need your research before implementing ONLY one item per day. Some may be applied dynamically. Suggested cfg/ini values follow, could be modify, add or remove.
innodb_print_all_deadlocks=1 # from OFF to record in error log for CORRECTION
slow_query_log=1 # from OFF to record for possible CORRECTION
innodb_log_buffer_size=838860800 REMOVE to allow default of 16M (buffer size is larger than file size today and likely your most significant difficulty) NOT a dynamic variable
thread_cache_size=100 #from 8 to support 500 created threads since start
innodb_read_ahead_threshold=8 #from 56 for your intensity of reading
innodb_stats_sample_pages=32 #from 8 for more accurate cardinality of ndx's
have_query_cache=NO #from yes default since you have no intention of using qc
query_cache_size=0 #from 16M to conserve RAM
key_buffer_size=1M #from 16M, only used by ISAM. Low requirements
key_cache_age_threshold=64800 #from 300 seconds, avoid RD for 8 hours
max_write_lock_count=128 #from enormous number to allow RD after this nn
sql_select_limit=1000000000 #from enormous number to 1 Billion rows selected limit
table_open_cache= 10000 #from 2000 to support 40,000 opened since start.
ulimit needs to be reviewed and made more tolerant, if possible.
When appropriate, provide feedback, please.