Why MySQL Restarted When Number of Processes Running Now 0 - mysql

Some background. I hosted a WordPress site in VPS and sometimes MYSQL down with error "Error Establishing a Database Connection". I've spent some time to research and believe the problem is due to when MySQL get restarted, it is not able to allocate enough RAM to proceed.
I believe I can improve the situation by increasing physical RAM or swap. But my question for this post is, why MySQL need to restart itself? My site is with pretty low traffic and doesn't seem like the DB is corrupted.
Below is the full log for this issue:
160103 18:39:54 mysqld_safe Number of processes running now: 0
160103 18:39:54 mysqld_safe mysqld restarted
160103 18:39:55 [Note] /usr/libexec/mysqld (mysqld 5.5.44-MariaDB) starting as process 22061 ...
160103 18:39:55 InnoDB: The InnoDB memory heap is disabled
160103 18:39:55 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160103 18:39:55 InnoDB: Compressed tables use zlib 1.2.7
160103 18:39:55 InnoDB: Using Linux native AIO
160103 18:39:55 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
160103 18:39:55 InnoDB: Completed initialization of buffer pool
160103 18:39:55 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160103 18:39:56 [ERROR] Plugin 'InnoDB' init function returned error.
160103 18:39:56 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
160103 18:39:56 [Note] Plugin 'FEEDBACK' is disabled.
160103 18:39:56 [ERROR] Unknown/unsupported storage engine: InnoDB
160103 18:39:56 [ERROR] Aborting

When Number of Processes Running Now 0 means that MySQL isn't running. So the daemon mysqld_safe "makes you a favor" and start MySQL

You have assigned very low RAM 128MB to innodb_buffer_pool_size (which is default RAM). So you should assign approx. 80% of total RAM to this variable if you are using innodb engine, as mysql uses intially memory from this variable to cache index as well data in innodb engine.
So update at least 1 GB RAM (should be 80% of total ram for innodb) to innodb_buffer_pool_size in your config file and restart mysql service.
Update:
You have 1 GB RAM: You can assign 800M RAM (or >=500M) to innodb_buffer_pool_size.
MySQL will auto restart when number of running process 0: As per errors shared by you...this error is coming at the time of mysql service start and server is not able to initiate mysql service with such a less RAM.
160103 18:39:55 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
160103 18:39:55 InnoDB: Completed initialization of buffer pool
160103 18:39:55 InnoDB: Fatal error: cannot allocate memory for the buffer pool

I had a similar problem with MySQL on CentOS 6.9. It was caused by memory starving due another PHP process and not by MySQL. MySQL was just a victim.
You can look at kernel messages. The location of this messages may be specific to OS. In case of CentOS 6.x they are located in /var/log/messages .
Here are some related messages from /var/log/messages:
Jul 25 20:34:46 myserver kernel: Out of memory: Kill process 21467 (mysqld) score 30 or sacrifice child
Jul 25 20:34:46 myserver kernel: Killed process 21467, UID 497, (mysqld) total-vm:757004kB, anon-rss:17728kB, file-rss:320kB
You can run the following command to see if kernel run out of memory:
cat /var/log/messages | grep out_of_memory

Related

Server is shutting down every few minuts and MYSQL Daemon is failing to start

I have a problem with my server. It shuts down every 2 minutes. I am using Digitalocean. I have changed to a newdroplet already it was fine for one month. then. I cannot even start MYSQL service.. this is my log:
160426 17:06:34 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160426 17:08:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160426 17:08:04 [Note] /usr/libexec/mysqld (mysqld 5.5.48) starting as process 3334 ...
160426 17:08:04 [Note] Plugin 'FEDERATED' is disabled.
160426 17:08:04 InnoDB: The InnoDB memory heap is disabled
160426 17:08:04 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160426 17:08:04 InnoDB: Compressed tables use zlib 1.2.3
160426 17:08:04 InnoDB: Using Linux native AIO
160426 17:08:04 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160426 17:08:04 InnoDB: Completed initialization of buffer pool
160426 17:08:04 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160426 17:08:04 [ERROR] Plugin 'InnoDB' init function returned error.
160426 17:08:04 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160426 17:08:04 [ERROR] Unknown/unsupported storage engine: InnoDB
160426 17:08:04 [ERROR] Aborting
Any help is appreciated..
Fatal error: cannot allocate memory for the buffer pool
How big is your buffer pool? How much memory do you have? Who installed MySQL on the server?
it was fine for one month
What has changed? What is using memory on your server?
For a tiny 0.5GB system, lots of things must be dialed back.
50 http processes -- from what? Apache: drastically decrease MaxClients, perhaps to 10. Others: ?
128MB is too much for 512MB. Try 50M.
max_connections should be perhaps 10.
Are your tables InnoDB? Or MyISAM?
If InnoDB, key_buffer_size = 5M, innodb_buffer_pool_size = 50M.
If MyISAM, key_buffer_size = 40M, innodb_buffer_pool_size = 0.
You have version 5.5.48 of MySQL. Did Digitalocean change it without your knowing it?

What to do when I get "mysqld dead but subsys locked" in my amazon ec2 micro instance?

I have an Amazon ec2 micro instance that holds a WordPress site. Since yesterday, my mysql db keep crashing saying:
[ec2-user#ip-xxx-xx-xx-xxx ~]$ sudo service mysqld status
mysqld dead but subsys locked
I tried restarting the mysqld but after every 5 mins it keeps crashing with the same message - "Error establishing a database connection"
This is what my DB log file reads -
160123 05:44:41 mysqld_safe mysqld restarted
160123 5:44:43 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 7928 ...
160123 5:44:44 [Note] Plugin 'FEDERATED' is disabled.
160123 5:44:46 InnoDB: The InnoDB memory heap is disabled
160123 5:44:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160123 5:44:46 InnoDB: Compressed tables use zlib 1.2.8
160123 5:44:46 InnoDB: Using Linux native AIO
160123 5:44:46 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160123 5:44:46 InnoDB: Completed initialization of buffer pool
160123 5:44:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160123 5:44:46 [ERROR] Plugin 'InnoDB' init function returned error.
160123 5:44:46 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160123 5:44:46 [ERROR] Unknown/unsupported storage engine: InnoDB
160123 5:44:46 [ERROR] Aborting
160123 5:44:46 [Note] /usr/libexec/mysql55/mysqld: Shutdown complete
160123 05:44:46 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
The common solution that found out there when I Googled for a solution for the issue - mysqld dead but subsys locked, is to create a swapfile. I did exactly as most of the solution suggest but since I'm not well versed with UNIX, I don't know what I have done is right or not.
Or, Is there any other solution for this issue. Kindly help.
Have you tried this mysql is dead but subsys locked
Seems quite straightforward. Also please consider moving to RDS instead of running a mysql server in ec2. Saves you from troubles of this kind.

MySQL has gone away on large queries

I have a MySQL installed and running on a CentOS 6.6 and MySQL version 5.5.40 on RackSpace. I always run into this error when running heavy queries.
Here is the settings of my.cnf
[mysqld]
2 datadir=/mnt/data/mysql
3 tmpdir=/mnt/data/temp
4 socket=/var/lib/mysql/mysql.sock
5 bind-address=0.0.0.0
6 port=3306
7 wait_timeout=432000
8 max_allowed_packet=1G
9 max_connections=500
10 query-cache-size=0
11 query-cache-type=0
12 #query_cache_size=64M
13 #query_cache_limit=64M
14 key_buffer_size=1G
15 sort_buffer_size=16M
16 tmp_table_size=32M
17 max_heap_table_size=32M
18 read_buffer_size=512K
19 read_rnd_buffer_size=512K
20 thread_cache_size=50
21
22 innodb_buffer_pool_size=12G
23 innodb_buffer_pool_instance=2
24 innodb_read_io_threads=12
25 innodb_write_io_threads=12
26 innodb_io_capacity=300
27 innodb_log_file_size=128M
28 innodb_thread_concurrency=0
Here is the error log I've caught after the crash:
150820 13:46:26 mysqld_safe Number of processes running now: 0
150820 13:46:26 mysqld_safe mysqld restarted
150820 13:46:26 [Note] Plugin 'FEDERATED' is disabled.
150820 13:46:26 [Warning] Using unique option prefix innodb_buffer_pool_instance instead of innodb-buffer-pool-instances is deprecated and will be removed in a future release. Please use the full name instead.
150820 13:46:26 InnoDB: The InnoDB memory heap is disabled
150820 13:46:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150820 13:46:26 InnoDB: Compressed tables use zlib 1.2.3
150820 13:46:26 InnoDB: Using Linux native AIO
150820 13:46:26 InnoDB: Initializing buffer pool, size = 12.0G
InnoDB: mmap(6593445888 bytes) failed; errno 12
150820 13:46:27 InnoDB: Completed initialization of buffer pool
150820 13:46:27 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150820 13:46:27 [ERROR] Plugin 'InnoDB' init function returned error.
150820 13:46:27 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150820 13:46:27 [ERROR] Unknown/unsupported storage engine: InnoDB
150820 13:46:27 [ERROR] Aborting
150820 13:46:27 [Note] /usr/libexec/mysqld: Shutdown complete
150820 13:46:27 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
EDIT:
RackSpace VM's specs are:
CPU: Intel(R) Xeon(R) CPU E5-2670 0 # 2.60GHz
RAM: 8GB
your server has only 8 GB RAM and you have assigned too much ram to mysql.
Even you need to change your configuration for many variables but first to comeout your issue do the below changes-
innodb_buffer_pool_instance=2 #comment it for the time being we can set it later.
innodb_buffer_pool_size=6G
key_buffer_size=20M #if your server is innodb but if you are also using myisam tables then keep as it is.
sort_buffer_size=2M # we can change later.
read_buffer_size=512K # comment it for time being.
read_rnd_buffer_size=512K #comment it for time being as used per session
tmp_table_size=1G #this can be reason of your problem so increase it.
max_heap_table_size=1G #this can be reason of your problem so increase it.
If possible decrease max_connections from 500 to 400 as each connection uses server resources.
Try and share the results.
In case it is useful, this is how I solve this problem. In my case the problem was caused by queries that were too large for the packets being sent to the server.
After running the following command against the server, the large queries process fine
SET GLOBAL max_allowed_packet=1073741824;

MYsql Shut down often for no reason and unable restart after

150505 16:57:01 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150505 16:58:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150505 16:58:01 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150505 16:58:01 [Note] Plugin 'FEDERATED' is disabled.
150505 16:58:01 InnoDB: The InnoDB memory heap is disabled
150505 16:58:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150505 16:58:01 InnoDB: Compressed tables use zlib 1.2.8
150505 16:58:01 InnoDB: Using Linux native AIO
150505 16:58:01 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150505 16:58:01 InnoDB: Completed initialization of buffer pool
150505 16:58:01 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150505 16:58:01 [ERROR] Plugin 'InnoDB' init function returned error.
150505 16:58:01 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150505 16:58:01 [ERROR] Unknown/unsupported storage engine: InnoDB
150505 16:58:01 [ERROR] Aborting
Above is mysql error log, sometime mysql will just crash, and I want to do a restart on the service, it will not able to start,the error is something like
the "unknown instance" error.
Anyone got any idea what the error above is about and how to solve it, I tried google but seems to not able find the answer.
Thanks!
I found this question searching for an answer myself, my server's /var/log/mysql/error.log had very similar statements to yours. Running dmesg | tail -20 revealed the culprit: the OS ran out of memory and killed mysqld:
[835661.446277] Out of memory: Kill process 1024 (mysqld) score 91 or sacrifice child
[835661.446912] Killed process 1024 (mysqld) total-vm:901544kB, anon-rss:92332kB, file-rss:0kB
[835661.488337] init: mysql main process (1024) killed by KILL signal
Now to figure out why memory ran out, on this newer server, with twice as much RAM as the one it's replacing, which never did this...

mysql Fatal error: cannot allocate memory for the buffer pool

I have this error log from MySQL, any idea?
Website works for some time and then I get MySQL shutdown completely after a couple of hours.
140919 10:48:27 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140919 10:48:27 [Note] Plugin 'FEDERATED' is disabled.
140919 10:48:27 InnoDB: The InnoDB memory heap is disabled
140919 10:48:27 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140919 10:48:27 InnoDB: Compressed tables use zlib 1.2.3.4
140919 10:48:28 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140919 10:48:28 InnoDB: Completed initialization of buffer pool
140919 10:48:28 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140919 10:48:28 [ERROR] Plugin 'InnoDB' init function returned error.
140919 10:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140919 10:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB
140919 10:48:28 [ERROR] Aborting
140919 10:48:28 [Note] /usr/sbin/mysqld: Shutdown complete
140919 10:48:28 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140919 10:48:28 [Note] Plugin 'FEDERATED' is disabled.
140919 10:48:28 InnoDB: The InnoDB memory heap is disabled
140919 10:48:28 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140919 10:48:28 InnoDB: Compressed tables use zlib 1.2.3.4
140919 10:48:28 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140919 10:48:28 InnoDB: Completed initialization of buffer pool
140919 10:48:28 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140919 10:48:28 [ERROR] Plugin 'InnoDB' init function returned error.
140919 10:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140919 10:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB
140919 10:48:28 [ERROR] Aborting
140919 10:48:28 [Note] /usr/sbin/mysqld: Shutdown complete
TLDR;
Mysql can't restart because it's out of memory, check that you have an appropriate swapfile configured.
Didn't help? If that's not your issue, more qualified questions to continue research are:
mysqld service stops once a day on ec2 server
https://askubuntu.com/questions/422037/optimising-mysql-settings-mysqld-running-out-of-memory
Background
I had exactly this problem on the very first system I set up on EC2, characterised by the wordpress site hosted there going down on occasion with "Error establishing database connection".
The logs showed the same error that the OP posted. My reading of the error (timestamps removed) is:
Out of memory error:
InnoDB: Fatal error: cannot allocate memory for the buffer pool
InnoDB can't start without enough memory
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting
mysqld is shutting down, which in this context, really means failing to restart!
[Note] /usr/sbin/mysqld: Shutdown complete
Checking /var/log/syslog and searching for mysql yields:
Out of memory: Kill process 15452 (mysqld) score 93 or sacrifice child
Killed process 15452 (mysqld) total-vm:888672kB, anon-rss:56252kB, file-rss:0kB
init: mysql main process (15452) killed by KILL signal
init: mysql main process ended, respawning
type=1400 audit(1443812767.391:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=21984 comm="apparmor_parser"
init: mysql main process (21996) terminated with status 1
init: mysql main process ended, respawning
init: mysql post-start process (21997) terminated with status 1
<repeated>
Note: you may have to gunzip and search through archived logs if the error occurred before the logs were rotated by cron.
Solution
In my case the underlying issue was that I'd neglected to configure a swapfile.
You can check to see if you have one configured by running free -m.
total used free shared buffers cached
Mem: 604340 587364 16976 0 29260 72280
-/+ buffers/cache: 485824 118516
Swap: 0 0 0
In the example above, Swap: 0 indicates no swapfile.
Tutorials on setting one up:
https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04
https://help.ubuntu.com/community/SwapFaq
Note that bigger is not necessarily better! From the Ubuntu guide:
The "diminishing returns" means that if you need more swap space than twice your RAM size, you'd better add more RAM as Hard Disk Drive (HDD) access is about 10³ slower then RAM access, so something that would take 1 second, suddenly takes more then 15 minutes! And still more then a minute on a fast Solid State Drive (SSD)...
Regarding the other answers here...
The InnoDB memory heap is disabled
This isn’t really an error, just an indication that InnoDB is using the system’s internal memory allocator instead of its own. The default is yes/1, and is acceptable for production.
According to the docs, this command is deprecated, and will be removed in MySQL versions above 5.6 (and I assume MariaDB):
http://dev.mysql.com/doc/refman/5.6/en/innodb-performance-use_sys_malloc.html
Thanks to: Ruben Schade comment
[Note] Plugin 'FEDERATED' is disabled.
The message about FEDERATED disabled is not an error. It just meant that the FEDERATED engine its not ON for your mysql server. It's not used by default. If you don't need it, don't care about this message.
See: https://stackoverflow.com/a/16470822/2586761
I found this answer adds to the discussion: https://www.digitalocean.com/community/questions/mysql-server-keeps-stopping-unexpectedly?answer=26021
In short, on top of setting innodb_buffer_pool_size to something reasonable like 64M, you also may need to modify /etc/apache2/mods-enabled/mpm_prefork.conf to reduce the number of connections started by apache;
<IfModule mpm_prefork_module>
StartServers 3
MinSpareServers 3
MaxSpareServers 5
MaxRequestWorkers 25
MaxConnectionsPerChild 1024
</IfModule>
The solution is NOT more space, Problem is Apache web server not mysql, actually you need to decrease innodb-buffer-pool-size
This buffer is used by the mysql process right off the start, so when Apache needs more resources the kernel will clear RAM from services this means stopping mysql instead of crashing the server.
Would also add a CRON to check the db status and restart it if you dont want to change to ngnx or httplight.