150505 16:57:01 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150505 16:58:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150505 16:58:01 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150505 16:58:01 [Note] Plugin 'FEDERATED' is disabled.
150505 16:58:01 InnoDB: The InnoDB memory heap is disabled
150505 16:58:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150505 16:58:01 InnoDB: Compressed tables use zlib 1.2.8
150505 16:58:01 InnoDB: Using Linux native AIO
150505 16:58:01 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150505 16:58:01 InnoDB: Completed initialization of buffer pool
150505 16:58:01 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150505 16:58:01 [ERROR] Plugin 'InnoDB' init function returned error.
150505 16:58:01 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150505 16:58:01 [ERROR] Unknown/unsupported storage engine: InnoDB
150505 16:58:01 [ERROR] Aborting
Above is mysql error log, sometime mysql will just crash, and I want to do a restart on the service, it will not able to start,the error is something like
the "unknown instance" error.
Anyone got any idea what the error above is about and how to solve it, I tried google but seems to not able find the answer.
Thanks!
I found this question searching for an answer myself, my server's /var/log/mysql/error.log had very similar statements to yours. Running dmesg | tail -20 revealed the culprit: the OS ran out of memory and killed mysqld:
[835661.446277] Out of memory: Kill process 1024 (mysqld) score 91 or sacrifice child
[835661.446912] Killed process 1024 (mysqld) total-vm:901544kB, anon-rss:92332kB, file-rss:0kB
[835661.488337] init: mysql main process (1024) killed by KILL signal
Now to figure out why memory ran out, on this newer server, with twice as much RAM as the one it's replacing, which never did this...
Related
I have installed mysql server on ec2 AMI. Suddenly stopped working mysql server and its not getting started.
Here is mysql log :
160629 4:28:37 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 3059 ...
160629 4:28:37 [Note] Plugin 'FEDERATED' is disabled.
160629 4:28:37 InnoDB: The InnoDB memory heap is disabled
160629 4:28:37 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160629 4:28:37 InnoDB: Compressed tables use zlib 1.2.8
160629 4:28:37 InnoDB: Using Linux native AIO
160629 4:28:37 InnoDB: Initializing buffer pool, size = 512.0M
160629 4:28:37 InnoDB: Completed initialization of buffer pool
160629 4:28:37 InnoDB: highest supported file format is Barracuda.
InnoDB: No valid checkpoint found.
InnoDB: If this error appears when you are creating an InnoDB database,
InnoDB: the problem may be that during an earlier attempt you managed
InnoDB: to create the InnoDB data files, but log file creation failed.
InnoDB: If that is the case, please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/error-creating-innodb.html
160629 4:28:37 [ERROR] Plugin 'InnoDB' init function returned error.
160629 4:28:37 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160629 4:28:37 [ERROR] Unknown/unsupported storage engine: InnoDB
160629 4:28:37 [ERROR] Aborting
160629 4:28:37 [Note] /usr/libexec/mysql55/mysqld: Shutdown complete
160629 04:28:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Here are approach what I have tried yet:
1) Upgraded EC2 t2.micro to t2.medium.
2) Increased innodb_buffer_pool_size in my.cnf
3) Rename ib_logfile0 and ib_logfile1
Please suggest what next step should I do to start mysql.
Edit:
Sudo free -m
total used free shared buffers cached
Mem: 3954 149 3805 0 9 86
-/+ buffers/cache: 53 3901
Edit:
Is there only solution to uninstall mysql and re-install ?
If yes, then how can retain already created databases in mysql? Please Suggest.
Please go through the below link which is being given in your error logs.
It has mentioned certain steps to solve runtime exception .You can cross check and proceed. If nothing helped than you can opt for uninstallation.
http://dev.mysql.com/doc/refman/5.5/en/error-creating-innodb.html
I have a problem with my server. It shuts down every 2 minutes. I am using Digitalocean. I have changed to a newdroplet already it was fine for one month. then. I cannot even start MYSQL service.. this is my log:
160426 17:06:34 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
160426 17:08:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160426 17:08:04 [Note] /usr/libexec/mysqld (mysqld 5.5.48) starting as process 3334 ...
160426 17:08:04 [Note] Plugin 'FEDERATED' is disabled.
160426 17:08:04 InnoDB: The InnoDB memory heap is disabled
160426 17:08:04 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160426 17:08:04 InnoDB: Compressed tables use zlib 1.2.3
160426 17:08:04 InnoDB: Using Linux native AIO
160426 17:08:04 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160426 17:08:04 InnoDB: Completed initialization of buffer pool
160426 17:08:04 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160426 17:08:04 [ERROR] Plugin 'InnoDB' init function returned error.
160426 17:08:04 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160426 17:08:04 [ERROR] Unknown/unsupported storage engine: InnoDB
160426 17:08:04 [ERROR] Aborting
Any help is appreciated..
Fatal error: cannot allocate memory for the buffer pool
How big is your buffer pool? How much memory do you have? Who installed MySQL on the server?
it was fine for one month
What has changed? What is using memory on your server?
For a tiny 0.5GB system, lots of things must be dialed back.
50 http processes -- from what? Apache: drastically decrease MaxClients, perhaps to 10. Others: ?
128MB is too much for 512MB. Try 50M.
max_connections should be perhaps 10.
Are your tables InnoDB? Or MyISAM?
If InnoDB, key_buffer_size = 5M, innodb_buffer_pool_size = 50M.
If MyISAM, key_buffer_size = 40M, innodb_buffer_pool_size = 0.
You have version 5.5.48 of MySQL. Did Digitalocean change it without your knowing it?
I have an Amazon ec2 micro instance that holds a WordPress site. Since yesterday, my mysql db keep crashing saying:
[ec2-user#ip-xxx-xx-xx-xxx ~]$ sudo service mysqld status
mysqld dead but subsys locked
I tried restarting the mysqld but after every 5 mins it keeps crashing with the same message - "Error establishing a database connection"
This is what my DB log file reads -
160123 05:44:41 mysqld_safe mysqld restarted
160123 5:44:43 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 7928 ...
160123 5:44:44 [Note] Plugin 'FEDERATED' is disabled.
160123 5:44:46 InnoDB: The InnoDB memory heap is disabled
160123 5:44:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160123 5:44:46 InnoDB: Compressed tables use zlib 1.2.8
160123 5:44:46 InnoDB: Using Linux native AIO
160123 5:44:46 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160123 5:44:46 InnoDB: Completed initialization of buffer pool
160123 5:44:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160123 5:44:46 [ERROR] Plugin 'InnoDB' init function returned error.
160123 5:44:46 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160123 5:44:46 [ERROR] Unknown/unsupported storage engine: InnoDB
160123 5:44:46 [ERROR] Aborting
160123 5:44:46 [Note] /usr/libexec/mysql55/mysqld: Shutdown complete
160123 05:44:46 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
The common solution that found out there when I Googled for a solution for the issue - mysqld dead but subsys locked, is to create a swapfile. I did exactly as most of the solution suggest but since I'm not well versed with UNIX, I don't know what I have done is right or not.
Or, Is there any other solution for this issue. Kindly help.
Have you tried this mysql is dead but subsys locked
Seems quite straightforward. Also please consider moving to RDS instead of running a mysql server in ec2. Saves you from troubles of this kind.
Some background. I hosted a WordPress site in VPS and sometimes MYSQL down with error "Error Establishing a Database Connection". I've spent some time to research and believe the problem is due to when MySQL get restarted, it is not able to allocate enough RAM to proceed.
I believe I can improve the situation by increasing physical RAM or swap. But my question for this post is, why MySQL need to restart itself? My site is with pretty low traffic and doesn't seem like the DB is corrupted.
Below is the full log for this issue:
160103 18:39:54 mysqld_safe Number of processes running now: 0
160103 18:39:54 mysqld_safe mysqld restarted
160103 18:39:55 [Note] /usr/libexec/mysqld (mysqld 5.5.44-MariaDB) starting as process 22061 ...
160103 18:39:55 InnoDB: The InnoDB memory heap is disabled
160103 18:39:55 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160103 18:39:55 InnoDB: Compressed tables use zlib 1.2.7
160103 18:39:55 InnoDB: Using Linux native AIO
160103 18:39:55 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
160103 18:39:55 InnoDB: Completed initialization of buffer pool
160103 18:39:55 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160103 18:39:56 [ERROR] Plugin 'InnoDB' init function returned error.
160103 18:39:56 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
160103 18:39:56 [Note] Plugin 'FEEDBACK' is disabled.
160103 18:39:56 [ERROR] Unknown/unsupported storage engine: InnoDB
160103 18:39:56 [ERROR] Aborting
When Number of Processes Running Now 0 means that MySQL isn't running. So the daemon mysqld_safe "makes you a favor" and start MySQL
You have assigned very low RAM 128MB to innodb_buffer_pool_size (which is default RAM). So you should assign approx. 80% of total RAM to this variable if you are using innodb engine, as mysql uses intially memory from this variable to cache index as well data in innodb engine.
So update at least 1 GB RAM (should be 80% of total ram for innodb) to innodb_buffer_pool_size in your config file and restart mysql service.
Update:
You have 1 GB RAM: You can assign 800M RAM (or >=500M) to innodb_buffer_pool_size.
MySQL will auto restart when number of running process 0: As per errors shared by you...this error is coming at the time of mysql service start and server is not able to initiate mysql service with such a less RAM.
160103 18:39:55 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
160103 18:39:55 InnoDB: Completed initialization of buffer pool
160103 18:39:55 InnoDB: Fatal error: cannot allocate memory for the buffer pool
I had a similar problem with MySQL on CentOS 6.9. It was caused by memory starving due another PHP process and not by MySQL. MySQL was just a victim.
You can look at kernel messages. The location of this messages may be specific to OS. In case of CentOS 6.x they are located in /var/log/messages .
Here are some related messages from /var/log/messages:
Jul 25 20:34:46 myserver kernel: Out of memory: Kill process 21467 (mysqld) score 30 or sacrifice child
Jul 25 20:34:46 myserver kernel: Killed process 21467, UID 497, (mysqld) total-vm:757004kB, anon-rss:17728kB, file-rss:320kB
You can run the following command to see if kernel run out of memory:
cat /var/log/messages | grep out_of_memory
I have this error log from MySQL, any idea?
Website works for some time and then I get MySQL shutdown completely after a couple of hours.
140919 10:48:27 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140919 10:48:27 [Note] Plugin 'FEDERATED' is disabled.
140919 10:48:27 InnoDB: The InnoDB memory heap is disabled
140919 10:48:27 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140919 10:48:27 InnoDB: Compressed tables use zlib 1.2.3.4
140919 10:48:28 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140919 10:48:28 InnoDB: Completed initialization of buffer pool
140919 10:48:28 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140919 10:48:28 [ERROR] Plugin 'InnoDB' init function returned error.
140919 10:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140919 10:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB
140919 10:48:28 [ERROR] Aborting
140919 10:48:28 [Note] /usr/sbin/mysqld: Shutdown complete
140919 10:48:28 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140919 10:48:28 [Note] Plugin 'FEDERATED' is disabled.
140919 10:48:28 InnoDB: The InnoDB memory heap is disabled
140919 10:48:28 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140919 10:48:28 InnoDB: Compressed tables use zlib 1.2.3.4
140919 10:48:28 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140919 10:48:28 InnoDB: Completed initialization of buffer pool
140919 10:48:28 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140919 10:48:28 [ERROR] Plugin 'InnoDB' init function returned error.
140919 10:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140919 10:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB
140919 10:48:28 [ERROR] Aborting
140919 10:48:28 [Note] /usr/sbin/mysqld: Shutdown complete
TLDR;
Mysql can't restart because it's out of memory, check that you have an appropriate swapfile configured.
Didn't help? If that's not your issue, more qualified questions to continue research are:
mysqld service stops once a day on ec2 server
https://askubuntu.com/questions/422037/optimising-mysql-settings-mysqld-running-out-of-memory
Background
I had exactly this problem on the very first system I set up on EC2, characterised by the wordpress site hosted there going down on occasion with "Error establishing database connection".
The logs showed the same error that the OP posted. My reading of the error (timestamps removed) is:
Out of memory error:
InnoDB: Fatal error: cannot allocate memory for the buffer pool
InnoDB can't start without enough memory
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting
mysqld is shutting down, which in this context, really means failing to restart!
[Note] /usr/sbin/mysqld: Shutdown complete
Checking /var/log/syslog and searching for mysql yields:
Out of memory: Kill process 15452 (mysqld) score 93 or sacrifice child
Killed process 15452 (mysqld) total-vm:888672kB, anon-rss:56252kB, file-rss:0kB
init: mysql main process (15452) killed by KILL signal
init: mysql main process ended, respawning
type=1400 audit(1443812767.391:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=21984 comm="apparmor_parser"
init: mysql main process (21996) terminated with status 1
init: mysql main process ended, respawning
init: mysql post-start process (21997) terminated with status 1
<repeated>
Note: you may have to gunzip and search through archived logs if the error occurred before the logs were rotated by cron.
Solution
In my case the underlying issue was that I'd neglected to configure a swapfile.
You can check to see if you have one configured by running free -m.
total used free shared buffers cached
Mem: 604340 587364 16976 0 29260 72280
-/+ buffers/cache: 485824 118516
Swap: 0 0 0
In the example above, Swap: 0 indicates no swapfile.
Tutorials on setting one up:
https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04
https://help.ubuntu.com/community/SwapFaq
Note that bigger is not necessarily better! From the Ubuntu guide:
The "diminishing returns" means that if you need more swap space than twice your RAM size, you'd better add more RAM as Hard Disk Drive (HDD) access is about 10³ slower then RAM access, so something that would take 1 second, suddenly takes more then 15 minutes! And still more then a minute on a fast Solid State Drive (SSD)...
Regarding the other answers here...
The InnoDB memory heap is disabled
This isn’t really an error, just an indication that InnoDB is using the system’s internal memory allocator instead of its own. The default is yes/1, and is acceptable for production.
According to the docs, this command is deprecated, and will be removed in MySQL versions above 5.6 (and I assume MariaDB):
http://dev.mysql.com/doc/refman/5.6/en/innodb-performance-use_sys_malloc.html
Thanks to: Ruben Schade comment
[Note] Plugin 'FEDERATED' is disabled.
The message about FEDERATED disabled is not an error. It just meant that the FEDERATED engine its not ON for your mysql server. It's not used by default. If you don't need it, don't care about this message.
See: https://stackoverflow.com/a/16470822/2586761
I found this answer adds to the discussion: https://www.digitalocean.com/community/questions/mysql-server-keeps-stopping-unexpectedly?answer=26021
In short, on top of setting innodb_buffer_pool_size to something reasonable like 64M, you also may need to modify /etc/apache2/mods-enabled/mpm_prefork.conf to reduce the number of connections started by apache;
<IfModule mpm_prefork_module>
StartServers 3
MinSpareServers 3
MaxSpareServers 5
MaxRequestWorkers 25
MaxConnectionsPerChild 1024
</IfModule>
The solution is NOT more space, Problem is Apache web server not mysql, actually you need to decrease innodb-buffer-pool-size
This buffer is used by the mysql process right off the start, so when Apache needs more resources the kernel will clear RAM from services this means stopping mysql instead of crashing the server.
Would also add a CRON to check the db status and restart it if you dont want to change to ngnx or httplight.