I'm having a very weird issue, I have an Apache server running with mod_wsgi. The website runs fine but every once in a while I get the
IOError: failed to write data
error on all the pages of the website. I then gets solved with
sudo service mysqld restart
Since the website can't be down for long, I have not time to debug this problem and I just run the command every time this happens. I only see the error in the logs that's why I can't really debug it, and it has no clear replication steps, it just occurs randomly.
Any help would be appreciated and let me know if you need me to post any configuration files.
Edit: The exact error displayed by django is:
(2002, "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)")
I saved the error message and it is hosted here. (passwords edited out)
Edit:
Here is an extract from the mysql server error log.
160610 10:51:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160610 10:51:53 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 7658 ...
160617 14:35:47 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 32054 ...
160617 14:35:47 [Note] Plugin 'FEDERATED' is disabled.
160617 14:35:47 InnoDB: The InnoDB memory heap is disabled
160617 14:35:47 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160617 14:35:47 InnoDB: Compressed tables use zlib 1.2.8
160617 14:35:47 InnoDB: Using Linux native AIO
160617 14:35:47 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160617 14:35:47 InnoDB: Completed initialization of buffer pool
160617 14:35:47 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160617 14:35:47 [ERROR] Plugin 'InnoDB' init function returned error.
160617 14:35:47 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160617 14:35:47 [ERROR] Unknown/unsupported storage engine: InnoDB
160617 14:35:47 [ERROR] Aborting
I saved the full mysqld.log here.
Why it crashes
This seems to be the culprit
160617 14:35:47 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
This condition is caused by the server running out of physical memory. That's what errno 12 stands for in the kernel errors.
This topic has been discussed on both Stack Overflow and dba.stackexchange. If you want to simulate low memory situations or try to manually reproduce the error try some of these tools:
How to fill 90% of the free memory?
Quick Solution
If you are able to upgrade memory, you can try that. If not, you can try creating a large swap file. It's possible that you don't have a swap at all. Some AWS EC2 instances don't have one by default. You can find out by typing top in the shell. If you don't see swap near the top, that means you don't have one.
A swap file would make the queries a lot slower but at least it's better than the site getting offline.
You might be tempted to try to modify the systemd files to make mysql auto start. Update: #PeterBrittain points out that mysql is auto restarting anyway as shown by the logs. Sometimes though databases can take a bit of time to restart and if the data get's corrupted it will refuse to restart.
Why is memory being exhausted?
If you don't have any other server running on it, 2GB will be more than enough to host a site that serves 6000 pages per day. It could be that you have some rather heavy queries that put an unnecessary load on the db. There are some remedial actions that can be taken
Use django-debug-toolbar to identify pages that execute many queries and see if select_related or prefetch_related can be used to reduce the number.
use mysql slow query log to find the queries that take a long time to execute and optimize them.
Use caching to save the results of complex queries.
Related
I have this error log from MySQL, any idea?
Website works for some time and then I get MySQL shutdown completely after a couple of hours.
140919 10:48:27 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140919 10:48:27 [Note] Plugin 'FEDERATED' is disabled.
140919 10:48:27 InnoDB: The InnoDB memory heap is disabled
140919 10:48:27 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140919 10:48:27 InnoDB: Compressed tables use zlib 1.2.3.4
140919 10:48:28 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140919 10:48:28 InnoDB: Completed initialization of buffer pool
140919 10:48:28 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140919 10:48:28 [ERROR] Plugin 'InnoDB' init function returned error.
140919 10:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140919 10:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB
140919 10:48:28 [ERROR] Aborting
140919 10:48:28 [Note] /usr/sbin/mysqld: Shutdown complete
140919 10:48:28 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140919 10:48:28 [Note] Plugin 'FEDERATED' is disabled.
140919 10:48:28 InnoDB: The InnoDB memory heap is disabled
140919 10:48:28 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140919 10:48:28 InnoDB: Compressed tables use zlib 1.2.3.4
140919 10:48:28 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140919 10:48:28 InnoDB: Completed initialization of buffer pool
140919 10:48:28 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140919 10:48:28 [ERROR] Plugin 'InnoDB' init function returned error.
140919 10:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140919 10:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB
140919 10:48:28 [ERROR] Aborting
140919 10:48:28 [Note] /usr/sbin/mysqld: Shutdown complete
TLDR;
Mysql can't restart because it's out of memory, check that you have an appropriate swapfile configured.
Didn't help? If that's not your issue, more qualified questions to continue research are:
mysqld service stops once a day on ec2 server
https://askubuntu.com/questions/422037/optimising-mysql-settings-mysqld-running-out-of-memory
Background
I had exactly this problem on the very first system I set up on EC2, characterised by the wordpress site hosted there going down on occasion with "Error establishing database connection".
The logs showed the same error that the OP posted. My reading of the error (timestamps removed) is:
Out of memory error:
InnoDB: Fatal error: cannot allocate memory for the buffer pool
InnoDB can't start without enough memory
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting
mysqld is shutting down, which in this context, really means failing to restart!
[Note] /usr/sbin/mysqld: Shutdown complete
Checking /var/log/syslog and searching for mysql yields:
Out of memory: Kill process 15452 (mysqld) score 93 or sacrifice child
Killed process 15452 (mysqld) total-vm:888672kB, anon-rss:56252kB, file-rss:0kB
init: mysql main process (15452) killed by KILL signal
init: mysql main process ended, respawning
type=1400 audit(1443812767.391:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=21984 comm="apparmor_parser"
init: mysql main process (21996) terminated with status 1
init: mysql main process ended, respawning
init: mysql post-start process (21997) terminated with status 1
<repeated>
Note: you may have to gunzip and search through archived logs if the error occurred before the logs were rotated by cron.
Solution
In my case the underlying issue was that I'd neglected to configure a swapfile.
You can check to see if you have one configured by running free -m.
total used free shared buffers cached
Mem: 604340 587364 16976 0 29260 72280
-/+ buffers/cache: 485824 118516
Swap: 0 0 0
In the example above, Swap: 0 indicates no swapfile.
Tutorials on setting one up:
https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04
https://help.ubuntu.com/community/SwapFaq
Note that bigger is not necessarily better! From the Ubuntu guide:
The "diminishing returns" means that if you need more swap space than twice your RAM size, you'd better add more RAM as Hard Disk Drive (HDD) access is about 10³ slower then RAM access, so something that would take 1 second, suddenly takes more then 15 minutes! And still more then a minute on a fast Solid State Drive (SSD)...
Regarding the other answers here...
The InnoDB memory heap is disabled
This isn’t really an error, just an indication that InnoDB is using the system’s internal memory allocator instead of its own. The default is yes/1, and is acceptable for production.
According to the docs, this command is deprecated, and will be removed in MySQL versions above 5.6 (and I assume MariaDB):
http://dev.mysql.com/doc/refman/5.6/en/innodb-performance-use_sys_malloc.html
Thanks to: Ruben Schade comment
[Note] Plugin 'FEDERATED' is disabled.
The message about FEDERATED disabled is not an error. It just meant that the FEDERATED engine its not ON for your mysql server. It's not used by default. If you don't need it, don't care about this message.
See: https://stackoverflow.com/a/16470822/2586761
I found this answer adds to the discussion: https://www.digitalocean.com/community/questions/mysql-server-keeps-stopping-unexpectedly?answer=26021
In short, on top of setting innodb_buffer_pool_size to something reasonable like 64M, you also may need to modify /etc/apache2/mods-enabled/mpm_prefork.conf to reduce the number of connections started by apache;
<IfModule mpm_prefork_module>
StartServers 3
MinSpareServers 3
MaxSpareServers 5
MaxRequestWorkers 25
MaxConnectionsPerChild 1024
</IfModule>
The solution is NOT more space, Problem is Apache web server not mysql, actually you need to decrease innodb-buffer-pool-size
This buffer is used by the mysql process right off the start, so when Apache needs more resources the kernel will clear RAM from services this means stopping mysql instead of crashing the server.
Would also add a CRON to check the db status and restart it if you dont want to change to ngnx or httplight.
I am having a problem with MySQL on my Rackspace Centos 6.4 server instance. The problem is similar to the one described in this StackOverflow question. MySQL is being restarted automatically at some point by mysqld_safe, and the restart fails because InnoDB tries to allocate 128Mb of RAM, which fails. The output of mysqld.log is as follows:
140129 18:05:26 mysqld_safe Number of processes running now: 0
140129 18:07:30 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140129 18:07:30 InnoDB: Compressed tables use zlib 1.2.3
140129 18:07:30 InnoDB: Using Linux native AIO
140129 18:07:35 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140129 18:07:46 InnoDB: Completed initialization of buffer pool
140129 18:07:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140129 18:07:47 [ERROR] Plugin 'InnoDB' init function returned error.
140129 18:07:47 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140129 18:08:07 [ERROR] Unknown/unsupported storage engine: InnoDB
140129 18:08:10 [ERROR] Aborting
140129 18:08:53 [Note] /usr/libexec/mysqld: Shutdown complete
140129 18:18:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
The solution provided in that other question seemed to be one of "create a swap file". I have checked my server and it seems there is already an active swap file:
# swapon -s
Filename Type Size Used Priority
/dev/xvdc1 partition 499992 34876 -1
and, looking at that output, it is the size I was thinking I needed (512Mb). For completeness, here are the contents of my /etc/fstab file:
/dev/xvda1 / ext3 defaults,noatime,barrier=0 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/xvdc1 none swap sw 0 0
So am I missing something, or do I already have a working swap file, of about 512Mb, which is reasonably empty and therefore should be capable of handling a request for 128Mb? Should I reduce the size of the InnoDB buffer to, say, 64Mb? Will there be any issues related to shrinking this buffer?
(My Rackspace server is the smallest one available, which has 512Mb RAM. Whenever I do a top on the server, it seems to have between 50 and 80 Mb free.)
From the output it does look like you have approximately 488MB of swap space.
I am not sure if MySQL allocates the innodb buffer pool based off how much memory+swap is free. Even if it did, you would want to avoid it going to swap as it is slower than keeping things in RAM. My guess is that it does not include swap.
The error "InnoDB: mmap(137363456 bytes) failed; errno 12" lets us know that you could not allocate the memory.
# perror 12
OS error code 12: Cannot allocate memory
I would reduce the size of the innodb buffer pool to 64MB; see if that works. If it does not, either increase the size of the cloud server or reduce the size of the buffer pool again.
In general InnoDB likes memory. It tries to keep as much of the data in memory as possible and reduce disk IO. By reducing the buffer size, you decrease how much MySQL can keep in memory. MySQL will go to disk more often to retrieve data.
This number should reflect the size of your data set. No point in having a buffer much larger than your actual data set.
You may be able to use some of the queries here to determine the size. http://www.mysqlperformanceblog.com/2008/03/17/researching-your-mysql-table-sizes/
If your data set is much larger than the amount of RAM you normally have on hand, you may need to increase the size of the server itself.
I have LAMP setup running on ubuntu. I am using this setup to run torrentflux with a web interface that requires mysql.
All works as expected for a few days, and then I get 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)'
I rebooted to fix the problem, but now the inconvenience of doing so has surpassed the reward of solving the problem, and I would like to find another way to solve the problem.
Mysql is the only failure in the LAMP setup for torrentflux. Torrentflux still leeches and seeds as it should; I simply lose the control interface without mysql. Apache still does its thing as expected.
Yes, mysql is running until the error is output when I try to use the web interface, and then it is impossible to restart the mysql server.
$ sudo service mysql start
start: Job failed to start
Ubuntu version:
Distributor ID: Ubuntu
Description: Ubuntu 12.04.1 LTS
Release: 12.04
Codename: precise
G/usr/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
130126 14:42:15 [Note] Plugin 'FEDERATED' is disabled.
130126 14:42:15 InnoDB: The InnoDB memory heap is disabled
130126 14:42:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130126 14:42:15 InnoDB: Compressed tables use zlib 1.2.3.4
130126 14:42:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
130126 14:42:16 InnoDB: Completed initialization of buffer pool
130126 14:42:16 InnoDB: Fatal error: cannot allocate memory for the buffer pool
130126 14:42:16 [ERROR] Plugin 'InnoDB' init function returned error.
130126 14:42:16 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
130126 14:42:16 [ERROR] Unknown/unsupported storage engine: InnoDB
130126 14:42:16 [ERROR] Aborting
mysql was running until I shut it down.
Rebooting fixes the problem for a while. I wish for a less drastic solution.
EDIT: You changed your question around a bit, so moved the old answer down below.
Either way, what you're getting in the log is fairly clear;
130126 14:42:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
130126 14:42:16 InnoDB: Fatal error: cannot allocate memory for the buffer pool
errno 12 means you're out of memory. MySQL tries to allocate 128MB for the buffer pool, but can't find that much free memory.
I'm not sure what kind of machine you're running on, but you basically have 3 options;
Find whichever application is hogging memory and see if you can update or swap it out.
Get more memory for the machine (probably the best option if the first fails)
Add some swap space (may slow things down quite a bit)
Old answer before edit;
You're getting that message because you're trying to start MySQL as your normal user, not as root as required.
Try;
sudo service mysql start
Add this to your my.ini:
[mysqld]
innodb=OFF
ignore-builtin-innodb
skip-innodb
default-storage-engine=myisam
default-tmp-storage-engine=myisam
it worked for me :)
Hi mistakenly I have deleted my MySQL database from Windos XP based PC (this database was used to host a Wiki service using Xampp package). Now I am trying to restore the data so that I can atleast retrieve the TEXT entered in the respective table of Mediawiki package.
After deletion I tried to recover the data using Minitool data recovery software and was able to recover the main schema folder containing .frm files, ib_logfile0, ib_logfile1 and ibdata1 (which was stored under C>xampp>mysql>data).
Now I am trying to restore the database so that I can take a SQL dump and use it accordingly. I have consulted some previous suggestions provided by the experts. Below are the links -
Restoring MySQL InnoDB files on Windows
Stackoverflow answers
But these did not help me very much, not sure what exactly I am missing from the instructions. At the end I was unable to start MySQL service and it failed with the below errors in log -
120823 12:17:04 [Note] Plugin 'FEDERATED' is disabled.
120823 12:17:04 InnoDB: The InnoDB memory heap is disabled
120823 12:17:04 InnoDB: Mutexes and rw_locks use Windows interlocked functions
120823 12:17:04 InnoDB: Compressed tables use zlib 1.2.3
120823 12:17:04 InnoDB: Initializing buffer pool, size = 16.0M
120823 12:17:04 InnoDB: Completed initialization of buffer pool
120823 12:17:04 InnoDB: highest supported file format is Barracuda.
InnoDB: No valid checkpoint found.
InnoDB: If this error appears when you are creating an InnoDB database,
InnoDB: the problem may be that during an earlier attempt you managed
InnoDB: to create the InnoDB data files, but log file creation failed.
InnoDB: If that is the case, please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/error-creating-innodb.html
120823 12:17:04 [ERROR] Plugin 'InnoDB' init function returned error.
120823 12:17:04 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
120823 12:17:04 [ERROR] Unknown/unsupported storage engine: InnoDB
120823 12:17:04 [ERROR] Aborting
Any help to recover from this situation will be highly appreciated. I am willing to provide some more data if required. [MySQL Server Version 5.5, xampp-win32-1.8.0-VC9]
Oh God finally it's solved :)
Actually I am a novice that's why I could not solve this quickly. Here is the workflow I followed.
I installed a fresh copy of MySQL server. Run Wizard config for transactional database.
I stopped MySQL from Services.msc
Then overwrote the ib* and *frm files (under the schema folder) in C/programdata/mysql/mysql server 5.5/data folder
Updated the my.ini in C/apps/mysql with the innodb_log_file_size=<actual size of iblogfile0>
ran the command mysqld --defaults-file="C:/apps/mysql/my.ini" --standalone --console --innodb_force_recovery=6
once the above completed closed cmd.exe
restarted MySQL service
Opened MySQL workbench and took a dump
Sorry to confuse you all. Thanks again. Let me know if anybody requires any clarity for the above steps.
I tried to change the data folder path of MySQL by changing the datadir parameter in the my.ini file. But when I restarted MySQL service again I get the following error :
Error 1067: the process terminated unexpectedly.
I checked the .err file, the following errors are present: (they are all about InnoDB)
120319 3:59:59 [Note] Plugin 'FEDERATED' is disabled.
120319 3:59:59 InnoDB: The InnoDB memory heap is disabled
120319 3:59:59 InnoDB: Mutexes and rw_locks use Windows interlocked functions
120319 3:59:59 InnoDB: Compressed tables use zlib 1.2.3
120319 3:59:59 InnoDB: Initializing buffer pool, size = 75.0M
120319 3:59:59 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file .\ib_logfile0 is of different size 0 99614720 bytes
InnoDB: than specified in the .cnf file 0 39845888 bytes!
120319 3:59:59 [ERROR] Plugin 'InnoDB' init function returned error.
120319 3:59:59 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
120319 3:59:59 [ERROR] Unknown/unsupported storage engine: INNODB
120319 3:59:59 [ERROR] Aborting
You have to specify the right size for:
innodb_log_file_size=95M in your my.cnf or my.ini. It is specified as 38M currently I guess.
Also check from your data folder as how many ib_logfiles you have. If you have more than 2, specify also: innodb_log_files_in_group=2.
Is this on Windows? If so one could use a VHD as storage device for MariaDB/MySQL. I've done so to prevent loss of data whenever I install or update the DB software (in Wamp, Xampp, Mamp, Wampserver etc.)
In order to do so I created a VHD with diskpart and mount it in boot time to the data folder in the respective MariaDB or MySQL folder (moving the original to data.orig first and copying it back as soon the device is running)
I use this script to mount the vhd at boot:
diskpart /s "C:\ProgramData\DB\connectvdisk\attach_mariadb-data.diskpart"
where attach_mariadb-data.diskpart contains
select vdisk file="c:\ProgramData\DB\mariadb-daten.vhdx"
attach vdisk
and the vhdx was set to mount to the proper location in Disk Manager.
The script is executed at boot time (trigger: At Startup) with Task Scheduler running cmd.exe with the following as parameter
/c "C:\ProgramData\DB\connectvdisk\connect-vhd.cmd"
I hijacked the ProgramData folder for this kind of storage. On some systems I've moved the scripts and device to secondary storage (D:) to be sure it survives a OS reinstall.
This continues to work on Windows 11