Every day or so our Wordpress sites stop responding, the pages begin returning the dreaded 'Error establishing database connection'. There is nothing in the MySQL logs, and I'm at a loss as to what could be causing the issue. We don't have a lot of site visitors, and the machine is a Medium EC2 instance. Anyone have any ideas on how to resolve this?
There's not a whole lot to work with here. But ... I had the same issue with my micro instance. My problem was that the server kept running out of memory and then the mysql server would stop. It would start again when restarting the computer but it was only a matter of time before it would crash again.
Here's what I was getting in my MySQL logs.
151023 6:15:44 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
151023 6:15:44 InnoDB: Completed initialization of buffer pool
151023 6:15:44 InnoDB: Fatal error: cannot allocate memory for the buffer pool
151023 6:15:44 [ERROR] Plugin 'InnoDB' init function returned error.
151023 6:15:44 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151023 6:15:44 [ERROR] Unknown/unsupported storage engine: InnoDB
151023 6:15:44 [ERROR] Aborting
You might want to check for something similar. I use Ubuntu and the log is at /var/log/mysql/ by default.
I solved the problem by setting up a swap file as per Amazon EC2, mysql aborting start because InnoDB: mmap (x bytes) failed; errno 12. The AWS instances don't come with a swap space setup by default (whereas the install I downloaded from Ubuntu back in the day did). You need to set it up manually. Here's the method -
ssh into your AWS instance. Then:
Run dd if=/dev/zero of=/swapfile bs=1M count=1024
Run mkswap /swapfile
Run swapon /swapfile
Add this line /swapfile swap swap defaults 0 0 to /etc/fstab
Read the linked question for more details. Hope that helps!
I had a similar issue with intermittent crashing of MySQL. It turned out to be Apache configurations. Bots were brute forcing the site and eventually caused Apache to crash (check your logs: $ cat /var/log/apache2/access.log). Apache automatically restarts, but there isn't enough spare memory to restart MySQL too, hence the Database Connection error. The short fix is to reduce the number of RequestWorkers in Apache to better fit the amount of RAM you have.
You can run a diagnostic on your apache configuration using Apache2Buddy. It'll calculate how many Apache Workers you can run given the amount of RAM you have and how big your application is:
$ curl -L http://apache2buddy.pl/ | perl
It's probably going to recommend changing MaxRequestWorkers (or MaxClients on older Apache systems) in your MPM-Prefork configurations. That file is at /etc/apache2/mods-available/mpm_prefork.conf on my system. After changing the value to what Apache2Buddy recommends, just restart Apache and you should be good to go.
I wrote an article on this situation if you want a deeper explanation, a method to stress test, or ideas on how to block some of the bot traffic: http://brunzino.github.io/blog/2016/05/21/solution-how-to-debug-intermittent-error-establishing-database-connection/
When I tried to install local wordpress the same error
error establishing database connection
occurred because I forget to stop the SQL and Apache which was started in xampp. I stopped it and reinstalled the wordpress for Windows and it worked.
Related
Version: Mysql server version is 8.0 and it is installed on Windows server 2019.
Problem statement: Error log file from location C:\programdata\mysql80\data\ is increasing and it is about ~100GB.
Error log statements are :
[Warning] [MY-011959] [InnoDB] Difficult to find free blocks in the buffer pool (6091 search iterations)! 6091 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 1738 OS file reads, 217 OS file writes, 45 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
Scenario:
There was failure to update more than 4MB data size to the database, so we changed the "max_allowed_packet" from default 4M to 256M in "mysql.ini" file.
After this settings restarted mysqld service, everything worked after this db changed.
But after 2 days we are facing the above mentioned issue, as a result disk space is getting full and DB connections are pending in a queue.
Tried to stop the mysql80 service and deleted the error.log file and reverted the max_allowed_packet to its default size.
But with the change also error file is getting created again with same warning.
What could be the possible issues or fix for it?
I have this really strange problem.
To give you some context, I've have a script that setsup my dev env which contains mysql and percona server and some other things using brew on OSX.
I've re-runned it to do some reconfiguration that got messed up.
But after the re-run mysql won't stay up. Every time I check the mysqld pid I get a different result. I checked the mysql logs and the problem is "mysqld_safe A mysqld process already exists".
So I decided to delete everything related to mysql. I followed the gist from https://gist.github.com/vitorbritto/0555879fe4414d18569d but somehow, the mysql and mysqld process still starts after I restart my laptop.
How is this possible? Has anyone else had this problem? How do I fix it without reinstalling OSX?
So tadman was right. I managed to get my environment to work, I still don't know what that zombie process is, but because it's not affecting me, I don't care.
The problem was that my script was installing the latest version of MySQL and percona-server#5.6. and there is a conflict with InnoDB.
After reinstalling manually everything in the MySQL log there was something along the lines:
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting
After installing MySQL#5.6 everything seems to work as expected.
Thanks tadman.
i have the following Issue.
I have an AWS EC2 t2.micro instance with LAMP installed on it.
My WebApp uses InnoDB tables. If there are a lot of users, InnoDB used to crash because of buffer pool size. It says it cannot allocate enought memory.
170511 10:32:05 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
I have put innodb_buffer_pool_size to 750M (if I put it more that 1GB - mysql doesn't start at all). And I have put stress test of my WebApp under LoadImpact. From 30 person and up WebApp was trying to allocate something like 800M and crashes MySQL again.
170511 12:16:04 InnoDB: Initializing buffer pool, size = 752.0M
InnoDB: mmap(807010304 bytes) failed; errno 12
Another problem is that I cannot just run sudo service mysql start or restart after mysql Crash. It says job failed to start.
How to make my server more stable. What options should I use.
How to prevent such "crashy" behaviour. I mean can I make something that makes server and mysql prosess not chash but freeze or something like that. I dont want do sudo reboo every time there are a lot of people on the website
t2.micro instance has only 512M RAM. If you run MySQL and Apache on same box you should allocate memory sparingly.
It looks like apache processes use up all memory so MySQL gets less than necessary minimum.
I would suggest to start with adding swap if you haven't added it yet.
dd if=/dev/zero of=/swapfile bs=1024 count=65536
chmod 0600 /swapfile
mkswap /swapfile
swapon /swapfile
and add it to /etc/fstab.
/swapfile swap swap defaults 0 0
Next, limit Apache workers to make sure MySQL always gets innodb_buffer_pool_size. Depending on what you use - prefork or worker configuration will be different, but you got an idea.
Downloaded MAMP4 and installed over MAMP1.9.4 (I know, it's old) with Mac OSX Sierra. Apache server launches, but not MySQL. Warned that previous db was copied to a folder named mysql56 and that on starting up MySQL need to convert db. But MySQL doesn't even start up! Activity Monitor shows no other mysql process going on. MAMP error log:
2017-01-16 12:35:13 47394 [ERROR] InnoDB: auto-extending data file ./ibdata1 is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages!
2017-01-16 12:35:13 47394 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2017-01-16 12:35:13 47394 [ERROR] Plugin 'InnoDB' init function returned error.
2017-01-16 12:35:13 47394 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-01-16 12:35:13 47394 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-01-16 12:35:13 47394 [ERROR] Aborting
I tried using the same ports as I used with MAMP1.9.4 (80 & 3306), then switched to default. Webserver address was pointing to my directory where I keep my websites. Changed it back to Applications/MAMP/htdocs. Nothing is working. :( Any help would be very much appreciated.
Oh, on my old machine running Snow Leopard, it's all still working with MAMP1.9.4.
Found the solution at http://rat32.com/rat32/2016/11/29/how-to-solve-mysql-not-starting-in-mamp-problem/#comment-175889. Removed all files except folders from Applications/MAMP/db/mysql56. Then restarted. MAMP immediately connected to the MySQL server and prompted me to upgrade my databases. All is well.
I tried other suggestions. The most common was to check for any other MySQL processes that might be running and to kill them via Terminal command. Another suggestion was to rename the files ib_logfile0 and ib_logfile1 in /Library/Application Support/MAMP PRO/db/mysql56, but as I had not initiated MAMP PRO, that folder did not exist. I had renamed the two files in Applications/MAMP/db/mysql56, but that didn't solve the problem. Only when I actually removed those two files and a third, then MAMP was able to connect to the MySQL server.
So it seems silly to answer my own question, but just want to put it here in case others run into the same problem.
Using MAMP 4.1 with Mac OSX Sierra.
I have problem starting the MySql server.
The log says:
InnoDB: Error in opening ./ibdata1
111220 16:16:43 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
/usr/libexec/mysqld: Disk is full writing './mysql-bin.000028' (Errcode: 28). Waiting for someone to free space... Retry
in 60 secs
After I check the disk - it says it's full.
So, after searching for solution - I found that I need to purge the binary log.
However, in order to purge - I need to start the MySql server, but all the spacein the disk is taken by the binary log, so I can't start...
It's also not advised to simply delete the binary logs.
So, I am kind of stuck.
Can't run the mysql to purge logs and can't purge logs because can't run server.
Any help? :)
Edit: The disk contains only the logs, there's nothing else.
If the disk is ext[2|3|4] you can use tune2fs to set the portion of the disk reserved for root to 0, giving you maybe enough breathing room to start the server
this would be tune2fs -m 0 /dev/whatever (after unmounting, ofcourse)
Try to start the mysql server with the option --expire_logs_days=, it should delete the log and older than days directory at startup.
bye
Gianluca