Version: Mysql server version is 8.0 and it is installed on Windows server 2019.
Problem statement: Error log file from location C:\programdata\mysql80\data\ is increasing and it is about ~100GB.
Error log statements are :
[Warning] [MY-011959] [InnoDB] Difficult to find free blocks in the buffer pool (6091 search iterations)! 6091 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 1738 OS file reads, 217 OS file writes, 45 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
Scenario:
There was failure to update more than 4MB data size to the database, so we changed the "max_allowed_packet" from default 4M to 256M in "mysql.ini" file.
After this settings restarted mysqld service, everything worked after this db changed.
But after 2 days we are facing the above mentioned issue, as a result disk space is getting full and DB connections are pending in a queue.
Tried to stop the mysql80 service and deleted the error.log file and reverted the max_allowed_packet to its default size.
But with the change also error file is getting created again with same warning.
What could be the possible issues or fix for it?
Related
I have a mysql database instance with more than 3000 database inside. Each database contains more than 200 tables. I have more than 100 gb of data in all these database at present. I am using windows server 2012R2 operating system with a 4GB of RAM. The RAM memory utilization of the server system was always showing very high. So I tried to restart the system and restart is not working. It is showing restarting for long time and not restarting. When i checked the logs I understood that there is a memory issue. I want to restart my mysql instance and continue. What is the best configuration for the mysql with above architecture? what i need to do to make this work with out failure in future?
[Warning] InnoDB: Difficult to find free blocks in the buffer pool (1486 search iterations)! 1486 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 26099 OS file reads, 1 OS file writes, 1 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
I have a droplet on DigitalOcean created using Laravel Forge and since a few days ago the MySQL server just crashes and the only way to make it work again is by rebooting the server (MySQL makes the server unresponsive).
When I type htop to see the list of processes is showing a few of /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysql.pid (currently is showing 33 of them).
The error log is bigger than 1GB (yes, I know!) and shows this message hundreds of times:
[Warning] InnoDB: Difficult to find free blocks in the buffer pool (21
search iterations)! 21 failed attempts to flush a page! Consider
increasing the buffer pool size. It is also possible that in your Unix
version fsync is very slow, or completely frozen inside the OS kernel.
Then upgrading to a newer version of your operating system may help.
Look at the number of fsyncs in diagnostic info below. Pending flushes
(fsync) log: 0; buffer pool: 0. 167678974 OS file reads, 2271392 OS
file writes, 758043 OS fsyncs. Starting InnoDB Monitor to print
further diagnostics to the standard output.
This droplet has been running during 6 months but this problem only started last week. The only thing that changed recently is now we send weekly notifications to customers (only the ones that subscribed to it) to let them know about certain events happening in the current week. This is kind of a intensive process, because we have a few thousands of customers, but we take advantage of Laravel Queues in order to process everything.
Is this a MySQL-settings related issue?
Try increasing innodb_buffer_pool_size in my.cnf
The recommendation for a dedicated DB server is 80% - if you're already at that level then you should consider moving to a bigger instance type.
in the my.cnf set this value:
innodb_buffer_pool_size = 12G
innodb_buffer_pool_instances = 12
innodb_page_cleaners = 12
I have a C# application performing some database operations on a MySQL 5.7 server. Once the complete system hunp up and I had to hard-reset it. When it comes to a specific table read/write operation now the database server crashes. The windows log shows
InnoDB: Trying to access page number 286720 in space 29,
space name myInstance/myTable, which is outside the tablespace bounds.
Byte offset 0, len 16384, i/o type read.
I tried to use mysqlcheck --repair but it fails because note : The storage engine for the table doesn't support repair.
I've read some advices that say I should start MySQL in a recovery mode, so I added
[mysqld]
innodb_force_recovery=4
to the my.ini config file whereupon I should be able to use mysqldump to export the affected database table. But unfortunatelly I am not.
mysqldump: Error 2013: Lost connection to MySQL server
during query when dumping table `myTable` at row: 1246
Edit:
I checked the error log again and found lots of entries saying
[ERROR] C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqld.exe:
The table 'myTable' is full
I'm running the server on a Windows 32bit OS with an NTFS formatted partition. The myTable.ibd file size is around 4.5 GB, checking C.10.3 Limits on Table Size states a file size limit of "2TB (possibly larger)" for Win32 w/ NTFS.
While checking the reasons for my error the only possible cause I found was a full InnoDB tablespace. The solution might be "Changing the Number or Size of InnoDB Redo Log Files" although the coherence is a bit vague to me. Nevertheless I increased the size of the Redo Log Files from 48M to 100M. But nothing changed.
If I perform a SQL select * from myTable order by Id desc the server crashes instantly. Error log entry is exactly the same as above.
I checked the chapter 15.7.1 Resizing the InnoDB System Tablespace as well and found out that innodb_data_file_path is not explicitly specified.
Any ideas what I can do now? Thanks a lot!
InnoDB cannot repair corruption in tablespaces. This was never implemented and mysqlcheck won't help in any way.
The corruption is in space id 29 which is table myInstance.myTable. To repair it you need to dump all records from this table with innodb_force_recovery. Try all values from 4 to 6 until MySQL doesn't crash. Then drop the table and reload the dump.
If MySQL crashes even with innodb_force_recovery=6 then restore the table from backup.
If you don't have backup - use script http://bazaar.launchpad.net/~percona-dev/percona-data-recovery-tool-for-innodb/trunk/view/head:/fetch_data.sh . It will fetch as many records as it can.
One possible cause could be that the ib_logfile* files are corrupted.
To fix this, remove these files using rm ib_logfile*.
Where are these files?
These files are in mysql datadir. The location of datadir depends on the OS. Check my.cnf, in osx that should be in /usr/local/etc.
Every day or so our Wordpress sites stop responding, the pages begin returning the dreaded 'Error establishing database connection'. There is nothing in the MySQL logs, and I'm at a loss as to what could be causing the issue. We don't have a lot of site visitors, and the machine is a Medium EC2 instance. Anyone have any ideas on how to resolve this?
There's not a whole lot to work with here. But ... I had the same issue with my micro instance. My problem was that the server kept running out of memory and then the mysql server would stop. It would start again when restarting the computer but it was only a matter of time before it would crash again.
Here's what I was getting in my MySQL logs.
151023 6:15:44 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
151023 6:15:44 InnoDB: Completed initialization of buffer pool
151023 6:15:44 InnoDB: Fatal error: cannot allocate memory for the buffer pool
151023 6:15:44 [ERROR] Plugin 'InnoDB' init function returned error.
151023 6:15:44 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151023 6:15:44 [ERROR] Unknown/unsupported storage engine: InnoDB
151023 6:15:44 [ERROR] Aborting
You might want to check for something similar. I use Ubuntu and the log is at /var/log/mysql/ by default.
I solved the problem by setting up a swap file as per Amazon EC2, mysql aborting start because InnoDB: mmap (x bytes) failed; errno 12. The AWS instances don't come with a swap space setup by default (whereas the install I downloaded from Ubuntu back in the day did). You need to set it up manually. Here's the method -
ssh into your AWS instance. Then:
Run dd if=/dev/zero of=/swapfile bs=1M count=1024
Run mkswap /swapfile
Run swapon /swapfile
Add this line /swapfile swap swap defaults 0 0 to /etc/fstab
Read the linked question for more details. Hope that helps!
I had a similar issue with intermittent crashing of MySQL. It turned out to be Apache configurations. Bots were brute forcing the site and eventually caused Apache to crash (check your logs: $ cat /var/log/apache2/access.log). Apache automatically restarts, but there isn't enough spare memory to restart MySQL too, hence the Database Connection error. The short fix is to reduce the number of RequestWorkers in Apache to better fit the amount of RAM you have.
You can run a diagnostic on your apache configuration using Apache2Buddy. It'll calculate how many Apache Workers you can run given the amount of RAM you have and how big your application is:
$ curl -L http://apache2buddy.pl/ | perl
It's probably going to recommend changing MaxRequestWorkers (or MaxClients on older Apache systems) in your MPM-Prefork configurations. That file is at /etc/apache2/mods-available/mpm_prefork.conf on my system. After changing the value to what Apache2Buddy recommends, just restart Apache and you should be good to go.
I wrote an article on this situation if you want a deeper explanation, a method to stress test, or ideas on how to block some of the bot traffic: http://brunzino.github.io/blog/2016/05/21/solution-how-to-debug-intermittent-error-establishing-database-connection/
When I tried to install local wordpress the same error
error establishing database connection
occurred because I forget to stop the SQL and Apache which was started in xampp. I stopped it and reinstalled the wordpress for Windows and it worked.
Yesterday, my local server HDD crashes and goes for automatic scandisk at start up. I was not in office so I really don't know what had happen. But today when we start Server we are getting
130523 10:49:36 InnoDB: Operating system error number 23 in a file operation.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
InnoDB: File name .\ibdata1
InnoDB: File operation call: 'Windows aio'.
I try to run scandisk on C: and it found 4 corrupted records, but no bad sector or something. Database Server has around 300 Database, I Am not sure how to recover it. Someone suggest delete Log files and try to set
innodb_Force_recovery = 4
I try that too, but some of database are unavailable and crash MySql when I try to use them. I am not sure as reloading 300 database is not an easy job.
Edit:
I am on Windows 2008 Web Edition if that matters.
Sounds like you're on Windows: the ENFILE error basically says you have too many open files. Not quite sure how exactly you get around it: in general it is somewhere between 512 and 2048 in a process.
Would suggest maybe moving all the files out of the MySQL data directory and trying to recover them one by one.
I have the same issue but the error is pointing to another file :
InnoDB: File name .\ib_logfile0
Deleting (or renaming) both logfiles from the data directory (.\ib_logfile0 and .\ib_logfile1) solves the issue for me.