I'm not even close to a server expert - used XAMPP to install phpmyadmin/mysql and everything was running smoothly until a few weeks ago. Was able to put a bandaid on the issue when these errors started happening by doubling some of the memory parameters (buffer size etc) but now these issues are occurring again. There is nothing else running on this machine except MySQL, so not sure how how we are running out of memory. I'm willing to try anything - haven't found very clear instructions on how to use 'ulimit' - any suggestions from any experienced MySQL Server admins out there on what might be causing this issue, and what I can try to fix it?
2016-08-24 12:26:48 2456 [ERROR] C:\xampp\mysql\bin\mysqld.exe: Out of memory (Needed 767416 bytes)
2016-08-24 12:26:48 2456 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2016-08-24 12:26:48 2456 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
2016-08-24 13:41:33 6188 [Note] Plugin 'FEDERATED' is disabled.
2016-08-24 13:41:33 16d0 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
http://mysql.rjweb.org/doc.php/memory
All my answers were there. Again - i'm no server Admin, but basically my settings were all wrong in my.ini file. My innodb_buffer_pool_size was way too low, and my key_buffer_size was much too high. I'm guessing that the original settings were enough to get us buy as we were building the databases, but as tables/views/threads increased, performance was impacted.
Related
Version: Mysql server version is 8.0 and it is installed on Windows server 2019.
Problem statement: Error log file from location C:\programdata\mysql80\data\ is increasing and it is about ~100GB.
Error log statements are :
[Warning] [MY-011959] [InnoDB] Difficult to find free blocks in the buffer pool (6091 search iterations)! 6091 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 1738 OS file reads, 217 OS file writes, 45 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.
Scenario:
There was failure to update more than 4MB data size to the database, so we changed the "max_allowed_packet" from default 4M to 256M in "mysql.ini" file.
After this settings restarted mysqld service, everything worked after this db changed.
But after 2 days we are facing the above mentioned issue, as a result disk space is getting full and DB connections are pending in a queue.
Tried to stop the mysql80 service and deleted the error.log file and reverted the max_allowed_packet to its default size.
But with the change also error file is getting created again with same warning.
What could be the possible issues or fix for it?
i have the following Issue.
I have an AWS EC2 t2.micro instance with LAMP installed on it.
My WebApp uses InnoDB tables. If there are a lot of users, InnoDB used to crash because of buffer pool size. It says it cannot allocate enought memory.
170511 10:32:05 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
I have put innodb_buffer_pool_size to 750M (if I put it more that 1GB - mysql doesn't start at all). And I have put stress test of my WebApp under LoadImpact. From 30 person and up WebApp was trying to allocate something like 800M and crashes MySQL again.
170511 12:16:04 InnoDB: Initializing buffer pool, size = 752.0M
InnoDB: mmap(807010304 bytes) failed; errno 12
Another problem is that I cannot just run sudo service mysql start or restart after mysql Crash. It says job failed to start.
How to make my server more stable. What options should I use.
How to prevent such "crashy" behaviour. I mean can I make something that makes server and mysql prosess not chash but freeze or something like that. I dont want do sudo reboo every time there are a lot of people on the website
t2.micro instance has only 512M RAM. If you run MySQL and Apache on same box you should allocate memory sparingly.
It looks like apache processes use up all memory so MySQL gets less than necessary minimum.
I would suggest to start with adding swap if you haven't added it yet.
dd if=/dev/zero of=/swapfile bs=1024 count=65536
chmod 0600 /swapfile
mkswap /swapfile
swapon /swapfile
and add it to /etc/fstab.
/swapfile swap swap defaults 0 0
Next, limit Apache workers to make sure MySQL always gets innodb_buffer_pool_size. Depending on what you use - prefork or worker configuration will be different, but you got an idea.
I have XAMPP 5.6.15 for Windows which is running 10.1.9-MariaDB. I have a table, InnoDB, with large amount of records and size, 49888 records and 245 MiB size. The main of its size in a column contains text named tafText.
Using PHPMyAdmin, I try to set fulltext search on that column. However, mysqld is hanged and the database server is topped:
The following is a copy of the error log generated after the occurrence of the error:
2016-10-18 20:49:22 13360 [Note] InnoDB: Online DDL : Start
2016-10-18 20:49:22 13360 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files
2016-10-18 20:49:22 1d78 InnoDB: Assertion failure in thread 7544 in file row0merge.cc line 892
InnoDB: Failing assertion: b < &block[srv_sort_buf_size]
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
161018 20:49:22 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see http://kb.askmonty.org/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.1.9-MariaDB
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=2
max_threads=1001
thread_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 787099 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_parameter_handler()
mysqld.exe!my_mb_ctype_mb()
mysqld.exe!?set_charset#String##QAEXPBUcharset_info_st###Z()
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlSubscribeWnfStateChangeNotification()
ntdll.dll!RtlSubscribeWnfStateChangeNotification()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
What is the wrong in mysql settings that I have which makes this operation failed?
The following is copy of InnoDB related settings in my.ini:
# Comment the following if you are using InnoDB tables
#skip-innodb
innodb_data_home_dir = "C:/xampp-3/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp-3/mysql/data"
#innodb_log_arch_dir = "C:/xampp-3/mysql/data"
## You can set .._buffer_pool_size up to 50 - 80 %
## of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 256M
innodb_additional_mem_pool_size = 125M
## Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
Easy to find with google, if you type the "InnoDB: Failing assertion: b < &block[srv_sort_buf_size]" :)
This bug was reported in https://jira.mariadb.org/browse/MDEV-9129 , fixed in MariaDB 10.1.10. You have 10.1.9, so there it is not fixed.
Every day or so our Wordpress sites stop responding, the pages begin returning the dreaded 'Error establishing database connection'. There is nothing in the MySQL logs, and I'm at a loss as to what could be causing the issue. We don't have a lot of site visitors, and the machine is a Medium EC2 instance. Anyone have any ideas on how to resolve this?
There's not a whole lot to work with here. But ... I had the same issue with my micro instance. My problem was that the server kept running out of memory and then the mysql server would stop. It would start again when restarting the computer but it was only a matter of time before it would crash again.
Here's what I was getting in my MySQL logs.
151023 6:15:44 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
151023 6:15:44 InnoDB: Completed initialization of buffer pool
151023 6:15:44 InnoDB: Fatal error: cannot allocate memory for the buffer pool
151023 6:15:44 [ERROR] Plugin 'InnoDB' init function returned error.
151023 6:15:44 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151023 6:15:44 [ERROR] Unknown/unsupported storage engine: InnoDB
151023 6:15:44 [ERROR] Aborting
You might want to check for something similar. I use Ubuntu and the log is at /var/log/mysql/ by default.
I solved the problem by setting up a swap file as per Amazon EC2, mysql aborting start because InnoDB: mmap (x bytes) failed; errno 12. The AWS instances don't come with a swap space setup by default (whereas the install I downloaded from Ubuntu back in the day did). You need to set it up manually. Here's the method -
ssh into your AWS instance. Then:
Run dd if=/dev/zero of=/swapfile bs=1M count=1024
Run mkswap /swapfile
Run swapon /swapfile
Add this line /swapfile swap swap defaults 0 0 to /etc/fstab
Read the linked question for more details. Hope that helps!
I had a similar issue with intermittent crashing of MySQL. It turned out to be Apache configurations. Bots were brute forcing the site and eventually caused Apache to crash (check your logs: $ cat /var/log/apache2/access.log). Apache automatically restarts, but there isn't enough spare memory to restart MySQL too, hence the Database Connection error. The short fix is to reduce the number of RequestWorkers in Apache to better fit the amount of RAM you have.
You can run a diagnostic on your apache configuration using Apache2Buddy. It'll calculate how many Apache Workers you can run given the amount of RAM you have and how big your application is:
$ curl -L http://apache2buddy.pl/ | perl
It's probably going to recommend changing MaxRequestWorkers (or MaxClients on older Apache systems) in your MPM-Prefork configurations. That file is at /etc/apache2/mods-available/mpm_prefork.conf on my system. After changing the value to what Apache2Buddy recommends, just restart Apache and you should be good to go.
I wrote an article on this situation if you want a deeper explanation, a method to stress test, or ideas on how to block some of the bot traffic: http://brunzino.github.io/blog/2016/05/21/solution-how-to-debug-intermittent-error-establishing-database-connection/
When I tried to install local wordpress the same error
error establishing database connection
occurred because I forget to stop the SQL and Apache which was started in xampp. I stopped it and reinstalled the wordpress for Windows and it worked.
I am working on an embedded Linux system running MySQL 5.1. In rare cases QA reports systems not starting properly because of mysqld not starting. If this happens the MySQL log files looks similar to this excerpt:
150716 14:29:42 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150716 14:29:42 InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 133478.
InnoDB: Doing recovery: scanned up to log sequence number 0 133478
150716 14:29:42 InnoDB: Started; log sequence number 0 133478
/usr/libexec/mysqld: Unknown error 130
150716 14:29:42 [ERROR] Can't open the mysql.plugin table. Please run the mysql_upgrade script to create it.
150716 14:29:42 [ERROR] Fatal error: Can't open and lock privilege tables: Incorrect file format 'host'
This is probably due to the fact that this is an embedded device without a power switch and it is powered off by unplugging the network cable and thus killing the PoE supply. In this case of course mysqld will be terminated quite abnormaly.
my.cnf contains nothing fancy besides a size limitation to 18 MB.
Question
Is it possible that InnoDB tables get corrupted and recovery is NOT possible if there is NO bug involved (either in MySQL itself or e.g. a faulty fsync() implementation)? Are there situtations that can cause a corruption (that the DB cannot be recovered from) even if all software components are working correctly? Can such a DB be safely used in an environment where power failures occur "in normal operation"?
What I am ultimately asking is:
Is there a point searching for a fix to this problem or is there no fix to this problem whatsoever?
Is it possible that InnoDB tables get corrupted if there is NO bug involved (either in MySQL itself or e.g. a faulty fsync() implementation)?
Yes, eg: hardware failure
Are there situtations that can cause a corruption even if all software components are working correctly?
Yes, eg: hardware failure
Can such a DB be safely used in an environment where power failures occur "in normal operation"?
Yes, only if your hardware works "normally"
What I am ultimately asking is: Is there a point searching for a fix to this problem or is there no fix to this problem whatsoever?
Usually it's very difficult to fix the database if you meet a corruption. Do backup.
This article may help you:
https://dev.mysql.com/doc/refman/5.6/en/innodb-init-startup-configuration.html
Caution
InnoDB is a transaction-safe (ACID compliant) storage engine for MySQL that has commit, rollback, and crash-recovery capabilities to protect user data. However, it cannot do so if the underlying operating system or hardware does not work as advertised. Many operating systems or disk subsystems may delay or reorder write operations to improve performance. On some operating systems, the very fsync() system call that should wait until all unwritten data for a file has been flushed might actually return before the data has been flushed to stable storage. Because of this, an operating system crash or a power outage may destroy recently committed data, or in the worst case, even corrupt the database because of write operations having been reordered. If data integrity is important to you, perform some “pull-the-plug” tests before using anything in production. On OS X 10.3 and up, InnoDB uses a special fcntl() file flush method. Under Linux, it is advisable to disable the write-back cache.
On ATA/SATA disk drives, a command such hdparm -W0 /dev/hda may work to disable the write-back cache. Beware that some drives or disk controllers may be unable to disable the write-back cache.
With regard to InnoDB recovery capabilities that protect user data, InnoDB uses a file flush technique involving a structure called the doublewrite buffer, which is enabled by default (innodb_doublewrite=ON). The doublewrite buffer adds safety to recovery following a crash or power outage, and improves performance on most varieties of Unix by reducing the need for fsync() operations. It is recommended that the innodb_doublewrite option remains enabled if you are concerned with data integrity or possible failures. For additional information about the doublewrite buffer, see Section 14.9, “InnoDB Disk I/O and File Space Management”.
Caution
If reliability is a consideration for your data, do not configure InnoDB to use data files or log files on NFS volumes. Potential problems vary according to OS and version of NFS, and include such issues as lack of protection from conflicting writes, and limitations on maximum file sizes.
Did you upgrade the MySQL version, but fail to run mysql_upgrade ? That's what the error is saying.
I finally found the root cause. The problem does not actually occur with the InnoDB tables, but with the system tables.
In MySQL 5.1 system tables are stored using the MyISAM engine. This makes these tables very fragile on power loss.
For all system tables the content of the MYI (index) and MYD (data) files were lost.
Missing this data - of course - the rest of the databases had a problem...
The important hint for me was
mysql.plugin table
Finally looked into the directory containing the system tables and saw they were using the MyISAM storage engine. Then the consequences are quite obvious.
(Only) Solution:
Go to a newer version (I used MariaDB in my case).
You cannot use InnoDB as storage engine for the system tables on MySQL 5.1.