I have a MySQL installed and running on a CentOS 6.6 and MySQL version 5.5.40 on RackSpace. I always run into this error when running heavy queries.
Here is the settings of my.cnf
[mysqld]
2 datadir=/mnt/data/mysql
3 tmpdir=/mnt/data/temp
4 socket=/var/lib/mysql/mysql.sock
5 bind-address=0.0.0.0
6 port=3306
7 wait_timeout=432000
8 max_allowed_packet=1G
9 max_connections=500
10 query-cache-size=0
11 query-cache-type=0
12 #query_cache_size=64M
13 #query_cache_limit=64M
14 key_buffer_size=1G
15 sort_buffer_size=16M
16 tmp_table_size=32M
17 max_heap_table_size=32M
18 read_buffer_size=512K
19 read_rnd_buffer_size=512K
20 thread_cache_size=50
21
22 innodb_buffer_pool_size=12G
23 innodb_buffer_pool_instance=2
24 innodb_read_io_threads=12
25 innodb_write_io_threads=12
26 innodb_io_capacity=300
27 innodb_log_file_size=128M
28 innodb_thread_concurrency=0
Here is the error log I've caught after the crash:
150820 13:46:26 mysqld_safe Number of processes running now: 0
150820 13:46:26 mysqld_safe mysqld restarted
150820 13:46:26 [Note] Plugin 'FEDERATED' is disabled.
150820 13:46:26 [Warning] Using unique option prefix innodb_buffer_pool_instance instead of innodb-buffer-pool-instances is deprecated and will be removed in a future release. Please use the full name instead.
150820 13:46:26 InnoDB: The InnoDB memory heap is disabled
150820 13:46:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150820 13:46:26 InnoDB: Compressed tables use zlib 1.2.3
150820 13:46:26 InnoDB: Using Linux native AIO
150820 13:46:26 InnoDB: Initializing buffer pool, size = 12.0G
InnoDB: mmap(6593445888 bytes) failed; errno 12
150820 13:46:27 InnoDB: Completed initialization of buffer pool
150820 13:46:27 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150820 13:46:27 [ERROR] Plugin 'InnoDB' init function returned error.
150820 13:46:27 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150820 13:46:27 [ERROR] Unknown/unsupported storage engine: InnoDB
150820 13:46:27 [ERROR] Aborting
150820 13:46:27 [Note] /usr/libexec/mysqld: Shutdown complete
150820 13:46:27 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
EDIT:
RackSpace VM's specs are:
CPU: Intel(R) Xeon(R) CPU E5-2670 0 # 2.60GHz
RAM: 8GB
your server has only 8 GB RAM and you have assigned too much ram to mysql.
Even you need to change your configuration for many variables but first to comeout your issue do the below changes-
innodb_buffer_pool_instance=2 #comment it for the time being we can set it later.
innodb_buffer_pool_size=6G
key_buffer_size=20M #if your server is innodb but if you are also using myisam tables then keep as it is.
sort_buffer_size=2M # we can change later.
read_buffer_size=512K # comment it for time being.
read_rnd_buffer_size=512K #comment it for time being as used per session
tmp_table_size=1G #this can be reason of your problem so increase it.
max_heap_table_size=1G #this can be reason of your problem so increase it.
If possible decrease max_connections from 500 to 400 as each connection uses server resources.
Try and share the results.
In case it is useful, this is how I solve this problem. In my case the problem was caused by queries that were too large for the packets being sent to the server.
After running the following command against the server, the large queries process fine
SET GLOBAL max_allowed_packet=1073741824;
Related
In the last few weeks our mysql DB has been crashing randomly. -
I have ran a check against all the databases for corruption but all are OK.
EDIT 2016.01.14 - The error that keeps popping up commonly is the following -
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 0.
Although, the 'File Read Page' is always different, the last time it was Zero, but before that it was 4500.
A little background.
Been running Xampp for over a year without a problem
We have Applications constantly inserting data into MYSQL from lots of data sources, this is 24/7.
Apart from the above applications, nothing else runs around the time of the crash, I have also checked the system logs to see if anything sticks out but it doesn't.
Two weeks ago, we moved to another server with better spec, the old server was cloned and place on the new one.
Nothing has been altered in DB or new tables added.
Within a week, mysql started to produce errors.
The new server specs are as follows -
Windows Server 2008 R2 Standard - Running Service Pack 1
Proccessor: Intel(R) Xeon(R) # 3.3ghz 2x Dual processor
8GB Ram
64Bit OS (Same as old server)
MySQL Version: 5.6.20
PHP Version 5.5.15
Here are the recent error logs -
*************MYSQL ERROR LOG*****************
//////////////////////////////////*********************
ERROR on 2016-01-05
******************///////////////////////
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
2016-01-05 08:42:52 cc8 InnoDB: Operating system error number 995 in a file operation.
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
2016-01-05 08:42:52 cc8 InnoDB: Operating system error number 995 in a file operation.
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
2016-01-05 08:42:53 cc8 InnoDB: Operating system error number 995 in a file operation.
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
2016-01-05 08:42:53 cc8 InnoDB: Operating system error number 995 in a file operation.
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
//////////////////////////////////*********************
Error- 2015-12-22
********************************///////////////////////////
2015-12-22 16:46:02 12d4 InnoDB: uncompressed page, stored checksum in field1 1169359801, calculated checksums for field1: crc32 4003191917, innodb 1169359801, none 3735928559, stored checksum in field2 4049981449, calculated checksums for field2: crc32 4003191917, innodb 587193584, none 3735928559, page LSN 128 4137924960, low 4 bytes of LSN at page end 4137853843, page number (if stored to page already) 0, space id (if created with >= MySQL-4.1.1 and stored already) 252
InnoDB: Page may be a file space header page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 0.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 0.
InnoDB: You may have to recover from a backup.
2015-12-22 16:46:02 12d4 InnoDB: Page dump in ascii and hex (16384 bytes): // Lots of binary here ..
InnoDB: End of page dump
2015-12-22 16:46:03 12d4 InnoDB: uncompressed page, stored checksum in field1 1169359801, calculated checksums for field1: crc32 4003191917, innodb 1169359801, none 3735928559, stored checksum in field2 4049981449, calculated checksums for field2: crc32 4003191917, innodb 587193584, none 3735928559, page LSN 128 4137924960, low 4 bytes of LSN at page end 4137853843, page number (if stored to page already) 0, space id (if created with >= MySQL-4.1.1 and stored already) 252
InnoDB: Page may be a file space header page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 0.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Error: Unable to read tablespace 252 page no 0 into the buffer pool after 100 attempts
InnoDB: The most probable cause of this error may be that the table has been corrupted.
InnoDB: You can try to fix this problem by using innodb_force_recovery.
InnoDB: Please see reference manual for more details.
InnoDB: Aborting...
2015-12-22 16:46:03 12d4 InnoDB: Assertion failure in thread 4820 in file buf0buf.cc line 2641
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:46:03 UTC - mysqld got exception 0x80000003 ;
This could be because you hit a bug.
It is also possible that this binary
or one of the libraries it was linked against is corrupt,
improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem,
but since we have already crashed,
something is definitely wrong and this may
fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=107
max_threads=151
thread_count=5
connection_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133778 K bytes
of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x22020238
Attempting backtrace. You can use the following information to find out
where mysqld died.
If you see no messages after this, something went
terribly wrong...
50a500
mysqld.exe!my_thread_name()
74392d mysqld.exe!my_mb_ctype_mb()
5ecdb6
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
61d611
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
647350
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
6492fa
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
649383
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
6493b3
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
64945d
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
6495cb
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
5d48e3
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
53f011
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
2be48e
mysqld.exe!?ha_write_row#handler##QAEHPAE#Z()
3d8d10
mysqld.exe!?write_record##YAHPAVTHD##PAUTABLE##PAVCOPY_INFO##2#Z()
3df599
mysqld.exe!?mysql_insert##YA_NPAVTHD##PAUTABLE_LIST##AAV?$List#VItem####AAV?$List#V?$List#VItem######22W4enum_duplicates##_N#Z()
2ecdf5
mysqld.exe!?mysql_execute_command##YAHPAVTHD###Z()
2ef81e
mysqld.exe!?mysql_parse##YAXPAVTHD##PADIPAVParser_state###Z()
2f06f8
mysqld.exe!?dispatch_command##YA_NW4enum_server_command##PAVTHD##PADI#Z()
2f135a
mysqld.exe!?do_command##YA_NPAVTHD###Z()
364e59 mysqld.exe!?do_handle_one_connection##YAXPAVTHD###Z()
364efd
mysqld.exe!handle_one_connection()
6bb8cb mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
510336
mysqld.exe!win_pthread_mutex_trylock()
746c00 mysqld.exe!my_mb_ctype_mb()
746c8a
mysqld.exe!my_mb_ctype_mb()
7728338a kernel32.dll!BaseThreadInitThunk()
77c297f2
ntdll.dll!RtlInitializeExceptionChain()
77c297c5
ntdll.dll!RtlInitializeExceptionChain()
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (21e3387c): INSERT INTO 'WPM'(
`Pos`,
`FileMod`,
`Duration`) VALUES (
'IWM',
'2015-12-14 07:47:47 ',
0)Connection ID (thread ID): 3722
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2015-12-23 08:21:58 4920
[Note] Plugin 'FEDERATED' is disabled.
2015-12-23 08:21:58 10c4 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
2015-12-23 08:21:58 4920 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-12-23 08:21:58 4920 [Note] InnoDB: The InnoDB memory heap is disabled
2015-12-23 08:21:58 4920 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2015-12-23 08:21:58 4920 [Note] InnoDB: Memory barrier is not used
2015-12-23 08:21:58 4920 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-12-23 08:21:58 4920 [Note] InnoDB: Not using CPU crc32 instructions
2015-12-23 08:21:58 4920 [Note] InnoDB: Initializing buffer pool, size = 16.0M
2015-12-23 08:21:58 4920 [Note] InnoDB: Completed initialization of buffer pool
2015-12-23 08:21:58 4920 [Note] InnoDB: Highest supported file format is Barracuda.
2015-12-23 08:21:58 4920 [Note] InnoDB: Log scan progressed past the checkpoint lsn 553893690098
2015-12-23 08:21:58 4920 [Note] InnoDB: Database was not shutdown normally!
2015-12-23 08:21:58 4920 [Note] InnoDB: Starting crash recovery.
2015-12-23 08:21:58 4920 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-12-23 08:21:59 4920 [Note] InnoDB: Restoring possible half-written data pages
2015-12-23 08:21:59 4920 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 553893828751
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 151 row operations to undo
InnoDB: Trx id counter is 152858880
2015-12-23 08:22:00 4920 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2015-12-23 08:22:00 4920 [Note] InnoDB: 128 rollback segment(s) are active.
InnoDB: Starting in background the rollback of uncommitted transactions
2015-12-23 08:22:00 2c4 InnoDB: Rolling back trx with id 152858375, 151 rows to undo
2015-12-23 08:22:00 4920 [Note] InnoDB: Waiting for purge to start
2015-12-23 08:22:00 4920 [Note] InnoDB: Rollback of trx with id 152858375 completed
2015-12-23 08:22:00 2c4 InnoDB: Rollback of non-prepared transactions completed
2015-12-23 08:22:00 4920 [Note] InnoDB: 5.6.20 started; log sequence number 553893828751
2015-12-23 08:22:00 4920 [Note] Server hostname (bind-address): '*'; port: 3306
2015-12-23 08:22:00 4920 [Note] IPv6 is available.
2015-12-23 08:22:00 4920 [Note] - '::' resolves to '::';
2015-12-23 08:22:00 4920 [Note] Server socket created on IP: '::'.
2015-12-23 08:22:00 4920 [Note] Event Scheduler: Loaded 11 events
2015-12-23 08:22:00 4920 [Note] c:\xampp\mysql\bin\mysqld.exe: ready for connections.
Version: '5.6.20' socket: '' port: 3306 MySQL Community Server (GPL)
2015-12-23 08:24:07 5736 [Note] Plugin 'FEDERATED' is disabled.
2015-12-23 08:24:07 fc8 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
2015-12-23 08:24:07 5736 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-12-23 08:24:07 5736 [Note] InnoDB: The InnoDB memory heap is disabled
2015-12-23 08:24:07 5736 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2015-12-23 08:24:07 5736 [Note] InnoDB: Memory barrier is not used
2015-12-23 08:24:07 5736 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-12-23 08:24:07 5736 [Note] InnoDB: Not using CPU crc32 instructions
2015-12-23 08:24:07 5736 [Note] InnoDB: Initializing buffer pool, size = 16.0M
2015-12-23 08:24:07 5736 [Note] InnoDB: Completed initialization of buffer pool
2015-12-23 08:24:07 5736 [Note] InnoDB: Highest supported file format is Barracuda.
2015-12-23 08:24:07 5736 [Note] InnoDB: Log scan progressed past the checkpoint lsn 553896050587
2015-12-23 08:24:07 5736 [Note] InnoDB: Database was not shutdown normally!
2015-12-23 08:24:07 5736 [Note] InnoDB: Starting crash recovery.
2015-12-23 08:24:07 5736 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-12-23 08:24:07 5736 [Note] InnoDB: Restoring possible half-written data pages
2015-12-23 08:24:07 5736 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 553896774958
2015-12-23 08:24:07 5736 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
//////////////////////////////////*********************
Error on 2016-01-08
********************////////////////////////
2016-01-08 00:00:14 1584 InnoDB: uncompressed page, stored checksum in field1 1299653197, calculated checksums for field1: crc32 90466284, innodb 1299653197, none 3735928559, stored checksum in field2 341870525, calculated checksums for field2: crc32 90466284, innodb 3590763946, none 3735928559, page LSN 129 3920058837, low 4 bytes of LSN at page end 3920015931, page number (if stored to page already) 75791, space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an update undo log page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 75791.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
2016-01-08 00:00:14 1584 InnoDB: Assertion failure in thread 5508 in file buf0buf.cc line 4195
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
00:00:14 UTC - mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=113
max_threads=151
thread_count=5
connection_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133778 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
116a500 mysqld.exe!my_thread_name()
13a392d mysqld.exe!my_mb_ctype_mb()
12c7aa2 mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
12c82ad mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
1225a72 mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11d423b mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11d4694 mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11d548a mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11b147d mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11b167d mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
7728338a kernel32.dll!BaseThreadInitThunk()
77c297f2 ntdll.dll!RtlInitializeExceptionChain()
77c297c5 ntdll.dll!RtlInitializeExceptionChain()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
InnoDB: Warning: a long semaphore wait:
--Thread 6764 has waited at trx0undo.cc line 1778 for 29770.00 seconds the semaphore:
Mutex at 1B4F60EC created file trx0rseg.cc line 196, lock var 1
waiters flag 1
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
Here is the mysql.ini settings file (If this helps)
# The MySQL server
[mysqld]
port= 3306
socket = "C:/xampp/mysql/mysql.sock"
basedir = "C:/xampp/mysql"
tmpdir = "C:/xampp/tmp"
datadir = "C:/xampp/mysql/data"
pid_file = "mysql.pid"
# enable-named-pipe
key_buffer = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error = "mysql_error.log"
# Change here for bind listening
# bind-address="127.0.0.1"
# bind-address = ::1 # for ipv6
# Where do all the plugins live
plugin_dir = "C:/xampp/mysql/lib/plugin/"
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
# commented in by lampp security
#skip-networking
skip-federated
# Replication Master Server (default)
# binary logging is required for replication
# log-bin deactivated by default since XAMPP 1.4.11
#log-bin=mysql-bin
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
# MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
# where you replace <host>, <user>, <password> by quoted strings and
# <port> by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host = <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user = <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password = <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port = <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin
# Point the following paths to different dedicated disks
#tmpdir = "C:/xampp/tmp"
#log-update = /path-to-dedicated-directory/hostname
# Uncomment the following if you are using BDB tables
#bdb_cache_size = 4M
#bdb_max_lock = 10000
# Comment the following if you are using InnoDB tables
#skip-innodb
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
#innodb_log_arch_dir = "C:/xampp/mysql/data"
## You can set .._buffer_pool_size up to 50 - 80 %
## of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 16M
innodb_additional_mem_pool_size = 2M
## Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
## UTF 8 Settings
#init-connect=\'SET NAMES utf8\'
#collation_server=utf8_unicode_ci
#character_set_server=utf8
#skip-character-set-client-handshake
#character_sets-dir="C:/xampp/mysql/share/charsets"
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
I simply have no idea what to do, I have no experience in the technical side to troubleshooting Mysql. I have looked on the MYsql site, are apparently error code 995 is a known bug, it was meant to be fixed, but I read that it never got pushed into the build.
If you need any more info please ask!
InnoDB deliberately crashes MySQL if it reads a page from "disk" (might come from RAM though if the relevant blocks are in OS cache) and the checksum mismatches with the one in the page header. This is what's happening to you.
Reasons could be:
If you're not using ECC ram you might just have some corrupted data in OS cache. Rebooting the server will clear the cache and you can get some ECC ram to prevent it happening. (less likely)
On disk corruption (more likely):
If you're lucky the corruption is on index pages and you can fully recover by doing a innodb_recovery (http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html)
If the corruption is on data page you most certainly will lose data but you still should be able to recover most of the information using the same method.
In such cases I would just restore from backup if that's possible. The other error you pasted also indicates system level I/O issues so I would look into it soon before it crashes completely.
I believe some pages in Innodb tablespace got corrupted..
First backup your DB!
First suggestion would be to run check table using this command:
mysqlcheck -A --auto-repair
Reference: Here
Add the following line below the mysqld section in the mysql config file (my.ini) and restart both the apache and mysql service afterwards.
innodb_force_recovery = 1
For your information: read this
Note: Innodb becomes read only for data operations when running in innodb_force_recovery mode
If this is not working, my second suggestion would be to decrease the innodb_buffer_pool_size and convert some "heavey loading" tables into MyISAM:
ALTER TABLE table_name ENGINE=MYISAM
Although you increased the specs of your machine, due to high usage of your DB table can still cause a problem.
(Please update me and which you good luck!)
Some background. I hosted a WordPress site in VPS and sometimes MYSQL down with error "Error Establishing a Database Connection". I've spent some time to research and believe the problem is due to when MySQL get restarted, it is not able to allocate enough RAM to proceed.
I believe I can improve the situation by increasing physical RAM or swap. But my question for this post is, why MySQL need to restart itself? My site is with pretty low traffic and doesn't seem like the DB is corrupted.
Below is the full log for this issue:
160103 18:39:54 mysqld_safe Number of processes running now: 0
160103 18:39:54 mysqld_safe mysqld restarted
160103 18:39:55 [Note] /usr/libexec/mysqld (mysqld 5.5.44-MariaDB) starting as process 22061 ...
160103 18:39:55 InnoDB: The InnoDB memory heap is disabled
160103 18:39:55 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160103 18:39:55 InnoDB: Compressed tables use zlib 1.2.7
160103 18:39:55 InnoDB: Using Linux native AIO
160103 18:39:55 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
160103 18:39:55 InnoDB: Completed initialization of buffer pool
160103 18:39:55 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160103 18:39:56 [ERROR] Plugin 'InnoDB' init function returned error.
160103 18:39:56 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
160103 18:39:56 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
160103 18:39:56 [Note] Plugin 'FEEDBACK' is disabled.
160103 18:39:56 [ERROR] Unknown/unsupported storage engine: InnoDB
160103 18:39:56 [ERROR] Aborting
When Number of Processes Running Now 0 means that MySQL isn't running. So the daemon mysqld_safe "makes you a favor" and start MySQL
You have assigned very low RAM 128MB to innodb_buffer_pool_size (which is default RAM). So you should assign approx. 80% of total RAM to this variable if you are using innodb engine, as mysql uses intially memory from this variable to cache index as well data in innodb engine.
So update at least 1 GB RAM (should be 80% of total ram for innodb) to innodb_buffer_pool_size in your config file and restart mysql service.
Update:
You have 1 GB RAM: You can assign 800M RAM (or >=500M) to innodb_buffer_pool_size.
MySQL will auto restart when number of running process 0: As per errors shared by you...this error is coming at the time of mysql service start and server is not able to initiate mysql service with such a less RAM.
160103 18:39:55 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
160103 18:39:55 InnoDB: Completed initialization of buffer pool
160103 18:39:55 InnoDB: Fatal error: cannot allocate memory for the buffer pool
I had a similar problem with MySQL on CentOS 6.9. It was caused by memory starving due another PHP process and not by MySQL. MySQL was just a victim.
You can look at kernel messages. The location of this messages may be specific to OS. In case of CentOS 6.x they are located in /var/log/messages .
Here are some related messages from /var/log/messages:
Jul 25 20:34:46 myserver kernel: Out of memory: Kill process 21467 (mysqld) score 30 or sacrifice child
Jul 25 20:34:46 myserver kernel: Killed process 21467, UID 497, (mysqld) total-vm:757004kB, anon-rss:17728kB, file-rss:320kB
You can run the following command to see if kernel run out of memory:
cat /var/log/messages | grep out_of_memory
150505 16:57:01 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150505 16:58:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150505 16:58:01 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150505 16:58:01 [Note] Plugin 'FEDERATED' is disabled.
150505 16:58:01 InnoDB: The InnoDB memory heap is disabled
150505 16:58:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150505 16:58:01 InnoDB: Compressed tables use zlib 1.2.8
150505 16:58:01 InnoDB: Using Linux native AIO
150505 16:58:01 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150505 16:58:01 InnoDB: Completed initialization of buffer pool
150505 16:58:01 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150505 16:58:01 [ERROR] Plugin 'InnoDB' init function returned error.
150505 16:58:01 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150505 16:58:01 [ERROR] Unknown/unsupported storage engine: InnoDB
150505 16:58:01 [ERROR] Aborting
Above is mysql error log, sometime mysql will just crash, and I want to do a restart on the service, it will not able to start,the error is something like
the "unknown instance" error.
Anyone got any idea what the error above is about and how to solve it, I tried google but seems to not able find the answer.
Thanks!
I found this question searching for an answer myself, my server's /var/log/mysql/error.log had very similar statements to yours. Running dmesg | tail -20 revealed the culprit: the OS ran out of memory and killed mysqld:
[835661.446277] Out of memory: Kill process 1024 (mysqld) score 91 or sacrifice child
[835661.446912] Killed process 1024 (mysqld) total-vm:901544kB, anon-rss:92332kB, file-rss:0kB
[835661.488337] init: mysql main process (1024) killed by KILL signal
Now to figure out why memory ran out, on this newer server, with twice as much RAM as the one it's replacing, which never did this...
I have this error log from MySQL, any idea?
Website works for some time and then I get MySQL shutdown completely after a couple of hours.
140919 10:48:27 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140919 10:48:27 [Note] Plugin 'FEDERATED' is disabled.
140919 10:48:27 InnoDB: The InnoDB memory heap is disabled
140919 10:48:27 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140919 10:48:27 InnoDB: Compressed tables use zlib 1.2.3.4
140919 10:48:28 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140919 10:48:28 InnoDB: Completed initialization of buffer pool
140919 10:48:28 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140919 10:48:28 [ERROR] Plugin 'InnoDB' init function returned error.
140919 10:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140919 10:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB
140919 10:48:28 [ERROR] Aborting
140919 10:48:28 [Note] /usr/sbin/mysqld: Shutdown complete
140919 10:48:28 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
140919 10:48:28 [Note] Plugin 'FEDERATED' is disabled.
140919 10:48:28 InnoDB: The InnoDB memory heap is disabled
140919 10:48:28 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140919 10:48:28 InnoDB: Compressed tables use zlib 1.2.3.4
140919 10:48:28 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140919 10:48:28 InnoDB: Completed initialization of buffer pool
140919 10:48:28 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140919 10:48:28 [ERROR] Plugin 'InnoDB' init function returned error.
140919 10:48:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140919 10:48:28 [ERROR] Unknown/unsupported storage engine: InnoDB
140919 10:48:28 [ERROR] Aborting
140919 10:48:28 [Note] /usr/sbin/mysqld: Shutdown complete
TLDR;
Mysql can't restart because it's out of memory, check that you have an appropriate swapfile configured.
Didn't help? If that's not your issue, more qualified questions to continue research are:
mysqld service stops once a day on ec2 server
https://askubuntu.com/questions/422037/optimising-mysql-settings-mysqld-running-out-of-memory
Background
I had exactly this problem on the very first system I set up on EC2, characterised by the wordpress site hosted there going down on occasion with "Error establishing database connection".
The logs showed the same error that the OP posted. My reading of the error (timestamps removed) is:
Out of memory error:
InnoDB: Fatal error: cannot allocate memory for the buffer pool
InnoDB can't start without enough memory
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[ERROR] Unknown/unsupported storage engine: InnoDB
[ERROR] Aborting
mysqld is shutting down, which in this context, really means failing to restart!
[Note] /usr/sbin/mysqld: Shutdown complete
Checking /var/log/syslog and searching for mysql yields:
Out of memory: Kill process 15452 (mysqld) score 93 or sacrifice child
Killed process 15452 (mysqld) total-vm:888672kB, anon-rss:56252kB, file-rss:0kB
init: mysql main process (15452) killed by KILL signal
init: mysql main process ended, respawning
type=1400 audit(1443812767.391:30): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=21984 comm="apparmor_parser"
init: mysql main process (21996) terminated with status 1
init: mysql main process ended, respawning
init: mysql post-start process (21997) terminated with status 1
<repeated>
Note: you may have to gunzip and search through archived logs if the error occurred before the logs were rotated by cron.
Solution
In my case the underlying issue was that I'd neglected to configure a swapfile.
You can check to see if you have one configured by running free -m.
total used free shared buffers cached
Mem: 604340 587364 16976 0 29260 72280
-/+ buffers/cache: 485824 118516
Swap: 0 0 0
In the example above, Swap: 0 indicates no swapfile.
Tutorials on setting one up:
https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04
https://help.ubuntu.com/community/SwapFaq
Note that bigger is not necessarily better! From the Ubuntu guide:
The "diminishing returns" means that if you need more swap space than twice your RAM size, you'd better add more RAM as Hard Disk Drive (HDD) access is about 10³ slower then RAM access, so something that would take 1 second, suddenly takes more then 15 minutes! And still more then a minute on a fast Solid State Drive (SSD)...
Regarding the other answers here...
The InnoDB memory heap is disabled
This isn’t really an error, just an indication that InnoDB is using the system’s internal memory allocator instead of its own. The default is yes/1, and is acceptable for production.
According to the docs, this command is deprecated, and will be removed in MySQL versions above 5.6 (and I assume MariaDB):
http://dev.mysql.com/doc/refman/5.6/en/innodb-performance-use_sys_malloc.html
Thanks to: Ruben Schade comment
[Note] Plugin 'FEDERATED' is disabled.
The message about FEDERATED disabled is not an error. It just meant that the FEDERATED engine its not ON for your mysql server. It's not used by default. If you don't need it, don't care about this message.
See: https://stackoverflow.com/a/16470822/2586761
I found this answer adds to the discussion: https://www.digitalocean.com/community/questions/mysql-server-keeps-stopping-unexpectedly?answer=26021
In short, on top of setting innodb_buffer_pool_size to something reasonable like 64M, you also may need to modify /etc/apache2/mods-enabled/mpm_prefork.conf to reduce the number of connections started by apache;
<IfModule mpm_prefork_module>
StartServers 3
MinSpareServers 3
MaxSpareServers 5
MaxRequestWorkers 25
MaxConnectionsPerChild 1024
</IfModule>
The solution is NOT more space, Problem is Apache web server not mysql, actually you need to decrease innodb-buffer-pool-size
This buffer is used by the mysql process right off the start, so when Apache needs more resources the kernel will clear RAM from services this means stopping mysql instead of crashing the server.
Would also add a CRON to check the db status and restart it if you dont want to change to ngnx or httplight.
I've downloaded mysql binaries and I'm trying to follow instructions here for installation. when I try to run bin/mysqld_safe --user=mysql & command I get the following output:
40211 19:04:56 mysqld_safe Logging to '/usr/local/mysql/data/irpowerweb.err'.
140211 19:04:56 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data
140211 19:04:57 mysqld_safe mysqld from pid file /usr/local/mysql/data/irpowerweb.pid ended
The content of the error log is:
140211 19:07:01 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data
2014-02-11 19:07:02 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-02-11 19:07:02 19511 [Note] Plugin 'FEDERATED' is disabled.
2014-02-11 19:07:02 19511 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2014-02-11 19:07:02 19511 [Note] InnoDB: The InnoDB memory heap is disabled
2014-02-11 19:07:02 19511 [Note] InnoDB: Mutexes and rw_locks use InnoDB's own implementation
2014-02-11 19:07:02 19511 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-02-11 19:07:02 19511 [Note] InnoDB: Using Linux native AIO
2014-02-11 19:07:02 19511 [Note] InnoDB: Not using CPU crc32 instructions
2014-02-11 19:07:02 19511 [Note] InnoDB: Initializing buffer pool, size = 64.0M
InnoDB: mmap(68370432 bytes) failed; errno 12
2014-02-11 19:07:02 19511 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2014-02-11 19:07:02 19511 [ERROR] Plugin 'InnoDB' init function returned error.
2014-02-11 19:07:02 19511 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2014-02-11 19:07:02 19511 [ERROR] Unknown/unsupported storage engine: InnoDB
2014-02-11 19:07:02 19511 [ERROR] Aborting
2014-02-11 19:07:02 19511 [Note] Binlog end
...
Shutting down stuff
...
2014-02-11 19:07:02 19511 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete
140211 19:07:02 mysqld_safe mysqld from pid file /usr/local/mysql/data/irpowerweb.pid ended
Here's my.cnf:
[mysqld]
innodb_buffer_pool_size = 64M
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
I just want to get mysql running in w/e way. I tried installing the package using yum install mysql-server, downloading rpm packages and failed to get it started. I settled for simple tar package and started following the link above but then again I failed to get it started.
Since the error states Cannot allocate memory for the buffer pool I tried reducing the memory to 64m (default was 128m) and it didn't work.
I'm a newbie in this, I've done a lot of searching around and haven't figured out how to fix this.
P.S. If it help, I'm using ssh to connect to a server with low memory (512MB of memory with no swap), and I think it's a red hat Linux.
I found my answer here. Used the following my.cnf and it worked:
[mysqld]
innodb=OFF
ignore-builtin-innodb
skip-innodb
default-storage-engine=myisam
default-tmp-storage-engine=myisam
Try adding:
[mysqld]
...
skip-bdb
skip-innodb
This will stop the Berkley DB and InnoDB from being started at all.
Your most significant error is:
Cannot allocate memory for the buffer pool
You'll have to lower it, in my.cnf.
When you run free, mysqld_safe is already done cleaning up, so you'd have to monitor memory via another terminal. But it's quite normal that a default my.cnf will use up your available memory. This is one of the reasons why Debian does not ship with a default my.cnf config.
update:
just in case, make sure to:
Follow MySQL Tuner's advices for your system.
Make sure SELinux is either turned off or set to allow mysql
Make sure your stack size is large enough using ulimit -s (you can increase it: ulimit -s <new_size> or even ulimit -s unlimited)