MariaDB service fails to start with InnoDB Space id error - mysql

I've been running MariaDB on a Raspberry Pi 4 (Raspbian 10 / buster) as the database server for my Webtrees and Owncloud sites. Recently, I noticed that the server wasn't looking right, and after a reboot, I can see that the MariaDB service is failing to start.
Contents of /var/log/mysql/error.log:
2021-03-26 14:45:06 0 [Note] InnoDB: Using Linux native AIO
2021-03-26 14:45:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-03-26 14:45:06 0 [Note] InnoDB: Uses event mutexes
2021-03-26 14:45:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-03-26 14:45:06 0 [Note] InnoDB: Number of pools: 1
2021-03-26 14:45:06 0 [Note] InnoDB: Using generic crc32 instructions
2021-03-26 14:45:06 0 [Note] InnoDB: Disabling background log and ibuf IO write threads.
2021-03-26 14:45:06 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-03-26 14:45:06 0 [Note] InnoDB: Completed initialization of buffer pool
2021-03-26 14:45:06 0 [Note] InnoDB: innodb_force_recovery=6 skips redo log apply
2021-03-26 14:45:06 0 [ERROR] InnoDB: Space id and page no stored in the page, read in are [page id: space=0, page number=399], should be [page id: space=0, page number=394]
210326 14:45:06 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.3.27-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 466214 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 12860 12860 processes
Max open files 16384 16384 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 12860 12860 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
Here's the terminal output from trying to start fro the terminal:
$ sudo systemctl restart mariadb.service
Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.
Googling hasn't turned up too much, though I've tried
innodb_force_recovery = 1
innodb_purge_threads=0
In /etc/my.cnf, which didn't seem to change anything.
Any thoughts before I attempt to restore from backup?

Related

CentOS 7 - MySQL 5.7 re-installation corrupted data

I have a CentOS 7 virtual machine with a DirectAdmin installation. It used to run a MySQL 5.6 installation, built using DirectAdmin Custombuild 2.0. I wanted some functionality (e.g. JSON ops) that was not available in MySQL 5.6, so I manually installed MySQL 5.7 from the MySQL repo using the guide on https://www.tecmint.com/install-latest-mysql-on-rhel-centos-and-fedora/.
This worked fine up until a few days ago, when I needed to rebuild my Custombuild installation and it obviously overwrote the MySQL installation with version 5.6. I did NOT make a backup of the databases (I know, really stupid and I regret it).
After the Custombuild overwrite, I re-installed MySQL 5.7 using the same guide. When I now try to start it, I get the following:
[riccardo#server ~]$ sudo systemctl start mysqld
[sudo] password for riccardo:
Job for mysqld.service failed because the control process exited with
error code. See "systemctl status mysqld.service" and "journalctl
-xe" for details.
The log in /var/log/mysqld.log says the following:
2018-05-27T17:42:02.769069Z 0 [Warning] TIMESTAMP with implicit
DEFAULT value is deprecated. Please use
--explicit_defaults_for_timestamp server option (see documentation for
more details).
2018-05-27T17:42:02.770667Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.22)
starting as process 31752 ...
2018-05-27T17:42:02.773699Z 0 [Note] InnoDB: PUNCH HOLE support
available
2018-05-27T17:42:02.773726Z 0 [Note] InnoDB: Mutexes and rw_locks use
GCC atomic builtins
2018-05-27T17:42:02.773731Z 0 [Note] InnoDB: Uses event mutexes
2018-05-27T17:42:02.773734Z 0 [Note] InnoDB: GCC builtin
__atomic_thread_fence() is used for memory barrier
2018-05-27T17:42:02.773739Z 0 [Note] InnoDB: Compressed tables use
zlib 1.2.3
2018-05-27T17:42:02.773742Z 0 [Note] InnoDB: Using Linux native AIO
2018-05-27T17:42:02.773999Z 0 [Note] InnoDB: Number of pools: 1
2018-05-27T17:42:02.774100Z 0 [Note] InnoDB: Using CPU crc32
instructions
2018-05-27T17:42:02.775492Z 0 [Note] InnoDB: Initializing buffer pool,
total size = 128M, instances = 1, chunk size = 128M
2018-05-27T17:42:02.783306Z 0 [Note] InnoDB: Completed initialization
of buffer pool
2018-05-27T17:42:02.786037Z 0 [Note] InnoDB: If the mysqld execution
user is authorized, page cleaner thread priority can be changed. See
the man page of setpriority().
2018-05-27T17:42:02.797885Z 0 [ERROR] [FATAL] InnoDB: Table flags are
0 in the data dictionary but the flags in file ./ibdata1 are 0x4000!
2018-05-27 19:42:02 0x7efd82b32740 InnoDB: Assertion failure in
thread 139627284604736 in file ut0ut.cc line 942
Hence, it looks like the InnoDB data is corrupted. I tried starting using
[mysqld]
innodb_force_recovery = x
in /etc/my.cnf with values 1-6 for x. This didn't help.
As for my question: is there any chance of recovering the data, with or without the help of MySQL?

mysqld.exe has stopped working. Trying to start mysql in XAMPP

Right now I am learning how to code HTML and PHP and use SQL. I am using XAMPP for mysql. Whenever I try to start mysql I always get an error. This is the error log.
2018-02-24 17:22:30 f04 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
2018-02-24 17:22:30 3844 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
2018-02-24 17:22:31 3844 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2018-02-24 17:22:31 3844 [Note] InnoDB: The InnoDB memory heap is disabled
2018-02-24 17:22:31 3844 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2018-02-24 17:22:31 3844 [Note] InnoDB: _mm_lfence() and _mm_sfence() are used for memory barrier
2018-02-24 17:22:31 3844 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-02-24 17:22:31 3844 [Note] InnoDB: Using generic crc32 instructions
2018-02-24 17:22:31 3844 [Note] InnoDB: Initializing buffer pool, size = 16.0M
2018-02-24 17:22:31 3844 [Note] InnoDB: Completed initialization of buffer pool
2018-02-24 17:22:31 3844 [Note] InnoDB: The first specified data file C:\xampp\mysql\data\ibdata1 did not exist: a new database to be created!
2018-02-24 17:22:31 3844 [Note] InnoDB: Setting file C:\xampp\mysql\data\ibdata1 size to 10 MB
2018-02-24 17:22:31 3844 [Note] InnoDB: Setting log file C:\xampp\mysql\data\ib_logfile101 size to 5 MB
2018-02-24 17:22:31 3844 [Note] InnoDB: Setting log file C:\xampp\mysql\data\ib_logfile1 size to 5 MB
2018-02-24 17:22:31 f04 InnoDB: Error: Write to file C:\xampp\mysql\data\ib_logfile101 failed at offset 0.
InnoDB: 512 bytes should have been written, only 0 were written.
InnoDB: Operating system error number 87.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: FormatMessage: Error number 87 means 'The parameter is incorrect.
'.
InnoDB: Error number 87 means 'Unknown error'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2018-02-24 17:22:31 f04 InnoDB: Operating system error number 87 in a file operation.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
InnoDB: Operation os_file_write_func to file D:\win32-packages\build\src\storage\xtradb\os\os0file.cc and at line 5097
2018-02-24 17:22:31 3844 [ERROR] InnoDB: File C:\xampp\mysql\data\ib_logfile101: 'os_file_write_func' returned OS error 287. Cannot continue operation
180224 17:22:31 [ERROR] mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.1.30-MariaDB
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=1001
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 787129 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
mysqld.exe!my_parameter_handler()
mysqld.exe!strxnmov()
mysqld.exe!?compare_ulonglong##YAHPB_K0#Z()
mysqld.exe!no_key()
mysqld.exe!no_key()
mysqld.exe!no_key()
mysqld.exe!no_key()
mysqld.exe!no_key()
mysqld.exe!no_key()
mysqld.exe!no_key()
mysqld.exe!?compare_ulonglong##YAHPB_K0#Z()
mysqld.exe!?compare_ulonglong##YAHPB_K0#Z()
mysqld.exe!?str_result#Item##UAEPAVString##PAV2##Z()
mysqld.exe!?ha_initialize_handlerton##YAHPAUst_plugin_int###Z()
mysqld.exe!?plugin_init##YAHPAHPAPADH#Z()
mysqld.exe!?plugin_init##YAHPAHPAPADH#Z()
mysqld.exe!?init_net_server_extension##YAXPAVTHD###Z()
mysqld.exe!?win_main##YAHHPAPAD#Z()
mysqld.exe!?mysql_service##YAHPAX#Z()
mysqld.exe!strxnmov()
KERNEL32.DLL!BaseThreadInitThunk()
ntdll.dll!RtlGetAppContainerNamedObjectPath()
ntdll.dll!RtlGetAppContainerNamedObjectPath()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
I have tried deleting the ibdata1, ib_logfile1 and ib_logfile0 and the mysql_error in the C:\xampp\mysql\data folder but it hasn't helped. I dont know what to do and how to fix it. I have read other questions on this website and topic but none of the solutions given help me. Please help!

Repetitive mysqld.exe crashes - OS error number 995 & mysql Exception 0x80000003

In the last few weeks our mysql DB has been crashing randomly. -
I have ran a check against all the databases for corruption but all are OK.
EDIT 2016.01.14 - The error that keeps popping up commonly is the following -
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 0.
Although, the 'File Read Page' is always different, the last time it was Zero, but before that it was 4500.
A little background.
Been running Xampp for over a year without a problem
We have Applications constantly inserting data into MYSQL from lots of data sources, this is 24/7.
Apart from the above applications, nothing else runs around the time of the crash, I have also checked the system logs to see if anything sticks out but it doesn't.
Two weeks ago, we moved to another server with better spec, the old server was cloned and place on the new one.
Nothing has been altered in DB or new tables added.
Within a week, mysql started to produce errors.
The new server specs are as follows -
Windows Server 2008 R2 Standard - Running Service Pack 1
Proccessor: Intel(R) Xeon(R) # 3.3ghz 2x Dual processor
8GB Ram
64Bit OS (Same as old server)
MySQL Version: 5.6.20
PHP Version 5.5.15
Here are the recent error logs -
*************MYSQL ERROR LOG*****************
//////////////////////////////////*********************
ERROR on 2016-01-05
******************///////////////////////
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
2016-01-05 08:42:52 cc8 InnoDB: Operating system error number 995 in a file operation.
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
2016-01-05 08:42:52 cc8 InnoDB: Operating system error number 995 in a file operation.
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
2016-01-05 08:42:53 cc8 InnoDB: Operating system error number 995 in a file operation.
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
2016-01-05 08:42:53 cc8 InnoDB: Operating system error number 995 in a file operation.
InnoDB: The error means that the I/O operation has been aborted
InnoDB: because of either a thread exit or an application request.
InnoDB: Retry attempt is made.
//////////////////////////////////*********************
Error- 2015-12-22
********************************///////////////////////////
2015-12-22 16:46:02 12d4 InnoDB: uncompressed page, stored checksum in field1 1169359801, calculated checksums for field1: crc32 4003191917, innodb 1169359801, none 3735928559, stored checksum in field2 4049981449, calculated checksums for field2: crc32 4003191917, innodb 587193584, none 3735928559, page LSN 128 4137924960, low 4 bytes of LSN at page end 4137853843, page number (if stored to page already) 0, space id (if created with >= MySQL-4.1.1 and stored already) 252
InnoDB: Page may be a file space header page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 0.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 0.
InnoDB: You may have to recover from a backup.
2015-12-22 16:46:02 12d4 InnoDB: Page dump in ascii and hex (16384 bytes): // Lots of binary here ..
InnoDB: End of page dump
2015-12-22 16:46:03 12d4 InnoDB: uncompressed page, stored checksum in field1 1169359801, calculated checksums for field1: crc32 4003191917, innodb 1169359801, none 3735928559, stored checksum in field2 4049981449, calculated checksums for field2: crc32 4003191917, innodb 587193584, none 3735928559, page LSN 128 4137924960, low 4 bytes of LSN at page end 4137853843, page number (if stored to page already) 0, space id (if created with >= MySQL-4.1.1 and stored already) 252
InnoDB: Page may be a file space header page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 0.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Error: Unable to read tablespace 252 page no 0 into the buffer pool after 100 attempts
InnoDB: The most probable cause of this error may be that the table has been corrupted.
InnoDB: You can try to fix this problem by using innodb_force_recovery.
InnoDB: Please see reference manual for more details.
InnoDB: Aborting...
2015-12-22 16:46:03 12d4 InnoDB: Assertion failure in thread 4820 in file buf0buf.cc line 2641
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:46:03 UTC - mysqld got exception 0x80000003 ;
This could be because you hit a bug.
It is also possible that this binary
or one of the libraries it was linked against is corrupt,
improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem,
but since we have already crashed,
something is definitely wrong and this may
fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=107
max_threads=151
thread_count=5
connection_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133778 K bytes
of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x22020238
Attempting backtrace. You can use the following information to find out
where mysqld died.
If you see no messages after this, something went
terribly wrong...
50a500
mysqld.exe!my_thread_name()
74392d mysqld.exe!my_mb_ctype_mb()
5ecdb6
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
61d611
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
647350
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
6492fa
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
649383
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
6493b3
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
64945d
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
6495cb
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
5d48e3
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
53f011
mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
2be48e
mysqld.exe!?ha_write_row#handler##QAEHPAE#Z()
3d8d10
mysqld.exe!?write_record##YAHPAVTHD##PAUTABLE##PAVCOPY_INFO##2#Z()
3df599
mysqld.exe!?mysql_insert##YA_NPAVTHD##PAUTABLE_LIST##AAV?$List#VItem####AAV?$List#V?$List#VItem######22W4enum_duplicates##_N#Z()
2ecdf5
mysqld.exe!?mysql_execute_command##YAHPAVTHD###Z()
2ef81e
mysqld.exe!?mysql_parse##YAXPAVTHD##PADIPAVParser_state###Z()
2f06f8
mysqld.exe!?dispatch_command##YA_NW4enum_server_command##PAVTHD##PADI#Z()
2f135a
mysqld.exe!?do_command##YA_NPAVTHD###Z()
364e59 mysqld.exe!?do_handle_one_connection##YAXPAVTHD###Z()
364efd
mysqld.exe!handle_one_connection()
6bb8cb mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
510336
mysqld.exe!win_pthread_mutex_trylock()
746c00 mysqld.exe!my_mb_ctype_mb()
746c8a
mysqld.exe!my_mb_ctype_mb()
7728338a kernel32.dll!BaseThreadInitThunk()
77c297f2
ntdll.dll!RtlInitializeExceptionChain()
77c297c5
ntdll.dll!RtlInitializeExceptionChain()
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (21e3387c): INSERT INTO 'WPM'(
`Pos`,
`FileMod`,
`Duration`) VALUES (
'IWM',
'2015-12-14 07:47:47 ',
0)Connection ID (thread ID): 3722
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2015-12-23 08:21:58 4920
[Note] Plugin 'FEDERATED' is disabled.
2015-12-23 08:21:58 10c4 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
2015-12-23 08:21:58 4920 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-12-23 08:21:58 4920 [Note] InnoDB: The InnoDB memory heap is disabled
2015-12-23 08:21:58 4920 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2015-12-23 08:21:58 4920 [Note] InnoDB: Memory barrier is not used
2015-12-23 08:21:58 4920 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-12-23 08:21:58 4920 [Note] InnoDB: Not using CPU crc32 instructions
2015-12-23 08:21:58 4920 [Note] InnoDB: Initializing buffer pool, size = 16.0M
2015-12-23 08:21:58 4920 [Note] InnoDB: Completed initialization of buffer pool
2015-12-23 08:21:58 4920 [Note] InnoDB: Highest supported file format is Barracuda.
2015-12-23 08:21:58 4920 [Note] InnoDB: Log scan progressed past the checkpoint lsn 553893690098
2015-12-23 08:21:58 4920 [Note] InnoDB: Database was not shutdown normally!
2015-12-23 08:21:58 4920 [Note] InnoDB: Starting crash recovery.
2015-12-23 08:21:58 4920 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-12-23 08:21:59 4920 [Note] InnoDB: Restoring possible half-written data pages
2015-12-23 08:21:59 4920 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 553893828751
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 151 row operations to undo
InnoDB: Trx id counter is 152858880
2015-12-23 08:22:00 4920 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2015-12-23 08:22:00 4920 [Note] InnoDB: 128 rollback segment(s) are active.
InnoDB: Starting in background the rollback of uncommitted transactions
2015-12-23 08:22:00 2c4 InnoDB: Rolling back trx with id 152858375, 151 rows to undo
2015-12-23 08:22:00 4920 [Note] InnoDB: Waiting for purge to start
2015-12-23 08:22:00 4920 [Note] InnoDB: Rollback of trx with id 152858375 completed
2015-12-23 08:22:00 2c4 InnoDB: Rollback of non-prepared transactions completed
2015-12-23 08:22:00 4920 [Note] InnoDB: 5.6.20 started; log sequence number 553893828751
2015-12-23 08:22:00 4920 [Note] Server hostname (bind-address): '*'; port: 3306
2015-12-23 08:22:00 4920 [Note] IPv6 is available.
2015-12-23 08:22:00 4920 [Note] - '::' resolves to '::';
2015-12-23 08:22:00 4920 [Note] Server socket created on IP: '::'.
2015-12-23 08:22:00 4920 [Note] Event Scheduler: Loaded 11 events
2015-12-23 08:22:00 4920 [Note] c:\xampp\mysql\bin\mysqld.exe: ready for connections.
Version: '5.6.20' socket: '' port: 3306 MySQL Community Server (GPL)
2015-12-23 08:24:07 5736 [Note] Plugin 'FEDERATED' is disabled.
2015-12-23 08:24:07 fc8 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
2015-12-23 08:24:07 5736 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-12-23 08:24:07 5736 [Note] InnoDB: The InnoDB memory heap is disabled
2015-12-23 08:24:07 5736 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
2015-12-23 08:24:07 5736 [Note] InnoDB: Memory barrier is not used
2015-12-23 08:24:07 5736 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-12-23 08:24:07 5736 [Note] InnoDB: Not using CPU crc32 instructions
2015-12-23 08:24:07 5736 [Note] InnoDB: Initializing buffer pool, size = 16.0M
2015-12-23 08:24:07 5736 [Note] InnoDB: Completed initialization of buffer pool
2015-12-23 08:24:07 5736 [Note] InnoDB: Highest supported file format is Barracuda.
2015-12-23 08:24:07 5736 [Note] InnoDB: Log scan progressed past the checkpoint lsn 553896050587
2015-12-23 08:24:07 5736 [Note] InnoDB: Database was not shutdown normally!
2015-12-23 08:24:07 5736 [Note] InnoDB: Starting crash recovery.
2015-12-23 08:24:07 5736 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-12-23 08:24:07 5736 [Note] InnoDB: Restoring possible half-written data pages
2015-12-23 08:24:07 5736 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 553896774958
2015-12-23 08:24:07 5736 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
//////////////////////////////////*********************
Error on 2016-01-08
********************////////////////////////
2016-01-08 00:00:14 1584 InnoDB: uncompressed page, stored checksum in field1 1299653197, calculated checksums for field1: crc32 90466284, innodb 1299653197, none 3735928559, stored checksum in field2 341870525, calculated checksums for field2: crc32 90466284, innodb 3590763946, none 3735928559, page LSN 129 3920058837, low 4 bytes of LSN at page end 3920015931, page number (if stored to page already) 75791, space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an update undo log page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 75791.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
2016-01-08 00:00:14 1584 InnoDB: Assertion failure in thread 5508 in file buf0buf.cc line 4195
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
00:00:14 UTC - mysqld got exception 0x80000003 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=113
max_threads=151
thread_count=5
connection_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 133778 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
116a500 mysqld.exe!my_thread_name()
13a392d mysqld.exe!my_mb_ctype_mb()
12c7aa2 mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
12c82ad mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
1225a72 mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11d423b mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11d4694 mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11d548a mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11b147d mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
11b167d mysqld.exe!?my_aes_create_key##YAXPBEIPAEW4my_aes_opmode###Z()
7728338a kernel32.dll!BaseThreadInitThunk()
77c297f2 ntdll.dll!RtlInitializeExceptionChain()
77c297c5 ntdll.dll!RtlInitializeExceptionChain()
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
InnoDB: Warning: a long semaphore wait:
--Thread 6764 has waited at trx0undo.cc line 1778 for 29770.00 seconds the semaphore:
Mutex at 1B4F60EC created file trx0rseg.cc line 196, lock var 1
waiters flag 1
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
Here is the mysql.ini settings file (If this helps)
# The MySQL server
[mysqld]
port= 3306
socket = "C:/xampp/mysql/mysql.sock"
basedir = "C:/xampp/mysql"
tmpdir = "C:/xampp/tmp"
datadir = "C:/xampp/mysql/data"
pid_file = "mysql.pid"
# enable-named-pipe
key_buffer = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error = "mysql_error.log"
# Change here for bind listening
# bind-address="127.0.0.1"
# bind-address = ::1 # for ipv6
# Where do all the plugins live
plugin_dir = "C:/xampp/mysql/lib/plugin/"
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
#
# commented in by lampp security
#skip-networking
skip-federated
# Replication Master Server (default)
# binary logging is required for replication
# log-bin deactivated by default since XAMPP 1.4.11
#log-bin=mysql-bin
# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
# MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
# where you replace <host>, <user>, <password> by quoted strings and
# <port> by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host = <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user = <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password = <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port = <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin
# Point the following paths to different dedicated disks
#tmpdir = "C:/xampp/tmp"
#log-update = /path-to-dedicated-directory/hostname
# Uncomment the following if you are using BDB tables
#bdb_cache_size = 4M
#bdb_max_lock = 10000
# Comment the following if you are using InnoDB tables
#skip-innodb
innodb_data_home_dir = "C:/xampp/mysql/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "C:/xampp/mysql/data"
#innodb_log_arch_dir = "C:/xampp/mysql/data"
## You can set .._buffer_pool_size up to 50 - 80 %
## of RAM but beware of setting memory usage too high
innodb_buffer_pool_size = 16M
innodb_additional_mem_pool_size = 2M
## Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
## UTF 8 Settings
#init-connect=\'SET NAMES utf8\'
#collation_server=utf8_unicode_ci
#character_set_server=utf8
#skip-character-set-client-handshake
#character_sets-dir="C:/xampp/mysql/share/charsets"
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
I simply have no idea what to do, I have no experience in the technical side to troubleshooting Mysql. I have looked on the MYsql site, are apparently error code 995 is a known bug, it was meant to be fixed, but I read that it never got pushed into the build.
If you need any more info please ask!
InnoDB deliberately crashes MySQL if it reads a page from "disk" (might come from RAM though if the relevant blocks are in OS cache) and the checksum mismatches with the one in the page header. This is what's happening to you.
Reasons could be:
If you're not using ECC ram you might just have some corrupted data in OS cache. Rebooting the server will clear the cache and you can get some ECC ram to prevent it happening. (less likely)
On disk corruption (more likely):
If you're lucky the corruption is on index pages and you can fully recover by doing a innodb_recovery (http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html)
If the corruption is on data page you most certainly will lose data but you still should be able to recover most of the information using the same method.
In such cases I would just restore from backup if that's possible. The other error you pasted also indicates system level I/O issues so I would look into it soon before it crashes completely.
I believe some pages in Innodb tablespace got corrupted..
First backup your DB!
First suggestion would be to run check table using this command:
mysqlcheck -A --auto-repair
Reference: Here
Add the following line below the mysqld section in the mysql config file (my.ini) and restart both the apache and mysql service afterwards.
innodb_force_recovery = 1
For your information: read this
Note: Innodb becomes read only for data operations when running in innodb_force_recovery mode
If this is not working, my second suggestion would be to decrease the innodb_buffer_pool_size and convert some "heavey loading" tables into MyISAM:
ALTER TABLE table_name ENGINE=MYISAM
Although you increased the specs of your machine, due to high usage of your DB table can still cause a problem.
(Please update me and which you good luck!)

MySQL has gone away on large queries

I have a MySQL installed and running on a CentOS 6.6 and MySQL version 5.5.40 on RackSpace. I always run into this error when running heavy queries.
Here is the settings of my.cnf
[mysqld]
2 datadir=/mnt/data/mysql
3 tmpdir=/mnt/data/temp
4 socket=/var/lib/mysql/mysql.sock
5 bind-address=0.0.0.0
6 port=3306
7 wait_timeout=432000
8 max_allowed_packet=1G
9 max_connections=500
10 query-cache-size=0
11 query-cache-type=0
12 #query_cache_size=64M
13 #query_cache_limit=64M
14 key_buffer_size=1G
15 sort_buffer_size=16M
16 tmp_table_size=32M
17 max_heap_table_size=32M
18 read_buffer_size=512K
19 read_rnd_buffer_size=512K
20 thread_cache_size=50
21
22 innodb_buffer_pool_size=12G
23 innodb_buffer_pool_instance=2
24 innodb_read_io_threads=12
25 innodb_write_io_threads=12
26 innodb_io_capacity=300
27 innodb_log_file_size=128M
28 innodb_thread_concurrency=0
Here is the error log I've caught after the crash:
150820 13:46:26 mysqld_safe Number of processes running now: 0
150820 13:46:26 mysqld_safe mysqld restarted
150820 13:46:26 [Note] Plugin 'FEDERATED' is disabled.
150820 13:46:26 [Warning] Using unique option prefix innodb_buffer_pool_instance instead of innodb-buffer-pool-instances is deprecated and will be removed in a future release. Please use the full name instead.
150820 13:46:26 InnoDB: The InnoDB memory heap is disabled
150820 13:46:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150820 13:46:26 InnoDB: Compressed tables use zlib 1.2.3
150820 13:46:26 InnoDB: Using Linux native AIO
150820 13:46:26 InnoDB: Initializing buffer pool, size = 12.0G
InnoDB: mmap(6593445888 bytes) failed; errno 12
150820 13:46:27 InnoDB: Completed initialization of buffer pool
150820 13:46:27 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150820 13:46:27 [ERROR] Plugin 'InnoDB' init function returned error.
150820 13:46:27 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150820 13:46:27 [ERROR] Unknown/unsupported storage engine: InnoDB
150820 13:46:27 [ERROR] Aborting
150820 13:46:27 [Note] /usr/libexec/mysqld: Shutdown complete
150820 13:46:27 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
EDIT:
RackSpace VM's specs are:
CPU: Intel(R) Xeon(R) CPU E5-2670 0 # 2.60GHz
RAM: 8GB
your server has only 8 GB RAM and you have assigned too much ram to mysql.
Even you need to change your configuration for many variables but first to comeout your issue do the below changes-
innodb_buffer_pool_instance=2 #comment it for the time being we can set it later.
innodb_buffer_pool_size=6G
key_buffer_size=20M #if your server is innodb but if you are also using myisam tables then keep as it is.
sort_buffer_size=2M # we can change later.
read_buffer_size=512K # comment it for time being.
read_rnd_buffer_size=512K #comment it for time being as used per session
tmp_table_size=1G #this can be reason of your problem so increase it.
max_heap_table_size=1G #this can be reason of your problem so increase it.
If possible decrease max_connections from 500 to 400 as each connection uses server resources.
Try and share the results.
In case it is useful, this is how I solve this problem. In my case the problem was caused by queries that were too large for the packets being sent to the server.
After running the following command against the server, the large queries process fine
SET GLOBAL max_allowed_packet=1073741824;

MySQL fails on restart, InnoDB me alloc error 12

I am having a problem with MySQL on my Rackspace Centos 6.4 server instance. The problem is similar to the one described in this StackOverflow question. MySQL is being restarted automatically at some point by mysqld_safe, and the restart fails because InnoDB tries to allocate 128Mb of RAM, which fails. The output of mysqld.log is as follows:
140129 18:05:26 mysqld_safe Number of processes running now: 0
140129 18:07:30 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140129 18:07:30 InnoDB: Compressed tables use zlib 1.2.3
140129 18:07:30 InnoDB: Using Linux native AIO
140129 18:07:35 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
140129 18:07:46 InnoDB: Completed initialization of buffer pool
140129 18:07:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
140129 18:07:47 [ERROR] Plugin 'InnoDB' init function returned error.
140129 18:07:47 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140129 18:08:07 [ERROR] Unknown/unsupported storage engine: InnoDB
140129 18:08:10 [ERROR] Aborting
140129 18:08:53 [Note] /usr/libexec/mysqld: Shutdown complete
140129 18:18:18 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
The solution provided in that other question seemed to be one of "create a swap file". I have checked my server and it seems there is already an active swap file:
# swapon -s
Filename Type Size Used Priority
/dev/xvdc1 partition 499992 34876 -1
and, looking at that output, it is the size I was thinking I needed (512Mb). For completeness, here are the contents of my /etc/fstab file:
/dev/xvda1 / ext3 defaults,noatime,barrier=0 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/xvdc1 none swap sw 0 0
So am I missing something, or do I already have a working swap file, of about 512Mb, which is reasonably empty and therefore should be capable of handling a request for 128Mb? Should I reduce the size of the InnoDB buffer to, say, 64Mb? Will there be any issues related to shrinking this buffer?
(My Rackspace server is the smallest one available, which has 512Mb RAM. Whenever I do a top on the server, it seems to have between 50 and 80 Mb free.)
From the output it does look like you have approximately 488MB of swap space.
I am not sure if MySQL allocates the innodb buffer pool based off how much memory+swap is free. Even if it did, you would want to avoid it going to swap as it is slower than keeping things in RAM. My guess is that it does not include swap.
The error "InnoDB: mmap(137363456 bytes) failed; errno 12" lets us know that you could not allocate the memory.
# perror 12
OS error code 12: Cannot allocate memory
I would reduce the size of the innodb buffer pool to 64MB; see if that works. If it does not, either increase the size of the cloud server or reduce the size of the buffer pool again.
In general InnoDB likes memory. It tries to keep as much of the data in memory as possible and reduce disk IO. By reducing the buffer size, you decrease how much MySQL can keep in memory. MySQL will go to disk more often to retrieve data.
This number should reflect the size of your data set. No point in having a buffer much larger than your actual data set.
You may be able to use some of the queries here to determine the size. http://www.mysqlperformanceblog.com/2008/03/17/researching-your-mysql-table-sizes/
If your data set is much larger than the amount of RAM you normally have on hand, you may need to increase the size of the server itself.