Mysql server crash on start on my Ubuntu 12.04.2 server. Mysql just not starting. Not after reboot, not after service mysql start(it return start: Job failed to start).
Previously it worked fine, but at some point the server crashed and I was not able to run it.
/var/log/mysql/error.log:
151109 10:02:14 [Note] Plugin 'FEDERATED' is disabled.
151109 10:02:14 InnoDB: The InnoDB memory heap is disabled
151109 10:02:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151109 10:02:14 InnoDB: Compressed tables use zlib 1.2.3.4
151109 10:02:14 InnoDB: Initializing buffer pool, size = 128.0M
151109 10:02:14 InnoDB: Completed initialization of buffer pool
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 5.
InnoDB: You may have to recover from a backup.
151109 10:02:15 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 76607...(I removed the part)...3c3f438045ca5727; asc v` E W1 9 - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? ! " # $ % & ' ( ) * + , _ # _ # jPs <?C E W';
InnoDB: End of page dump
151109 10:02:15 InnoDB: Page checksum 1986035693, prior-to-4.0.14-form checksum 3548252131
InnoDB: stored checksum 1986035693, prior-to-4.0.14-form stored checksum 1010779008
InnoDB: Page lsn 0 1170888497, low 4 bytes of lsn at page end 1170888487
InnoDB: Page number (if stored to page already) 5,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a transaction system page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 5.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
151109 10:02:15 InnoDB: Assertion failure in thread 3065161472 in file buf0buf.c line 3629
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:02:15 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346064 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb733d003]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb71e9f24]
[0xb6ea0400]
/usr/sbin/mysqld(+0x590e78)[0xb7453e78]
/usr/sbin/mysqld(+0x591852)[0xb7454852]
/usr/sbin/mysqld(+0x58109a)[0xb744409a]
/usr/sbin/mysqld(+0x54f75d)[0xb741275d]
/usr/sbin/mysqld(+0x551d09)[0xb7414d09]
/usr/sbin/mysqld(+0x53d5e0)[0xb74005e0]
/usr/sbin/mysqld(+0x501a22)[0xb73c4a22]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4e)[0xb71ec74e]
/usr/sbin/mysqld(+0x1fbd82)[0xb70bed82]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xbc3)[0xb70c2723]
/usr/sbin/mysqld(+0x16691a)[0xb702991a]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x66a)[0xb702d61a]
/usr/sbin/mysqld(main+0x27)[0xb7022d27]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb6b5f4d3]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Related
My mysql server couldn' t not restart anymore
i was doing the installation of a wordpress plugin, it crashed, then i tried to relaunch the service, I have the error:
root#ns3371000:~# /etc/init.d/mysql start
[FAIL] Starting MySQL database server: mysqld . . . . . . . . . . . . . . failed!
This is the log when i launch it in safe mode:
mysqld_safe Logging to '/var/log/mysql.err'.
180822 16:33:07 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
180822 16:33:07 [Note] Plugin 'FEDERATED' is disabled.
180822 16:33:07 InnoDB: The InnoDB memory heap is disabled
180822 16:33:07 InnoDB: Mutexes and rw_locks use GCC atomic builtins
180822 16:33:07 InnoDB: Compressed tables use zlib 1.2.7
180822 16:33:07 InnoDB: Using Linux native AIO
180822 16:33:07 InnoDB: Initializing buffer pool, size = 128.0M
180822 16:33:07 InnoDB: Completed initialization of buffer pool
180822 16:33:07 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 12059835
180822 16:33:07 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: tried to read 65536 bytes at offset 0 2358272.
InnoDB: Was only able to read 1024.
InnoDB: Fatal error: cannot read from file. OS error number 17.
180822 16:33:10 InnoDB: Assertion failure in thread 140608317830944 in file os0file.c line 2549
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
14:33:10 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346701 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x56449ac71dc9]
/usr/sbin/mysqld(handle_fatal_signal+0x3d8)[0x56449ab57dc8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf0a0)[0x7fe1ec9460a0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fe1eb1d6125]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7fe1eb1d93a0]
/usr/sbin/mysqld(+0x67926b)[0x56449ada926b]
/usr/sbin/mysqld(+0x63eca0)[0x56449ad6eca0]
/usr/sbin/mysqld(+0x66b906)[0x56449ad9b906]
/usr/sbin/mysqld(+0x670eb7)[0x56449ada0eb7]
/usr/sbin/mysqld(+0x5ce70f)[0x56449acfe70f]
/usr/sbin/mysqld(+0x59aa2f)[0x56449accaa2f]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x41)[0x56449ab5a151]
/usr/sbin/mysqld(+0x336507)[0x56449aa66507]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xa73)[0x56449aa69553]
/usr/sbin/mysqld(+0x2bb695)[0x56449a9eb695]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x45b)[0x56449a9ec30b]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fe1eb1c2ead]
/usr/sbin/mysqld(+0x2b2c89)[0x56449a9e2c89]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
I don' t want to loose all my current databases, anything pointing me to sort out the issues would be greatful
Thanks
If the database is broken you can try to access it in a recovery mode.
In your configuration file (my.ini) try to add:
[mysqld]
innodb_force_recovery = N
where N is from 1 to 6. Then, restart the server.
More information here: https://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
At least, you should be able to access the database in read-only mode to do the dump.
I would like to ask your support to solve my problem, I have a server in mysql which suffered a download, to ignite I realize that the MySQL service had died.
When you want to start it simply mark the following error in console.
Job failed to start
Reviewing the mysql error.log file here is what it shows:
170411 11:11:30 [Note] Plugin 'FEDERATED' is disabled.
170411 11:11:30 InnoDB: The InnoDB memory heap is disabled
170411 11:11:30 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170411 11:11:30 InnoDB: Compressed tables use zlib 1.2.3.4
170411 11:11:30 InnoDB: Initializing buffer pool, size = 128.0M
170411 11:11:30 InnoDB: Completed initialization of buffer pool
170411 11:11:30 InnoDB: highest supported file format is Barracuda.
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is F6B400
InnoDB: Starting in background the rollback of uncommitted transactions
170411 11:11:32 InnoDB: Rolling back trx with id F6B22B, 1 rows to undo
170411 11:11:32 InnoDB: Waiting for the background threads to start
170411 11:11:32 InnoDB: Assertion failure in thread 2791775040 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:11:32 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346075 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb72f0a43]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb719cfa4]
[0xb6e50400]
/usr/sbin/mysqld(+0x5cb933)[0xb743e933]
/usr/sbin/mysqld(+0x5490e7)[0xb73bc0e7]
/usr/sbin/mysqld(+0x55f4c2)[0xb73d24c2]
/usr/sbin/mysqld(+0x556c54)[0xb73c9c54]
/usr/sbin/mysqld(+0x558093)[0xb73cb093]
/usr/sbin/mysqld(+0x5510ba)[0xb73c40ba]
/usr/sbin/mysqld(+0x610f7a)[0xb7483f7a]
/usr/sbin/mysqld(+0x611524)[0xb7484524]
/usr/sbin/mysqld(+0x551645)[0xb73c4645]
/usr/sbin/mysqld(+0x551d19)[0xb73c4d19]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb6dc9d4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb6bd8ace]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170411 11:11:32 [Note] Plugin 'FEDERATED' is disabled.
170411 11:11:32 InnoDB: The InnoDB memory heap is disabled
170411 11:11:32 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170411 11:11:32 InnoDB: Compressed tables use zlib 1.2.3.4
170411 11:11:32 InnoDB: Initializing buffer pool, size = 128.0M
170411 11:11:32 InnoDB: Completed initialization of buffer pool
170411 11:11:32 InnoDB: highest supported file format is Barracuda.
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is F6B400
170411 11:11:32 InnoDB: Waiting for the background threads to start
InnoDB: Starting in background the rollback of uncommitted transactions
170411 11:11:32 InnoDB: Rolling back trx with id F6B22B, 1 rows to undo
170411 11:11:32 InnoDB: Assertion failure in thread 2791775040 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:11:32 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346075 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb731ca43]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb71c8fa4]
[0xb6e7c400]
/usr/sbin/mysqld(+0x5cb933)[0xb746a933]
/usr/sbin/mysqld(+0x5490e7)[0xb73e80e7]
/usr/sbin/mysqld(+0x55f4c2)[0xb73fe4c2]
/usr/sbin/mysqld(+0x556c54)[0xb73f5c54]
/usr/sbin/mysqld(+0x558093)[0xb73f7093]
/usr/sbin/mysqld(+0x5510ba)[0xb73f00ba]
/usr/sbin/mysqld(+0x610f7a)[0xb74aff7a]
/usr/sbin/mysqld(+0x611524)[0xb74b0524]
/usr/sbin/mysqld(+0x551645)[0xb73f0645]
/usr/sbin/mysqld(+0x551d19)[0xb73f0d19]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb6df5d4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb6c04ace]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
170411 11:11:33 [Note] Plugin 'FEDERATED' is disabled.
170411 11:11:33 InnoDB: The InnoDB memory heap is disabled
170411 11:11:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170411 11:11:33 InnoDB: Compressed tables use zlib 1.2.3.4
170411 11:11:33 InnoDB: Initializing buffer pool, size = 128.0M
170411 11:11:33 InnoDB: Completed initialization of buffer pool
170411 11:11:33 InnoDB: highest supported file format is Barracuda.
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 1 row operations to undo
InnoDB: Trx id counter is F6B400
170411 11:11:33 InnoDB: Waiting for the background threads to start
InnoDB: Starting in background the rollback of uncommitted transactions
170411 11:11:33 InnoDB: Rolling back trx with id F6B22B, 1 rows to undo
170411 11:11:33 InnoDB: Assertion failure in thread 2791775040 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:11:33 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346075 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb72c5a43]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb7171fa4]
[0xb6e25400]
/usr/sbin/mysqld(+0x5cb933)[0xb7413933]
/usr/sbin/mysqld(+0x5490e7)[0xb73910e7]
/usr/sbin/mysqld(+0x55f4c2)[0xb73a74c2]
/usr/sbin/mysqld(+0x556c54)[0xb739ec54]
/usr/sbin/mysqld(+0x558093)[0xb73a0093]
/usr/sbin/mysqld(+0x5510ba)[0xb73990ba]
/usr/sbin/mysqld(+0x610f7a)[0xb7458f7a]
/usr/sbin/mysqld(+0x611524)[0xb7459524]
/usr/sbin/mysqld(+0x551645)[0xb7399645]
/usr/sbin/mysqld(+0x551d19)[0xb7399d19]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xb6d9ed4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb6badace]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
I would like to have your help, sorry if it is obvious the error in the log but I am not an expert in servers I only have a little knowledge.
Is it possible to drop a table/database manually if it is corrupt? How can I clear the ibdata doublewrite buffer?
I just restored my server. However mysql is not coming up. The error message recomments to set innodb_force_recovery=6. Which allowed to start mysql. So I identified the defect table. However no matter what I try check / repair / drop the corrupt table / database which always ends up in mysql stopping and throwing the error:
ERROR 2013 (HY000): Lost connection to MySQL server during query
Here is the mysql error log from starting mysql:
140702 13:40:36 [Note] Plugin 'FEDERATED' is disabled.
140702 13:40:36 [Warning] option 'innodb-buffer-pool-size': signed value 2097152 adjusted to 5242880
140702 13:40:36 [Warning] option 'innodb-additional-mem-pool-size': signed value 512000 adjusted to 524288
140702 13:40:36 InnoDB: The InnoDB memory heap is disabled
140702 13:40:36 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140702 13:40:36 InnoDB: Compressed tables use zlib 1.2.3.4
140702 13:40:36 InnoDB: Initializing buffer pool, size = 5.0M
140702 13:40:36 InnoDB: Completed initialization of buffer pool
140702 13:40:36 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
140702 13:40:37 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
140702 13:40:37 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
140702 13:40:37 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Warning: database page corruption or a failed
InnoDB: file read of space 0 page 79918.
InnoDB: Trying to recover it from the doublewrite buffer.
InnoDB: Dump of the page:
140702 13:40:37 InnoDB: Page dump in ascii and hex (16384 bytes):
Hexdata
InnoDB: End of page dump
140702 13:40:37 InnoDB: Page checksum 3463763561, prior-to-4.0.14-form checksum 3074131693
InnoDB: stored checksum 2986557383, prior-to-4.0.14-form stored checksum 3074131693
InnoDB: Page lsn 3 3511673317, low 4 bytes of lsn at page end 3511673317
InnoDB: Page number (if stored to page already) 79918,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 5918
InnoDB: Dump of corresponding page in doublewrite buffer:
140702 13:40:37 InnoDB: Page dump in ascii and hex (16384 bytes):
Hexdata
...
InnoDB: End of page dump
140702 13:40:37 InnoDB: Page checksum 1693035738, prior-to-4.0.14-form checksum 533526999
InnoDB: stored checksum 1060638083, prior-to-4.0.14-form stored checksum 533526999
InnoDB: Page lsn 3 3511586897, low 4 bytes of lsn at page end 3511586897
InnoDB: Page number (if stored to page already) 79918,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 5918
InnoDB: Also the page in the doublewrite buffer is corrupt.
InnoDB: Cannot continue operation.
InnoDB: You can try to recover the database with the my.cnf
InnoDB: option:
InnoDB: innodb_force_recovery=6
140702 13:40:37 InnoDB: Assertion failure in thread 3064526592 in file trx0sys.c line 604
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
11:40:37 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346063 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb72a35c3]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb7150004]
[0xb6e0c500]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Is there any way to drop a database / table while mysql is offline?
Is it possible to clear the ibdata doublewrite buffer since this is blogging mysql from starting normally?
Our server has been running mysql just fine for over a year. I ran a set of sql script to build a rather large database and in the middle of those scripts, I started getting errors that I had lost connection. Nobody did anything else happened as far as we know. When I tried to log in to mysql, I got:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
When I try to restart mysql, I get:
# sudo service mysql restart
stop: Unknown instance:
start: Job failed to start
The error.log shows:
130212 9:37:51 [Note] Plugin 'FEDERATED' is disabled.
130212 9:37:51 InnoDB: The InnoDB memory heap is disabled
130212 9:37:51 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130212 9:37:51 InnoDB: Compressed tables use zlib 1.2.7
130212 9:37:51 InnoDB: Using Linux native AIO
130212 9:37:51 InnoDB: Initializing buffer pool, size = 10.0G
130212 9:37:51 InnoDB: Completed initialization of buffer pool
130212 9:37:51 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 2186809272046
130212 9:37:51 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2186814514688
InnoDB: Doing recovery: scanned up to log sequence number 2186816162838
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 27964 row operations to undo
InnoDB: Trx id counter is 18834200
130212 9:37:51 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 4424818.
InnoDB: You may have to recover from a backup.
130212 9:37:51 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 058425e20043847200356a3f003e3720000001fd2807769645bf00000000000000000000000000ef31a083ca0000000031980005000003c8000$
InnoDB: End of page dump
18 130212 9:37:51 InnoDB: Page checksum 1501194131, prior-to-4.0.14-form checksum 441953139
InnoDB: stored checksum 92546530, prior-to-4.0.14-form stored checksum 1240647222
InnoDB: Page lsn 509 671577750, low 4 bytes of lsn at page end 441447404
InnoDB: Page number (if stored to page already) 4424818,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 1096815
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 4424818.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
130212 9:37:51 InnoDB: Assertion failure in thread 140114781574912 in file buf0buf.c line 3603
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:37:51 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346681 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f72aa2435b9]
/usr/sbin/mysqld(handle_fatal_signal+0x3d8)[0x7f72aa12c548]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f72a8c8dcb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f72a82f6425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f72a82f9b8b]
/usr/sbin/mysqld(+0x605429)[0x7f72aa32d429]
/usr/sbin/mysqld(+0x631b69)[0x7f72aa359b69]
/usr/sbin/mysqld(+0x5c20a8)[0x7f72aa2ea0a8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f72a8c85e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f72a83b3cbd]
I cannot find anything running for mysql including any sockets.
I am taking over for a previous SysAdmin and am fairly new to linux and MySql. We've got to get this system back online soon. Please help.
Have you tried adding the following line to your /etc/mysql/my.cnf and then restarting the server?
[mysqld]
innodb_force_recovery = 4
To add to #elico3000 you will now need to dump your corrupt table(s) and Data to repair the innodb fs. There are a number of ways to do this. You can read through the logs to determine the point of failure and possible tablenames, then dump and recreate those specific tables. Or you can dump the entire MySQL DB and all schemas using a single command, but that will take some time depending on how big your DB. Either way once you have addressed the corrupt table(s) you can set the innodb_force option to 0 and restart mysqld_safe.
Here is a good tutorial on recovery options for both MyISAM and InnoDB MySQL instances and covers a few options. It is far easier to point you here, than regurgitate the commands and concepts again in this answer.
Good luck and come back to ask more pointed questions once you have tried one of the options. There are probably more tutorials out there, but I have used this in Development to rebuild my Dev DB and it has plenty of information.
Look here
I have a Centos and discovered my MySQL wouldn't start after it suddenly failed. I have tried all possible research in google to no avail. Here is the log details
root#27583 [~]# tail -200 /var/log/mysql.log
18:53:49 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=134217728
read_buffer_size=2097152
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2184911 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7dad25]
/usr/sbin/mysqld(handle_fatal_signal+0x3e1)[0x693c61]
/lib64/libpthread.so.0[0x7ffa00611ca0]
/lib64/libc.so.6(gsignal+0x35)[0x7ff9ff8112c5]
/lib64/libc.so.6(abort+0x110)[0x7ff9ff812d70]
/usr/sbin/mysqld[0x8deb95]
/usr/sbin/mysqld[0x8b66c9]
/usr/sbin/mysqld[0x8b6c07]
/usr/sbin/mysqld[0x8a87e7]
/usr/sbin/mysqld[0x87eae2]
/usr/sbin/mysqld[0x874ef9]
/usr/sbin/mysqld[0x875514]
/usr/sbin/mysqld[0x876ab1]
/usr/sbin/mysqld[0x8643ba]
/usr/sbin/mysqld[0x82d2a9]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x46)[0x695c66]
/usr/sbin/mysqld[0x587e7c]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xcf6)[0x58aac6]
/usr/sbin/mysqld[0x5051e3]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x8ad)[0x50829d]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x7ff9ff7fe9c4]
/usr/sbin/mysqld[0x4ff119]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
130209 19:53:49 mysqld_safe mysqld from pid file /var/lib/mysql/deepserver.netaviva.net.pid ended
130209 19:58:50 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130209 19:58:50 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
130209 19:58:50 [Note] Plugin 'FEDERATED' is disabled.
130209 19:58:50 InnoDB: The InnoDB memory heap is disabled
130209 19:58:50 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130209 19:58:50 InnoDB: Compressed tables use zlib 1.2.3
130209 19:58:50 InnoDB: Using Linux native AIO
130209 19:58:50 InnoDB: Initializing buffer pool, size = 128.0M
130209 19:58:50 InnoDB: Completed initialization of buffer pool
130209 19:58:50 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130209 19:58:50 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130209 19:58:50 InnoDB: Error: page 7 log sequence number 7922893671086
InnoDB: is in the future! Current system log sequence number 6086292560908.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: for more information.
InnoDB: Error: trying to access page number 939544476 in space 0,
InnoDB: space name ./ibdata1,
InnoDB: which is outside the tablespace bounds.
InnoDB: Byte offset 0, len 16384, i/o type 10.
InnoDB: If you get this error at mysqld startup, please check that
InnoDB: your my.cnf matches the ibdata files that you have in the
InnoDB: MySQL server.
130209 19:58:50 InnoDB: Assertion failure in thread 139796076951264 in file fil0fil.c line 4436
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
Please i would greatly appreciate any help to get this working again.