recover mysql database from ibdata1 - mysql

I have a client who appears to have lost all of their mysql databases from their local machine. They are on a Mac, which I am somewhat unfamiliar with and I am on Ubuntu. There were no .MYD or .MYI files in the database folder, only .frm ones. I had them zip up the mysql and sight folders (with sight being the database we need), and the ibdata1, ib_logfile0, and ib_logfile1 files. I created a second folder for mysql, /var/lib/mysql2, and moved the files and folders into there. I chowned the new folder and files to mysql:mysql, edited /etc/mysql/my.cnf to point to the new folder, edited /etc/apparmor.d/usr.sbin.mysqld, and then restarted apparmor and mysql. However, I am getting the following errors in the mysql error log:
130308 17:38:16 [Note] Plugin 'FEDERATED' is disabled.
130308 17:38:16 InnoDB: Initializing buffer pool, size = 8.0M
130308 17:38:16 InnoDB: Completed initialization of buffer pool
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130308 17:38:16 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130308 17:38:16 InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 0:589824, should be 0:7!
130308 17:38:16 InnoDB: Error: page 589824 log sequence number 786432 0
InnoDB: is in the future! Current system log sequence number 0 63932940.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: for more information.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 7.
InnoDB: You may have to recover from a backup.
130308 17:38:16 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 0008000000090000000a0000000b0000000c00000000000000000000000202720000 (snipped because this goes on for a while)
Tg 9 <o q E i F / D ;InnoDB: End of page dump
130308 17:38:16 InnoDB: Page checksum 4146777650, prior-to-4.0.14-form checksum 1800374066
InnoDB: stored checksum 524288, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 786432 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 589824,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 7.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
I have tried adding innodb_force_recovery = 4 to the my.cnf file, which generates a whole slew of different errors:
130308 17:48:30 [Note] Plugin 'FEDERATED' is disabled.
130308 17:48:30 InnoDB: Initializing buffer pool, size = 8.0M
130308 17:48:30 InnoDB: Completed initialization of buffer pool
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130308 17:48:30 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130308 17:48:30 InnoDB: Error: space id and page n:o stored in the page
InnoDB: read in are 0:589824, should be 0:7!
130308 17:48:30 InnoDB: Error: page 589824 log sequence number 786432 0
InnoDB: is in the future! Current system log sequence number 0 63932940.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: for more information.
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 7.
InnoDB: You may have to recover from a backup.
and a whole bunch more, which I can supply if it helps. Any suggestions on what to try from here would be greatly appreciated, thanks.
Edit: I tried following the steps here, but was having issues getting mysql to run using the command line sequence he used:
http://blog.shiraj.com/2012/10/extract-data-from-mysql-ibdata1-data-file/

The following worked for my:
Set innodb_force_recovery = 1 in your my.cnf
Try and get your mysqld to restart. If not, repeat step #1 and increment
innodb_force_recovery by each until successful. Use the guide to help you understand what's happening each time you increment it: http://dev.mysql.com/doc/refman/5.0/en/forcing-innodb-recovery.html
Once mysqld is running, try and dump all your databases
mysqldump -u root -p --all-databases > /tmp/mysqldump-all.sql
If that isn't successful, you'll have to try it first on the database level
mysqldump -u root -p --databases db_name > mysqldump-db_name.sql
If that isn't successful, you'll have to try it on the table level
SELECT * FROM table_name INTO OUTFILE '/tmp/table_name.sql'
Once one of those is successful and either all your db's or all your tables are exported, stop the mysqld
Move your ib_logfile* > ib_logfile*.bak. These are typically in your mysql data directory.
If in the first step you incremented your innodb_force_recovery => 4, you need to set it below 4. As of 5.6.15, an innodb_force_recovery setting of 4 or greater places InnoDB in read-only mode.
Start the mysqld server
Import your exported databases or tables
mysql -u root -p < /tmp/mysqldump-all.sql
Incremented your innodb_force_recovery => 1
Restart the mysqld server

This is worked for me:
innodb_force_recovery => 2
Login into MySQL server using HeidiSQL and dump all database
Reinstall MySQL server and import all the database from dump file.

Related

How to fix corupted MySQL installation

Due to some issues with the server, a number of InnoDB database folders with ibd, frm and TRN files were deleted (only affects databases not really needed anymore). Currently MySQL service starts only with the following setting:
innodb_force_recovery = 5
Then it crashes every few seconds, here's an extract from the log file
2022-10-11 17:38:54 27975 [Note] InnoDB: 1.2.10 started; log sequence number 313646609430
2022-10-11 17:38:54 27975 [Note] InnoDB: !!! innodb_force_recovery is set to 5 !!!
2022-10-11 17:38:54 27975 [Note] Server hostname (bind-address): '*'; port: 3306
2022-10-11 17:38:54 27975 [Note] IPv6 is available.
2022-10-11 17:38:54 27975 [Note] - '::' resolves to '::';
2022-10-11 17:38:54 27975 [Note] Server socket created on IP: '::'.
2022-10-11 17:38:54 27975 [Note] Event Scheduler: Loaded 0 events
2022-10-11 17:38:54 27975 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.10' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2022-10-11 17:38:55 7f1d2c06a700 InnoDB: Error: page 4087 log sequence number 315137164614
InnoDB: is in the future! Current system log sequence number 313646609430.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2022-10-11 17:38:55 27975 [ERROR] InnoDB: Failed to find tablespace for table '"wp_website"."wp_options"' in the cache. Attempting to load the tablespace with space id 34249.
2022-10-11 17:38:55 27975 [Warning] InnoDB: Allocated tablespace 34249, old maximum was 0
2022-10-11 17:38:55 7f1d2c06a700 InnoDB: Error: page 252899 log sequence number 471118864984
InnoDB: is in the future! Current system log sequence number 313646609430.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2022-10-11 17:38:55 7f1d2c06a700 InnoDB: Error: page 85552 log sequence number 471118895771
InnoDB: is in the future! Current system log sequence number 313646609430.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2022-10-11 17:38:55 7f1d2c06a700 InnoDB: Error: page 147803 log sequence number 471118895778
InnoDB: is in the future! Current system log sequence number 313646609430.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2022-10-11 17:38:55 7f1d2c06a700 InnoDB: Error: page 4 log sequence number 470722597051
InnoDB: is in the future! Current system log sequence number 313646609440.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2022-10-11 17:38:55 7f1d2c06a700 InnoDB: Error: page 13 log sequence number 315130852444
InnoDB: is in the future! Current system log sequence number 313646609440.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2022-10-11 17:38:55 7f1d2c06a700 InnoDB: Error: page 3 log sequence number 315130852367
InnoDB: is in the future! Current system log sequence number 313646609440.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
2022-10-11 17:38:55 7f1d2c06a700 InnoDB: Error: page 22 log sequence number 315133587106
InnoDB: is in the future! Current system log sequence number 313646609440.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
InnoDB: A new raw disk partition was initialized or
InnoDB: innodb_force_recovery is on: we do not allow
InnoDB: database modifications by the user. Shut down
InnoDB: mysqld and edit my.cnf so that newraw is replaced
InnoDB: with raw, and innodb_force_... is removed.
14:38:55 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68216 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x1d46900
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f1d2c069e18 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8bfd45]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x65b074]
/lib64/libpthread.so.0[0x34c700f7e0]
/usr/sbin/mysqld[0x9a45b2]
/usr/sbin/mysqld[0x97e382]
/usr/sbin/mysqld[0x943807]
/usr/sbin/mysqld[0x9a236d]
/usr/sbin/mysqld[0x9a2bfd]
/usr/sbin/mysqld[0x8e1729]
/usr/sbin/mysqld(_Z15ha_rollback_lowP3THDb+0x87)[0x5a27a7]
/usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0x4c)[0x5a256c]
/usr/sbin/mysqld(_Z19trans_rollback_stmtP3THD+0x29)[0x764a19]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x23d)[0x6d2fdd]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x32f)[0x6d7a5f]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xe28)[0x6d8978]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xcf)[0x6a682f]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6a6957]
/usr/sbin/mysqld(pfs_spawn_thread+0x139)[0xadc7d9]
/lib64/libpthread.so.0[0x34c7007aa1]
/lib64/libc.so.6(clone+0x6d)[0x34c6ce8c4d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f1d04004f30): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
221011 17:38:55 mysqld_safe Number of processes running now: 0
221011 17:38:55 mysqld_safe mysqld restarted
How can I fix the problem? Or at least create the dumps of a few databases I need?

MAMP Pro can't start MySQL

I had to force a shutdown when my computer froze, and MAMP has been unable to start MySql since then. I have tried deleting the log files from MAMP/DB/mysql and restarting MAMP, but it hasn't worked. I also tried entering sudo killall -9 mysqld in the terminal, to no avail.
Here is the error log from MAMP:
2017-05-12 16:05:49 4691 [Note] InnoDB: The log sequence numbers 730255276 and 730255276 in ibdata files do not match the log sequence number 730624549 in the ib_logfiles!2017-05-12 16:05:49 4691 [Note] InnoDB: Database was not shutdown normally!
2017-05-12 16:05:49 4691 [Note] InnoDB: Starting crash recovery.
2017-05-12 16:05:49 4691 [Note] InnoDB: Reading tablespace information from the .ibd files...
2017-05-12 16:05:50 4691 [ERROR] InnoDB: Attempted to open a previously opened tablespace. Previous tablespace mysql/slave_worker_info uses space ID: 5 at filepath: ./mysql/slave_worker_info.ibd. Cannot open tablespace wordpress#0020test#0020site/wp_term_taxonomy which uses space ID: 5 at filepath: ./wordpress#0020test#0020site/wp_term_taxonomy.ibd
2017-05-12 16:05:50 7fff7303b000 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: Error: could not open single-table tablespace file ./wordpress#0020test#0020site/wp_term_taxonomy.ibd
InnoDB: We do not continue the crash recovery, because the table may become
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
InnoDB: To fix the problem and start mysqld:
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
InnoDB: open the file, you should modify the permissions.
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
InnoDB: crash recovery and ignore that table.
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
InnoDB: and force InnoDB to continue crash recovery here.
170512 16:05:50 mysqld_safe mysqld from pid file /Applications/MAMP/tmp/mysql/mysql.pid ended
Any ideas?
I've just had the same problem with MAMP.
To solve the problem, i was forced to delete the file that causes the error located in "Applications/MAMP/db/mysql56/bib" and then recreate the TABLE/BASE afterwards.
I hope that would be helpful.

Error Starting MySQL after Crash

I was merrily installing wordpress with MAMP pro when my Mac crashed. Now I cannot start MySql. Most stuff I read online tells me to throw some terminal commands at it, trying to kill the process.
killall -9 mysqld
But terminal report "No matching processes belonging to you were found"
So I have had a look in the log and can see this from the time of the first restart attempt after the crash
150506 21:11:33 mysqld_safe Starting mysqld daemon with databases from /Library/Application Support/appsolute/MAMP PRO/db/mysql
150506 21:11:33 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150506 21:11:33 [Warning] Setting lower_case_table_names=2 because file system for /Library/Application Support/appsolute/MAMP PRO/db/mysql/ is case insensitive
150506 21:11:33 [Note] Plugin 'FEDERATED' is disabled.
150506 21:11:33 InnoDB: The InnoDB memory heap is disabled
150506 21:11:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150506 21:11:33 InnoDB: Compressed tables use zlib 1.2.3
150506 21:11:33 InnoDB: Initializing buffer pool, size = 128.0M
150506 21:11:33 InnoDB: Completed initialization of buffer pool
InnoDB: The first specified data file ./ibdata1 did not exist:
InnoDB: a new database to be created!
150506 21:11:33 InnoDB: Setting file ./ibdata1 size to 10 MB
InnoDB: Database physically writes the file full: wait...
150506 21:11:33 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
150506 21:11:33 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
150506 21:11:34 InnoDB: Waiting for the background threads to start
150506 21:11:35 InnoDB: 5.5.42 started; log sequence number 0
150506 21:11:35 [Note] Event Scheduler: Loaded 0 events
150506 21:11:35 [Note] /Applications/MAMP/Library/bin/mysqld: ready for connections.
Version: '5.5.42' socket: '/Applications/MAMP/tmp/mysql/mysql.sock' port: 0 Source distribution
150506 21:16:44 [Note] /Applications/MAMP/Library/bin/mysqld: Normal shutdown
So Table Space size error.. I am very much a front end guy and have no idea where to begin with this...??
Help please, I really can't afford to loose the last week of work.
Thanks you
As you pointed out InnoDB tablespace is corrupt, so InnoDB fails to start.
InnoDB can tolerate some errors if you start it with option innodb_force_recovery. Try values from 1 to 6 until MySQL starts.
If you're lucky and MySQL starts with innodb_force_recovery then dump all tables into an sql file with mysqldump and re-create MySQL database from scratch. I.e. move existing files to some safe place, start mysql_install_db, change file permissions and start MySQL. Then load the database back.
If you're less lucky and MySQL doesn't start with innodb_force_recovery=6 then check how to Recover Corrupt MySQL Database .
And to avoid this experience in future use XtraBackup for Mac OS to take backups.
Please see her:
MAMP PRO crashes; MySQL will not start on reboot
Thanks for the heads up with Xtrabackup for MacOS, one to keep an eye on that.

mysql database and doublewrite buffer corrupt. How to drop table/database?

Is it possible to drop a table/database manually if it is corrupt? How can I clear the ibdata doublewrite buffer?
I just restored my server. However mysql is not coming up. The error message recomments to set innodb_force_recovery=6. Which allowed to start mysql. So I identified the defect table. However no matter what I try check / repair / drop the corrupt table / database which always ends up in mysql stopping and throwing the error:
ERROR 2013 (HY000): Lost connection to MySQL server during query
Here is the mysql error log from starting mysql:
140702 13:40:36 [Note] Plugin 'FEDERATED' is disabled.
140702 13:40:36 [Warning] option 'innodb-buffer-pool-size': signed value 2097152 adjusted to 5242880
140702 13:40:36 [Warning] option 'innodb-additional-mem-pool-size': signed value 512000 adjusted to 524288
140702 13:40:36 InnoDB: The InnoDB memory heap is disabled
140702 13:40:36 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140702 13:40:36 InnoDB: Compressed tables use zlib 1.2.3.4
140702 13:40:36 InnoDB: Initializing buffer pool, size = 5.0M
140702 13:40:36 InnoDB: Completed initialization of buffer pool
140702 13:40:36 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 5 MB
InnoDB: Database physically writes the file full: wait...
140702 13:40:37 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 5 MB
InnoDB: Database physically writes the file full: wait...
140702 13:40:37 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
140702 13:40:37 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Warning: database page corruption or a failed
InnoDB: file read of space 0 page 79918.
InnoDB: Trying to recover it from the doublewrite buffer.
InnoDB: Dump of the page:
140702 13:40:37 InnoDB: Page dump in ascii and hex (16384 bytes):
Hexdata
InnoDB: End of page dump
140702 13:40:37 InnoDB: Page checksum 3463763561, prior-to-4.0.14-form checksum 3074131693
InnoDB: stored checksum 2986557383, prior-to-4.0.14-form stored checksum 3074131693
InnoDB: Page lsn 3 3511673317, low 4 bytes of lsn at page end 3511673317
InnoDB: Page number (if stored to page already) 79918,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 5918
InnoDB: Dump of corresponding page in doublewrite buffer:
140702 13:40:37 InnoDB: Page dump in ascii and hex (16384 bytes):
Hexdata
...
InnoDB: End of page dump
140702 13:40:37 InnoDB: Page checksum 1693035738, prior-to-4.0.14-form checksum 533526999
InnoDB: stored checksum 1060638083, prior-to-4.0.14-form stored checksum 533526999
InnoDB: Page lsn 3 3511586897, low 4 bytes of lsn at page end 3511586897
InnoDB: Page number (if stored to page already) 79918,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 5918
InnoDB: Also the page in the doublewrite buffer is corrupt.
InnoDB: Cannot continue operation.
InnoDB: You can try to recover the database with the my.cnf
InnoDB: option:
InnoDB: innodb_force_recovery=6
140702 13:40:37 InnoDB: Assertion failure in thread 3064526592 in file trx0sys.c line 604
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
11:40:37 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346063 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x33)[0xb72a35c3]
/usr/sbin/mysqld(handle_fatal_signal+0x484)[0xb7150004]
[0xb6e0c500]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Is there any way to drop a database / table while mysql is offline?
Is it possible to clear the ibdata doublewrite buffer since this is blogging mysql from starting normally?

Cannot Start MySql

Our server has been running mysql just fine for over a year. I ran a set of sql script to build a rather large database and in the middle of those scripts, I started getting errors that I had lost connection. Nobody did anything else happened as far as we know. When I tried to log in to mysql, I got:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
When I try to restart mysql, I get:
# sudo service mysql restart
stop: Unknown instance:
start: Job failed to start
The error.log shows:
130212 9:37:51 [Note] Plugin 'FEDERATED' is disabled.
130212 9:37:51 InnoDB: The InnoDB memory heap is disabled
130212 9:37:51 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130212 9:37:51 InnoDB: Compressed tables use zlib 1.2.7
130212 9:37:51 InnoDB: Using Linux native AIO
130212 9:37:51 InnoDB: Initializing buffer pool, size = 10.0G
130212 9:37:51 InnoDB: Completed initialization of buffer pool
130212 9:37:51 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 2186809272046
130212 9:37:51 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2186814514688
InnoDB: Doing recovery: scanned up to log sequence number 2186816162838
InnoDB: 1 transaction(s) which must be rolled back or cleaned up
InnoDB: in total 27964 row operations to undo
InnoDB: Trx id counter is 18834200
130212 9:37:51 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 4424818.
InnoDB: You may have to recover from a backup.
130212 9:37:51 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 058425e20043847200356a3f003e3720000001fd2807769645bf00000000000000000000000000ef31a083ca0000000031980005000003c8000$
InnoDB: End of page dump
18 130212 9:37:51 InnoDB: Page checksum 1501194131, prior-to-4.0.14-form checksum 441953139
InnoDB: stored checksum 92546530, prior-to-4.0.14-form stored checksum 1240647222
InnoDB: Page lsn 509 671577750, low 4 bytes of lsn at page end 441447404
InnoDB: Page number (if stored to page already) 4424818,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 1096815
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 4424818.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
InnoDB: Ending processing because of a corrupt database page.
130212 9:37:51 InnoDB: Assertion failure in thread 140114781574912 in file buf0buf.c line 3603
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
16:37:51 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346681 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f72aa2435b9]
/usr/sbin/mysqld(handle_fatal_signal+0x3d8)[0x7f72aa12c548]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f72a8c8dcb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f72a82f6425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f72a82f9b8b]
/usr/sbin/mysqld(+0x605429)[0x7f72aa32d429]
/usr/sbin/mysqld(+0x631b69)[0x7f72aa359b69]
/usr/sbin/mysqld(+0x5c20a8)[0x7f72aa2ea0a8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f72a8c85e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f72a83b3cbd]
I cannot find anything running for mysql including any sockets.
I am taking over for a previous SysAdmin and am fairly new to linux and MySql. We've got to get this system back online soon. Please help.
Have you tried adding the following line to your /etc/mysql/my.cnf and then restarting the server?
[mysqld]
innodb_force_recovery = 4
To add to #elico3000 you will now need to dump your corrupt table(s) and Data to repair the innodb fs. There are a number of ways to do this. You can read through the logs to determine the point of failure and possible tablenames, then dump and recreate those specific tables. Or you can dump the entire MySQL DB and all schemas using a single command, but that will take some time depending on how big your DB. Either way once you have addressed the corrupt table(s) you can set the innodb_force option to 0 and restart mysqld_safe.
Here is a good tutorial on recovery options for both MyISAM and InnoDB MySQL instances and covers a few options. It is far easier to point you here, than regurgitate the commands and concepts again in this answer.
Good luck and come back to ask more pointed questions once you have tried one of the options. There are probably more tutorials out there, but I have used this in Development to rebuild my Dev DB and it has plenty of information.
Look here