I recently experienced a DDoS attack. It overwhelmed and crashed my server. Upon restart all of the innodb tables were corrupted in every database on the server.
I have since rebuilt the databases and all of the tables. I had to recreate innodb_table_stats, innodb_index_stats.
Everything now seems to be running fine, the website is up but I have one persistent error that keeps coming up. The general log is filling with these quite rapidly.
2014-07-13 15:28:37 7fd70b374700 InnoDB: Error: page 193 log sequence number 526819726
InnoDB: is in the future! Current system log sequence number 156433332.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
I tried Changing the Number and Size of mySQL logs. It all went fine with no errors. But I am still seeing these errors mount up in the new log.
What else can I try to sync up these log sequence numbers? I am pretty new to this backend database work. Being forced to learn since my hosts tech support sucks.
I am currently on: CENTOS 6.5 x86_64 virtuozzo – vps- WHM 11.44.0 (build 22)
mySQL: 5.6.17
"page ... log sequence number" is when the page was last modified. The LSN in the header is larger than one in the redo log.
The easiest way to fix it is to rebuild the table with noop ALTER TABLE.
ALTER TABLE mytable ENGINE InnoDB;
It will rebuild PRIMARY index as well as its secondary indexes. After that the error should go away.
The ALTER is going to block the table, so if it's large and if the site is in production the best option is to rebuild it with pt-online-schema-change. It will do the same, but won't block the table but for brief moment.
pt-online-schema-change --alter "ENGINE=InnoDB" D=sakila,t=actor
Related
i have Windows 2016 Standard OS with installed MySQL 5.5.33.0.
This morning I found out that the mysql server is not running and cannot be turned on. When i try to turn it on, it crashes immediately, and there is only exception information 0xc0000005 in Event Viewer. I can't even run mysqlcheck because it doesn't work when mysqld isn't running.
I will be very grateful for any advice.
I was able to enable logging and set paths in MySQL settings. There are quite a few entries in the log with this text (numbers change):
210107 10:28:21 InnoDB: Error: page 33382 log sequence number 114951961648
InnoDB: is in the future! Current system log sequence number 111386529682.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files.
See InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: for more information.
210107 10:32:33 InnoDB: Error: Insert buffer insert fails; page free 43, dtuple size 50
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
0: len 40; hex 436f6d70757465722050726573732020202020202020202020202020202020202020202020202020;
asc Computer Press ;;
1: len 4; hex 800070da; asc p ;;
InnoDB: The table where this index record belongs
InnoDB: is now probably corrupt. Please run CHECK TABLE on
InnoDB: that table.
InnoDB: space 0, page 109621, zip_size 0, bitmap bits 1
Progress update:
I managed to start MySQL with innodb_force_recovery = 3 and it even runs for a while. If I run mysqlcheck.exe --all-databases --auto-repair, it reports corrupt in some tables of this type:
Warning: InnoDB: Index 'ix_linear_pole' contains 1683314 entries, should be 1683294.
However, the check does not run to the end and for one specific table the whole mysql server crashes again.
UPDATE: I found out that the error is in the loaded command in the InnoDB buffer, but I don't know how to delete it. Each time I turn on the server, a command in the buffer crash it.
After several hours of research, I found that MySQL was failing to load InnoDB Buffer. The easiest way was to add a command to my.ini
innodb_force_recovery = 4
However, the database is started only for SHOW data. Using mysqldump, I backed up the data without indexes. Then I deleted the database from the server, I also deleted the logs and database data from the system (C: \ ProgramData \ ... \ MySQL \ data \ ibdata and iblog0 and iblog1). Then I stopped MySQL and removed innodb_force_recovery. The database then ran in but without data.
I uploaded the original data back using normal recovery and created indexes.
From that moment on, everything works without problems.
It seems that startup is failing due to corruption in a database that we have already dropped.
MySQL is not starting, and the log shows the following error (a few hundred):
170905 5:46:05 InnoDB: Error: page 1 log sequence number 87166865838
and the DB could be corrupt and to see: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
As suggested, we started MySQL with innodb_force_recovery = 2 and innodb_purge_threads = 0, backed up the databases, and then dropped the database with the corrupted table(s).
MySQL still does not start. The error log ASCII/HEX dump shows segments of data that was in the deleted database.
Why are we still having problems with the deleted database? It feels like a process is still hanging, or something might not have deleted properly.
Any ideas? Thanks
Using MySQL 5.5.3 on Ubuntu 14.04
I have a C# application performing some database operations on a MySQL 5.7 server. Once the complete system hunp up and I had to hard-reset it. When it comes to a specific table read/write operation now the database server crashes. The windows log shows
InnoDB: Trying to access page number 286720 in space 29,
space name myInstance/myTable, which is outside the tablespace bounds.
Byte offset 0, len 16384, i/o type read.
I tried to use mysqlcheck --repair but it fails because note : The storage engine for the table doesn't support repair.
I've read some advices that say I should start MySQL in a recovery mode, so I added
[mysqld]
innodb_force_recovery=4
to the my.ini config file whereupon I should be able to use mysqldump to export the affected database table. But unfortunatelly I am not.
mysqldump: Error 2013: Lost connection to MySQL server
during query when dumping table `myTable` at row: 1246
Edit:
I checked the error log again and found lots of entries saying
[ERROR] C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqld.exe:
The table 'myTable' is full
I'm running the server on a Windows 32bit OS with an NTFS formatted partition. The myTable.ibd file size is around 4.5 GB, checking C.10.3 Limits on Table Size states a file size limit of "2TB (possibly larger)" for Win32 w/ NTFS.
While checking the reasons for my error the only possible cause I found was a full InnoDB tablespace. The solution might be "Changing the Number or Size of InnoDB Redo Log Files" although the coherence is a bit vague to me. Nevertheless I increased the size of the Redo Log Files from 48M to 100M. But nothing changed.
If I perform a SQL select * from myTable order by Id desc the server crashes instantly. Error log entry is exactly the same as above.
I checked the chapter 15.7.1 Resizing the InnoDB System Tablespace as well and found out that innodb_data_file_path is not explicitly specified.
Any ideas what I can do now? Thanks a lot!
InnoDB cannot repair corruption in tablespaces. This was never implemented and mysqlcheck won't help in any way.
The corruption is in space id 29 which is table myInstance.myTable. To repair it you need to dump all records from this table with innodb_force_recovery. Try all values from 4 to 6 until MySQL doesn't crash. Then drop the table and reload the dump.
If MySQL crashes even with innodb_force_recovery=6 then restore the table from backup.
If you don't have backup - use script http://bazaar.launchpad.net/~percona-dev/percona-data-recovery-tool-for-innodb/trunk/view/head:/fetch_data.sh . It will fetch as many records as it can.
One possible cause could be that the ib_logfile* files are corrupted.
To fix this, remove these files using rm ib_logfile*.
Where are these files?
These files are in mysql datadir. The location of datadir depends on the OS. Check my.cnf, in osx that should be in /usr/local/etc.
I am working on an embedded Linux system running MySQL 5.1. In rare cases QA reports systems not starting properly because of mysqld not starting. If this happens the MySQL log files looks similar to this excerpt:
150716 14:29:42 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150716 14:29:42 InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 133478.
InnoDB: Doing recovery: scanned up to log sequence number 0 133478
150716 14:29:42 InnoDB: Started; log sequence number 0 133478
/usr/libexec/mysqld: Unknown error 130
150716 14:29:42 [ERROR] Can't open the mysql.plugin table. Please run the mysql_upgrade script to create it.
150716 14:29:42 [ERROR] Fatal error: Can't open and lock privilege tables: Incorrect file format 'host'
This is probably due to the fact that this is an embedded device without a power switch and it is powered off by unplugging the network cable and thus killing the PoE supply. In this case of course mysqld will be terminated quite abnormaly.
my.cnf contains nothing fancy besides a size limitation to 18 MB.
Question
Is it possible that InnoDB tables get corrupted and recovery is NOT possible if there is NO bug involved (either in MySQL itself or e.g. a faulty fsync() implementation)? Are there situtations that can cause a corruption (that the DB cannot be recovered from) even if all software components are working correctly? Can such a DB be safely used in an environment where power failures occur "in normal operation"?
What I am ultimately asking is:
Is there a point searching for a fix to this problem or is there no fix to this problem whatsoever?
Is it possible that InnoDB tables get corrupted if there is NO bug involved (either in MySQL itself or e.g. a faulty fsync() implementation)?
Yes, eg: hardware failure
Are there situtations that can cause a corruption even if all software components are working correctly?
Yes, eg: hardware failure
Can such a DB be safely used in an environment where power failures occur "in normal operation"?
Yes, only if your hardware works "normally"
What I am ultimately asking is: Is there a point searching for a fix to this problem or is there no fix to this problem whatsoever?
Usually it's very difficult to fix the database if you meet a corruption. Do backup.
This article may help you:
https://dev.mysql.com/doc/refman/5.6/en/innodb-init-startup-configuration.html
Caution
InnoDB is a transaction-safe (ACID compliant) storage engine for MySQL that has commit, rollback, and crash-recovery capabilities to protect user data. However, it cannot do so if the underlying operating system or hardware does not work as advertised. Many operating systems or disk subsystems may delay or reorder write operations to improve performance. On some operating systems, the very fsync() system call that should wait until all unwritten data for a file has been flushed might actually return before the data has been flushed to stable storage. Because of this, an operating system crash or a power outage may destroy recently committed data, or in the worst case, even corrupt the database because of write operations having been reordered. If data integrity is important to you, perform some “pull-the-plug” tests before using anything in production. On OS X 10.3 and up, InnoDB uses a special fcntl() file flush method. Under Linux, it is advisable to disable the write-back cache.
On ATA/SATA disk drives, a command such hdparm -W0 /dev/hda may work to disable the write-back cache. Beware that some drives or disk controllers may be unable to disable the write-back cache.
With regard to InnoDB recovery capabilities that protect user data, InnoDB uses a file flush technique involving a structure called the doublewrite buffer, which is enabled by default (innodb_doublewrite=ON). The doublewrite buffer adds safety to recovery following a crash or power outage, and improves performance on most varieties of Unix by reducing the need for fsync() operations. It is recommended that the innodb_doublewrite option remains enabled if you are concerned with data integrity or possible failures. For additional information about the doublewrite buffer, see Section 14.9, “InnoDB Disk I/O and File Space Management”.
Caution
If reliability is a consideration for your data, do not configure InnoDB to use data files or log files on NFS volumes. Potential problems vary according to OS and version of NFS, and include such issues as lack of protection from conflicting writes, and limitations on maximum file sizes.
Did you upgrade the MySQL version, but fail to run mysql_upgrade ? That's what the error is saying.
I finally found the root cause. The problem does not actually occur with the InnoDB tables, but with the system tables.
In MySQL 5.1 system tables are stored using the MyISAM engine. This makes these tables very fragile on power loss.
For all system tables the content of the MYI (index) and MYD (data) files were lost.
Missing this data - of course - the rest of the databases had a problem...
The important hint for me was
mysql.plugin table
Finally looked into the directory containing the system tables and saw they were using the MyISAM storage engine. Then the consequences are quite obvious.
(Only) Solution:
Go to a newer version (I used MariaDB in my case).
You cannot use InnoDB as storage engine for the system tables on MySQL 5.1.
I fairly new to MySQL database. I am repeatedly seeing an error from MySQL saying the table is marked as crashed and should be repaired. However I am able to repair the crashed table by using the command myisamchk. By the way, I am using MYISAM database engine in MySQL.
I just wanted to know under what circumstances would a DB table crash and how I can prevent it from happening again?
I am connecting to MySQL(5.0) database from Tcl (8.5) script using mysqltcl library (3.0).
MyISAM tables are very easy to crash. There is header info in each table that keeps track of how many open file handles a MyISAM table has.
If mysqld crashes, any MyISAM table that had open file handles to it never had the opportunity to decrement the file handle count upon each file handle closing. Thus, if a new file handle opens a MyISAM table (.MYD file) and mysqld discovers a mismatch between the number of file handles a MyISAM table believes is open and the the number of file handles the MyISAM table actually has open, the table is declared crashed.
There are four(4) methods for handling this:
METHOD #1 : Setup automatic MyISAM repair
See my post https://dba.stackexchange.com/a/15079/877 on how to set this up upon a MySQL restart (Mar 15, 2012)
METHOD #2 : Use InnoDB instead of MyISAM
InnoDB has crash recovery built into the Storage Engine's initialization. MyISAM does not
METHOD #3 : Use Aria instead of MyISAM
Aria is MariaDB's drop-in replacement for MyISAM. It features crash recovery mechanisms for individual tables.
METHOD #4 : Don't kill -9 on mysqld
If mysqld crashes, deliberately or involuntarily, header info for all open MyISAM tables will get them into a crashed state. Avoid having to manually kill mysqld.
I noticed that when I attempt to do a LVM snapshot of my database volume, after running FLUSH TABLES WITH READ LOCK, then rsync that snapshot to a new system, the tables are marked as crashed and have to be repaired.
I suspect this has to do with there being a file handle on the original machine with the table open, and then I'm syncing the that status to the new machine and it sees a mismatch in the file handles and decides it needs to repair.
This repair is problematic because it takes hours (it is a giant table). So the only reliable way to actually get a snapshot that isn't crashed is to shutdown the database before taking the snapshot, but then I cannot get the SHOW MASTER STATUS to setup replication.