We know in SQL server , the transaction log file contains both redo and undo log, however in the MYSQL transaction log ,it only contains the undo log , so how the MYSQL do the redo operation if it just writing just partial of the transaction data for a successfully transaction ,then the server down? As it is necessary to do the redo operation when reboot the server.
Eidt, Sorry, I typo, I mean why MYSQL doesn't have the redo log instead of the undo log.
This is not done on MySQL level, this is taken care of the various table types that support transactions (e.g. innodb).
Innodb being the most widely used transactional table type in MySQL, my answer focuses on this table type only. For other transactional table types, see their own documentation on more info on the redo logs.
MySQL documentation on innodb redo logs describe how innodb stores redo logs:
By default, the redo log is physically represented on disk as a set of
files, named ib_logfile0 and ib_logfile1. MySQL writes to the redo log
files in a circular fashion. Data in the redo log is encoded in terms
of records affected; this data is collectively referred to as redo.
The passage of data through the redo log is represented by an
ever-increasing LSN value.
Related
Long transactions generate several old versions of the rows in the undo log which are stored in ibdata file.Is there any possibility that undo log entries are also stored in the innodb buffer pool.
They cannot be stored "permanently" in the buffer_pool because it is lost in a power failure. Undo is needed then, too.
I am working on an embedded Linux system running MySQL 5.1. In rare cases QA reports systems not starting properly because of mysqld not starting. If this happens the MySQL log files looks similar to this excerpt:
150716 14:29:42 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
150716 14:29:42 InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 133478.
InnoDB: Doing recovery: scanned up to log sequence number 0 133478
150716 14:29:42 InnoDB: Started; log sequence number 0 133478
/usr/libexec/mysqld: Unknown error 130
150716 14:29:42 [ERROR] Can't open the mysql.plugin table. Please run the mysql_upgrade script to create it.
150716 14:29:42 [ERROR] Fatal error: Can't open and lock privilege tables: Incorrect file format 'host'
This is probably due to the fact that this is an embedded device without a power switch and it is powered off by unplugging the network cable and thus killing the PoE supply. In this case of course mysqld will be terminated quite abnormaly.
my.cnf contains nothing fancy besides a size limitation to 18 MB.
Question
Is it possible that InnoDB tables get corrupted and recovery is NOT possible if there is NO bug involved (either in MySQL itself or e.g. a faulty fsync() implementation)? Are there situtations that can cause a corruption (that the DB cannot be recovered from) even if all software components are working correctly? Can such a DB be safely used in an environment where power failures occur "in normal operation"?
What I am ultimately asking is:
Is there a point searching for a fix to this problem or is there no fix to this problem whatsoever?
Is it possible that InnoDB tables get corrupted if there is NO bug involved (either in MySQL itself or e.g. a faulty fsync() implementation)?
Yes, eg: hardware failure
Are there situtations that can cause a corruption even if all software components are working correctly?
Yes, eg: hardware failure
Can such a DB be safely used in an environment where power failures occur "in normal operation"?
Yes, only if your hardware works "normally"
What I am ultimately asking is: Is there a point searching for a fix to this problem or is there no fix to this problem whatsoever?
Usually it's very difficult to fix the database if you meet a corruption. Do backup.
This article may help you:
https://dev.mysql.com/doc/refman/5.6/en/innodb-init-startup-configuration.html
Caution
InnoDB is a transaction-safe (ACID compliant) storage engine for MySQL that has commit, rollback, and crash-recovery capabilities to protect user data. However, it cannot do so if the underlying operating system or hardware does not work as advertised. Many operating systems or disk subsystems may delay or reorder write operations to improve performance. On some operating systems, the very fsync() system call that should wait until all unwritten data for a file has been flushed might actually return before the data has been flushed to stable storage. Because of this, an operating system crash or a power outage may destroy recently committed data, or in the worst case, even corrupt the database because of write operations having been reordered. If data integrity is important to you, perform some “pull-the-plug” tests before using anything in production. On OS X 10.3 and up, InnoDB uses a special fcntl() file flush method. Under Linux, it is advisable to disable the write-back cache.
On ATA/SATA disk drives, a command such hdparm -W0 /dev/hda may work to disable the write-back cache. Beware that some drives or disk controllers may be unable to disable the write-back cache.
With regard to InnoDB recovery capabilities that protect user data, InnoDB uses a file flush technique involving a structure called the doublewrite buffer, which is enabled by default (innodb_doublewrite=ON). The doublewrite buffer adds safety to recovery following a crash or power outage, and improves performance on most varieties of Unix by reducing the need for fsync() operations. It is recommended that the innodb_doublewrite option remains enabled if you are concerned with data integrity or possible failures. For additional information about the doublewrite buffer, see Section 14.9, “InnoDB Disk I/O and File Space Management”.
Caution
If reliability is a consideration for your data, do not configure InnoDB to use data files or log files on NFS volumes. Potential problems vary according to OS and version of NFS, and include such issues as lack of protection from conflicting writes, and limitations on maximum file sizes.
Did you upgrade the MySQL version, but fail to run mysql_upgrade ? That's what the error is saying.
I finally found the root cause. The problem does not actually occur with the InnoDB tables, but with the system tables.
In MySQL 5.1 system tables are stored using the MyISAM engine. This makes these tables very fragile on power loss.
For all system tables the content of the MYI (index) and MYD (data) files were lost.
Missing this data - of course - the rest of the databases had a problem...
The important hint for me was
mysql.plugin table
Finally looked into the directory containing the system tables and saw they were using the MyISAM storage engine. Then the consequences are quite obvious.
(Only) Solution:
Go to a newer version (I used MariaDB in my case).
You cannot use InnoDB as storage engine for the system tables on MySQL 5.1.
I recently experienced a DDoS attack. It overwhelmed and crashed my server. Upon restart all of the innodb tables were corrupted in every database on the server.
I have since rebuilt the databases and all of the tables. I had to recreate innodb_table_stats, innodb_index_stats.
Everything now seems to be running fine, the website is up but I have one persistent error that keeps coming up. The general log is filling with these quite rapidly.
2014-07-13 15:28:37 7fd70b374700 InnoDB: Error: page 193 log sequence number 526819726
InnoDB: is in the future! Current system log sequence number 156433332.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
I tried Changing the Number and Size of mySQL logs. It all went fine with no errors. But I am still seeing these errors mount up in the new log.
What else can I try to sync up these log sequence numbers? I am pretty new to this backend database work. Being forced to learn since my hosts tech support sucks.
I am currently on: CENTOS 6.5 x86_64 virtuozzo – vps- WHM 11.44.0 (build 22)
mySQL: 5.6.17
"page ... log sequence number" is when the page was last modified. The LSN in the header is larger than one in the redo log.
The easiest way to fix it is to rebuild the table with noop ALTER TABLE.
ALTER TABLE mytable ENGINE InnoDB;
It will rebuild PRIMARY index as well as its secondary indexes. After that the error should go away.
The ALTER is going to block the table, so if it's large and if the site is in production the best option is to rebuild it with pt-online-schema-change. It will do the same, but won't block the table but for brief moment.
pt-online-schema-change --alter "ENGINE=InnoDB" D=sakila,t=actor
I fairly new to MySQL database. I am repeatedly seeing an error from MySQL saying the table is marked as crashed and should be repaired. However I am able to repair the crashed table by using the command myisamchk. By the way, I am using MYISAM database engine in MySQL.
I just wanted to know under what circumstances would a DB table crash and how I can prevent it from happening again?
I am connecting to MySQL(5.0) database from Tcl (8.5) script using mysqltcl library (3.0).
MyISAM tables are very easy to crash. There is header info in each table that keeps track of how many open file handles a MyISAM table has.
If mysqld crashes, any MyISAM table that had open file handles to it never had the opportunity to decrement the file handle count upon each file handle closing. Thus, if a new file handle opens a MyISAM table (.MYD file) and mysqld discovers a mismatch between the number of file handles a MyISAM table believes is open and the the number of file handles the MyISAM table actually has open, the table is declared crashed.
There are four(4) methods for handling this:
METHOD #1 : Setup automatic MyISAM repair
See my post https://dba.stackexchange.com/a/15079/877 on how to set this up upon a MySQL restart (Mar 15, 2012)
METHOD #2 : Use InnoDB instead of MyISAM
InnoDB has crash recovery built into the Storage Engine's initialization. MyISAM does not
METHOD #3 : Use Aria instead of MyISAM
Aria is MariaDB's drop-in replacement for MyISAM. It features crash recovery mechanisms for individual tables.
METHOD #4 : Don't kill -9 on mysqld
If mysqld crashes, deliberately or involuntarily, header info for all open MyISAM tables will get them into a crashed state. Avoid having to manually kill mysqld.
I noticed that when I attempt to do a LVM snapshot of my database volume, after running FLUSH TABLES WITH READ LOCK, then rsync that snapshot to a new system, the tables are marked as crashed and have to be repaired.
I suspect this has to do with there being a file handle on the original machine with the table open, and then I'm syncing the that status to the new machine and it sees a mismatch in the file handles and decides it needs to repair.
This repair is problematic because it takes hours (it is a giant table). So the only reliable way to actually get a snapshot that isn't crashed is to shutdown the database before taking the snapshot, but then I cannot get the SHOW MASTER STATUS to setup replication.
When exactly an event on the master(update/insert/alter) is logged on the binary log ?
When it started on the master server, it will be written to the bin log
When it started and completed on the master, then will be logged
In specific, I would like to know the behaviour in case of an ALTER
If answer is 1, there won't be lag between master and slave in normal conditions only due to ALTERs started on the master.
Non-transactional statements are written to the binary log immediately. For MySQL 5+, this means any statement against a storage engine that doesn't support transactions are written immediately.
You shouldn't be executing statements against a non-transactional table inside a transaction though.
Transactional statements are written to the binary log when the transaction is committed. For MySQL 5+, this means any statement against a transactional table.
Currently, ALTER TABLE forces an implicit commit, so this will immediately be written to the binary log. According to MySQL documentation:
Binary logging is done immediately after a statement completes but before any locks are released or any commit is done. This ensures that the log is logged in execution order.