I have a C# application performing some database operations on a MySQL 5.7 server. Once the complete system hunp up and I had to hard-reset it. When it comes to a specific table read/write operation now the database server crashes. The windows log shows
InnoDB: Trying to access page number 286720 in space 29,
space name myInstance/myTable, which is outside the tablespace bounds.
Byte offset 0, len 16384, i/o type read.
I tried to use mysqlcheck --repair but it fails because note : The storage engine for the table doesn't support repair.
I've read some advices that say I should start MySQL in a recovery mode, so I added
[mysqld]
innodb_force_recovery=4
to the my.ini config file whereupon I should be able to use mysqldump to export the affected database table. But unfortunatelly I am not.
mysqldump: Error 2013: Lost connection to MySQL server
during query when dumping table `myTable` at row: 1246
Edit:
I checked the error log again and found lots of entries saying
[ERROR] C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqld.exe:
The table 'myTable' is full
I'm running the server on a Windows 32bit OS with an NTFS formatted partition. The myTable.ibd file size is around 4.5 GB, checking C.10.3 Limits on Table Size states a file size limit of "2TB (possibly larger)" for Win32 w/ NTFS.
While checking the reasons for my error the only possible cause I found was a full InnoDB tablespace. The solution might be "Changing the Number or Size of InnoDB Redo Log Files" although the coherence is a bit vague to me. Nevertheless I increased the size of the Redo Log Files from 48M to 100M. But nothing changed.
If I perform a SQL select * from myTable order by Id desc the server crashes instantly. Error log entry is exactly the same as above.
I checked the chapter 15.7.1 Resizing the InnoDB System Tablespace as well and found out that innodb_data_file_path is not explicitly specified.
Any ideas what I can do now? Thanks a lot!
InnoDB cannot repair corruption in tablespaces. This was never implemented and mysqlcheck won't help in any way.
The corruption is in space id 29 which is table myInstance.myTable. To repair it you need to dump all records from this table with innodb_force_recovery. Try all values from 4 to 6 until MySQL doesn't crash. Then drop the table and reload the dump.
If MySQL crashes even with innodb_force_recovery=6 then restore the table from backup.
If you don't have backup - use script http://bazaar.launchpad.net/~percona-dev/percona-data-recovery-tool-for-innodb/trunk/view/head:/fetch_data.sh . It will fetch as many records as it can.
One possible cause could be that the ib_logfile* files are corrupted.
To fix this, remove these files using rm ib_logfile*.
Where are these files?
These files are in mysql datadir. The location of datadir depends on the OS. Check my.cnf, in osx that should be in /usr/local/etc.
Related
It seems that startup is failing due to corruption in a database that we have already dropped.
MySQL is not starting, and the log shows the following error (a few hundred):
170905 5:46:05 InnoDB: Error: page 1 log sequence number 87166865838
and the DB could be corrupt and to see: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
As suggested, we started MySQL with innodb_force_recovery = 2 and innodb_purge_threads = 0, backed up the databases, and then dropped the database with the corrupted table(s).
MySQL still does not start. The error log ASCII/HEX dump shows segments of data that was in the deleted database.
Why are we still having problems with the deleted database? It feels like a process is still hanging, or something might not have deleted properly.
Any ideas? Thanks
Using MySQL 5.5.3 on Ubuntu 14.04
I recently experienced a DDoS attack. It overwhelmed and crashed my server. Upon restart all of the innodb tables were corrupted in every database on the server.
I have since rebuilt the databases and all of the tables. I had to recreate innodb_table_stats, innodb_index_stats.
Everything now seems to be running fine, the website is up but I have one persistent error that keeps coming up. The general log is filling with these quite rapidly.
2014-07-13 15:28:37 7fd70b374700 InnoDB: Error: page 193 log sequence number 526819726
InnoDB: is in the future! Current system log sequence number 156433332.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
I tried Changing the Number and Size of mySQL logs. It all went fine with no errors. But I am still seeing these errors mount up in the new log.
What else can I try to sync up these log sequence numbers? I am pretty new to this backend database work. Being forced to learn since my hosts tech support sucks.
I am currently on: CENTOS 6.5 x86_64 virtuozzo – vps- WHM 11.44.0 (build 22)
mySQL: 5.6.17
"page ... log sequence number" is when the page was last modified. The LSN in the header is larger than one in the redo log.
The easiest way to fix it is to rebuild the table with noop ALTER TABLE.
ALTER TABLE mytable ENGINE InnoDB;
It will rebuild PRIMARY index as well as its secondary indexes. After that the error should go away.
The ALTER is going to block the table, so if it's large and if the site is in production the best option is to rebuild it with pt-online-schema-change. It will do the same, but won't block the table but for brief moment.
pt-online-schema-change --alter "ENGINE=InnoDB" D=sakila,t=actor
I am running MySQL 5.1 on Ubuntu 10.04, and until recently, I used the default data dir location. Some other parts of the config file were tuned for performance, but the paths stayed default.
Recently, I started running out of space and decided to add another hard disk, mount it as /mysql and use it solely for MySQL data. So, I changed the paths, copied the old data dir into the new data dir and thought that would be the end of it.
Unfortunately, it wasn't - and it later turned out that apparmor was the issue, even though I updated the MySQL profile in apparmor to reflect the new path(s). After some messing about, disabling apparmor, the server would work and I was able to import the big database that is the original reason I needed more space.
Now, that was yesterday - the whole 200GB database was imported, keys were sorted and everything seemed fine until I tried to start the server today. Here's the error that I see in the log:
120913 13:53:38 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name /home/{my_username}/mysql/data/ibdata1
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.
Here's a few strange things with that:
a) I'm sudo-ed in as root, and I'm using the 'service mysql start' command to start it
b) There's no mention of /home/{my_username}... path ANYWHERE in any of the configs.
I couldn't find any info or bug reports regarding this type of a problem. I don't even know what I would search for, since the problem can't really be explained in less than 2 paragraphs.
Further information: Manually setting innodb_data_home_dir eliminates the earlier problem, however, now I get this instead:
120913 14:08:06 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name /home/poplar/mysql/innodb-logs/ib_logfile0
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.
Now, there's no "poplar" user on this box, and I haven't got the faintest idea why would it want to be trying to put the log file there.
Well, it turned out that setting innodb_log_group_home_dir to the new MySQL data dir (where my log was before, and where it should default to anyway) did the trick.
The server now starts properly and all the data seems to be there.
I still don't know where it got the 'poplar' username as a good idea to try placing log files, but it could be some leftover (mis)configuration from AppArmor that wasn't cleanly re-set when I uninstalled it.
I fairly new to MySQL database. I am repeatedly seeing an error from MySQL saying the table is marked as crashed and should be repaired. However I am able to repair the crashed table by using the command myisamchk. By the way, I am using MYISAM database engine in MySQL.
I just wanted to know under what circumstances would a DB table crash and how I can prevent it from happening again?
I am connecting to MySQL(5.0) database from Tcl (8.5) script using mysqltcl library (3.0).
MyISAM tables are very easy to crash. There is header info in each table that keeps track of how many open file handles a MyISAM table has.
If mysqld crashes, any MyISAM table that had open file handles to it never had the opportunity to decrement the file handle count upon each file handle closing. Thus, if a new file handle opens a MyISAM table (.MYD file) and mysqld discovers a mismatch between the number of file handles a MyISAM table believes is open and the the number of file handles the MyISAM table actually has open, the table is declared crashed.
There are four(4) methods for handling this:
METHOD #1 : Setup automatic MyISAM repair
See my post https://dba.stackexchange.com/a/15079/877 on how to set this up upon a MySQL restart (Mar 15, 2012)
METHOD #2 : Use InnoDB instead of MyISAM
InnoDB has crash recovery built into the Storage Engine's initialization. MyISAM does not
METHOD #3 : Use Aria instead of MyISAM
Aria is MariaDB's drop-in replacement for MyISAM. It features crash recovery mechanisms for individual tables.
METHOD #4 : Don't kill -9 on mysqld
If mysqld crashes, deliberately or involuntarily, header info for all open MyISAM tables will get them into a crashed state. Avoid having to manually kill mysqld.
I noticed that when I attempt to do a LVM snapshot of my database volume, after running FLUSH TABLES WITH READ LOCK, then rsync that snapshot to a new system, the tables are marked as crashed and have to be repaired.
I suspect this has to do with there being a file handle on the original machine with the table open, and then I'm syncing the that status to the new machine and it sees a mismatch in the file handles and decides it needs to repair.
This repair is problematic because it takes hours (it is a giant table). So the only reliable way to actually get a snapshot that isn't crashed is to shutdown the database before taking the snapshot, but then I cannot get the SHOW MASTER STATUS to setup replication.
Is it possible to clean a mysql innodb storage engine so it is not storing data from deleted tables?
Or do I have to rebuild a fresh database every time?
Here is a more complete answer with regard to InnoDB. It is a bit of a lengthy process, but can be worth the effort.
Keep in mind that /var/lib/mysql/ibdata1 is the busiest file in the InnoDB infrastructure. It normally houses six types of information:
Table Data
Table Indexes
MVCC (Multiversioning Concurrency Control) Data
Rollback Segments
Undo Space
Table Metadata (Data Dictionary)
Double Write Buffer (background writing to prevent reliance on OS caching)
Insert Buffer (managing changes to non-unique secondary indexes)
See the Pictorial Representation of ibdata1
InnoDB Architecture
Many people create multiple ibdata files hoping for better disk-space management and performance, however that belief is mistaken.
Can I run OPTIMIZE TABLE ?
Unfortunately, running OPTIMIZE TABLE against an InnoDB table stored in the shared table-space file ibdata1 does two things:
Makes the table’s data and indexes contiguous inside ibdata1
Makes ibdata1 grow because the contiguous data and index pages are appended to ibdata1
You can however, segregate Table Data and Table Indexes from ibdata1 and manage them independently.
Can I run OPTIMIZE TABLE with innodb_file_per_table ?
Suppose you were to add innodb_file_per_table to /etc/my.cnf (my.ini). Can you then just run OPTIMIZE TABLE on all the InnoDB Tables?
Good News : When you run OPTIMIZE TABLE with innodb_file_per_table enabled, this will produce a .ibd file for that table. For example, if you have table mydb.mytable witha datadir of /var/lib/mysql, it will produce the following:
/var/lib/mysql/mydb/mytable.frm
/var/lib/mysql/mydb/mytable.ibd
The .ibd will contain the Data Pages and Index Pages for that table. Great.
Bad News : All you have done is extract the Data Pages and Index Pages of mydb.mytable from living in ibdata. The data dictionary entry for every table, including mydb.mytable, still remains in the data dictionary (See the Pictorial Representation of ibdata1). YOU CANNOT JUST SIMPLY DELETE ibdata1 AT THIS POINT !!! Please note that ibdata1 has not shrunk at all.
InnoDB Infrastructure Cleanup
To shrink ibdata1 once and for all you must do the following:
Dump (e.g., with mysqldump) all databases into a .sql text file (SQLData.sql is used below)
Drop all databases (except for mysql and information_schema) CAVEAT : As a precaution, please run this script to make absolutely sure you have all user grants in place:
mkdir /var/lib/mysql_grants
cp /var/lib/mysql/mysql/* /var/lib/mysql_grants/.
chown -R mysql:mysql /var/lib/mysql_grants
Login to mysql and run SET GLOBAL innodb_fast_shutdown = 0; (This will completely flush all remaining transactional changes from ib_logfile0 and ib_logfile1)
Shutdown MySQL
Add the following lines to /etc/my.cnf (or my.ini on Windows)
[mysqld]
innodb_file_per_table
innodb_flush_method=O_DIRECT
innodb_log_file_size=1G
innodb_buffer_pool_size=4G
(Sidenote: Whatever your set for innodb_buffer_pool_size, make sure innodb_log_file_size is 25% of innodb_buffer_pool_size.
Also: innodb_flush_method=O_DIRECT is not available on Windows)
Delete ibdata* and ib_logfile*, Optionally, you can remove all folders in /var/lib/mysql, except /var/lib/mysql/mysql.
Start MySQL (This will recreate ibdata1 [10MB by default] and ib_logfile0 and ib_logfile1 at 1G each).
Import SQLData.sql
Now, ibdata1 will still grow but only contain table metadata because each InnoDB table will exist outside of ibdata1. ibdata1 will no longer contain InnoDB data and indexes for other tables.
For example, suppose you have an InnoDB table named mydb.mytable. If you look in /var/lib/mysql/mydb, you will see two files representing the table:
mytable.frm (Storage Engine Header)
mytable.ibd (Table Data and Indexes)
With the innodb_file_per_table option in /etc/my.cnf, you can run OPTIMIZE TABLE mydb.mytable and the file /var/lib/mysql/mydb/mytable.ibd will actually shrink.
I have done this many times in my career as a MySQL DBA. In fact, the first time I did this, I shrank a 50GB ibdata1 file down to only 500MB!
Give it a try. If you have further questions on this, just ask. Trust me; this will work in the short term as well as over the long haul.
CAVEAT
At Step 6, if mysql cannot restart because of the mysql schema begin dropped, look back at Step 2. You made the physical copy of the mysql schema. You can restore it as follows:
mkdir /var/lib/mysql/mysql
cp /var/lib/mysql_grants/* /var/lib/mysql/mysql
chown -R mysql:mysql /var/lib/mysql/mysql
Go back to Step 6 and continue
UPDATE 2013-06-04 11:13 EDT
With regard to setting innodb_log_file_size to 25% of innodb_buffer_pool_size in Step 5, that's blanket rule is rather old school.
Back on July 03, 2006, Percona had a nice article why to choose a proper innodb_log_file_size. Later, on Nov 21, 2008, Percona followed up with another article on how to calculate the proper size based on peak workload keeping one hour's worth of changes.
I have since written posts in the DBA StackExchange about calculating the log size and where I referenced those two Percona articles.
Aug 27, 2012 : Proper tuning for 30GB InnoDB table on server with 48GB RAM
Jan 17, 2013 : MySQL 5.5 - Innodb - innodb_log_file_size higher than 4GB combined?
Personally, I would still go with the 25% rule for an initial setup. Then, as the workload can more accurate be determined over time in production, you could resize the logs during a maintenance cycle in just minutes.
The InnoDB engine does not store deleted data. As you insert and delete rows, unused space is left allocated within the InnoDB storage files. Over time, the overall space will not decrease, but over time the 'deleted and freed' space will be automatically reused by the DB server.
You can further tune and manage the space used by the engine through an manual re-org of the tables. To do this, dump the data in the affected tables using mysqldump, drop the tables, restart the mysql service, and then recreate the tables from the dump files.
I follow this guide for a complete reset (as root):
mysqldump --all-databases --single-transaction | gzip -c > /tmp/mysql.all.sql.gz
service mysql stop
mv /var/lib/mysql /var/lib/mysql.old; mkdir -m700 /var/lib/mysql; chown mysql:mysql /var/lib/mysql
mysql_install_db # mysql 5.5
mysqld --initialize-insecure # mysql 5.7
service mysql start
zcat /tmp/mysql.all.sql.gz | mysql
service mysql restart
What nobody seems to mention is the impact innodb_undo_log_truncate setting can have.
Take a look at my answer at How to shrink/purge ibdata1 file in MySQL.