Yesterday, my local server HDD crashes and goes for automatic scandisk at start up. I was not in office so I really don't know what had happen. But today when we start Server we are getting
130523 10:49:36 InnoDB: Operating system error number 23 in a file operation.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
InnoDB: File name .\ibdata1
InnoDB: File operation call: 'Windows aio'.
I try to run scandisk on C: and it found 4 corrupted records, but no bad sector or something. Database Server has around 300 Database, I Am not sure how to recover it. Someone suggest delete Log files and try to set
innodb_Force_recovery = 4
I try that too, but some of database are unavailable and crash MySql when I try to use them. I am not sure as reloading 300 database is not an easy job.
Edit:
I am on Windows 2008 Web Edition if that matters.
Sounds like you're on Windows: the ENFILE error basically says you have too many open files. Not quite sure how exactly you get around it: in general it is somewhere between 512 and 2048 in a process.
Would suggest maybe moving all the files out of the MySQL data directory and trying to recover them one by one.
I have the same issue but the error is pointing to another file :
InnoDB: File name .\ib_logfile0
Deleting (or renaming) both logfiles from the data directory (.\ib_logfile0 and .\ib_logfile1) solves the issue for me.
Related
I have a C# application performing some database operations on a MySQL 5.7 server. Once the complete system hunp up and I had to hard-reset it. When it comes to a specific table read/write operation now the database server crashes. The windows log shows
InnoDB: Trying to access page number 286720 in space 29,
space name myInstance/myTable, which is outside the tablespace bounds.
Byte offset 0, len 16384, i/o type read.
I tried to use mysqlcheck --repair but it fails because note : The storage engine for the table doesn't support repair.
I've read some advices that say I should start MySQL in a recovery mode, so I added
[mysqld]
innodb_force_recovery=4
to the my.ini config file whereupon I should be able to use mysqldump to export the affected database table. But unfortunatelly I am not.
mysqldump: Error 2013: Lost connection to MySQL server
during query when dumping table `myTable` at row: 1246
Edit:
I checked the error log again and found lots of entries saying
[ERROR] C:\Program Files\MySQL\MySQL Server 5.7\bin\mysqld.exe:
The table 'myTable' is full
I'm running the server on a Windows 32bit OS with an NTFS formatted partition. The myTable.ibd file size is around 4.5 GB, checking C.10.3 Limits on Table Size states a file size limit of "2TB (possibly larger)" for Win32 w/ NTFS.
While checking the reasons for my error the only possible cause I found was a full InnoDB tablespace. The solution might be "Changing the Number or Size of InnoDB Redo Log Files" although the coherence is a bit vague to me. Nevertheless I increased the size of the Redo Log Files from 48M to 100M. But nothing changed.
If I perform a SQL select * from myTable order by Id desc the server crashes instantly. Error log entry is exactly the same as above.
I checked the chapter 15.7.1 Resizing the InnoDB System Tablespace as well and found out that innodb_data_file_path is not explicitly specified.
Any ideas what I can do now? Thanks a lot!
InnoDB cannot repair corruption in tablespaces. This was never implemented and mysqlcheck won't help in any way.
The corruption is in space id 29 which is table myInstance.myTable. To repair it you need to dump all records from this table with innodb_force_recovery. Try all values from 4 to 6 until MySQL doesn't crash. Then drop the table and reload the dump.
If MySQL crashes even with innodb_force_recovery=6 then restore the table from backup.
If you don't have backup - use script http://bazaar.launchpad.net/~percona-dev/percona-data-recovery-tool-for-innodb/trunk/view/head:/fetch_data.sh . It will fetch as many records as it can.
One possible cause could be that the ib_logfile* files are corrupted.
To fix this, remove these files using rm ib_logfile*.
Where are these files?
These files are in mysql datadir. The location of datadir depends on the OS. Check my.cnf, in osx that should be in /usr/local/etc.
I'm trying to start MySQL in safe mode but the log file just keeps spitting out:
[ERROR] InnoDB: Unable to lock ./ibdata1 error: 35
[Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
Notes:
I've stopped all mysql processes and have checked using the ps command
I've read this having repeated trouble with MySQL and Snow Leopard - Unable to lock ./ibdata1, error: 35
Just had this issue with MAMP Pro. Couldn't stop or start MySQL. Typed ps aux | grep mysql, then kill [process id] for each line. Then I could successfully launch MySQL again from MAMP Pro.
Strange error codes, both from MySQL and the operating system, can be explained by running a shell command perror which is a part of MySQL distribution. On Linux, for example, perror 35 returns OS error code 35: Resource deadlock avoided. I do not have an OSX system handy to test, but from this:
http://www.opensource.apple.com/source/xnu/xnu-1456.1.26/bsd/sys/errno.h
I gather that error 35 is "Resource temporarily unavailable", meaning that some other process (possibly another instance of mysqld?) has already locked the file.
To debug, first I would check if there is another MySQL process running by chance (looks like you already have), and if not, reboot for good measure to make sure there are no strange stale file locks, and then try again.
Somebody has run into this error before, and found a solution:
mysql_install_db, error: 35, on a Mac OS X 10.9.1
I had a similar issue after updating MAMP from 4.x to 5.x
Unfortunately the only way that I could manage to resolve this was by manually removing all of my database files (moving them into another directory) then starting MySQL via MAMP; this seemed to work.
The files in question (for me) were in:
/Library/Application Support/appsolute/MAMP PRO/db/mysql57
All of the files in this directory (except the mysql directory) need moving somewhere else. I was then able to start MySQL.
I realise this isn't a very useful answer for most situations, but if this is a local development environment (like MAMP, as in my case) then I think losing the local databases is probably acceptable.
I have a Master - Slave setup with MySql v5.1.39 running ~10 db's on 12 core Linux machine. I had to move the bin-log files to a separate disk for performance issues. So I followed these steps:
Stop everything that's using the db
Stop Slave
Stop master
Change paths in /my.cfg to /mysql/log/* to /mysql/newlog/* on master and slave
Copy /mysql/log/* to /mysql/newlog/. on master and slave
Start Slave
All Ok!
Start Master
First problem! on the slave:
150113 12:21:22 [ERROR] Got fatal error 1236: 'Could not find first log file name in binary log index file' from master when reading data from binary log
150113 12:21:22 [Note] Slave I/O thread exiting, read up to log 'bin-log.005523', position 716864371
Now a quick Google didn't resolve anything and since downtime is an issue. I stopped the Master, changed the configuration back and restarted. Now the second "problem"!
...
150113 13:02:22 InnoDB: Error: page 182380 log sequence number 3407 300161079
InnoDB: is in the future! Current system log sequence number 3407 299353326.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: for more information.
...
I quote problem because everything works fine. Replication to the slave restarted and worked. I started the applications and those work fine. But when start MySql on the Master, I get the errors above, about 50 of them with different page- and sequence numbers.
How does moving files affect page- and sequence numbers and where do they come from? How big is my problem? everything seems to work fine.
Please ask if you need anymore information and thanks for your help.
First problem was caused by the file /mysql/log/bin-log.index. I forgot to change the contents of this file to point to the new directory of the log files:
/mysql/log/bin-log.000028 -> /mysql/newlog/bin-log.000028
/mysql/log/bin-log.000029 -> /mysql/newlog/bin-log.000029
/mysql/log/bin-log.000030 -> /mysql/newlog/bin-log.000030
/mysql/log/bin-log.000031 -> /mysql/newlog/bin-log.000031
The second problem was caused by the timestamps of some files. I should have preserved the timestamps with cp -p log/* newlog/. or rsync -avrx log/* newlog/..
I'm trying to clone raw data from all databases on a MySQL instance in Live to a test environment. The network guys have told me the data has been synched and copied across but I can't start the MySQL instance in the test environment. I'm using the innodb engine and I can see the ibdata1 file, mysql-bin files and ib_logfiles copied over along with the relevant db folders.
The error I'm getting in the error log looks like the following:
130911 13:53:08 InnoDB: Error: table <table-name>
InnoDB: in InnoDB data dictionary has tablespace id <id>,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
The cloning process doesn't stop the Live MySQL instance and I'm wondering is this the problem. I don't want to use mysqldump or another backup tool. I just want to copy the raw data across. Thanks for any advice.
You can't hot-copy these files and expect them to magically work.
You can use the innobackupex tool to create a stable snapshot. This will take care of adjusting the files as necessary to be consistent and complete.
I am running MySQL 5.1 on Ubuntu 10.04, and until recently, I used the default data dir location. Some other parts of the config file were tuned for performance, but the paths stayed default.
Recently, I started running out of space and decided to add another hard disk, mount it as /mysql and use it solely for MySQL data. So, I changed the paths, copied the old data dir into the new data dir and thought that would be the end of it.
Unfortunately, it wasn't - and it later turned out that apparmor was the issue, even though I updated the MySQL profile in apparmor to reflect the new path(s). After some messing about, disabling apparmor, the server would work and I was able to import the big database that is the original reason I needed more space.
Now, that was yesterday - the whole 200GB database was imported, keys were sorted and everything seemed fine until I tried to start the server today. Here's the error that I see in the log:
120913 13:53:38 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name /home/{my_username}/mysql/data/ibdata1
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.
Here's a few strange things with that:
a) I'm sudo-ed in as root, and I'm using the 'service mysql start' command to start it
b) There's no mention of /home/{my_username}... path ANYWHERE in any of the configs.
I couldn't find any info or bug reports regarding this type of a problem. I don't even know what I would search for, since the problem can't really be explained in less than 2 paragraphs.
Further information: Manually setting innodb_data_home_dir eliminates the earlier problem, however, now I get this instead:
120913 14:08:06 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name /home/poplar/mysql/innodb-logs/ib_logfile0
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.
Now, there's no "poplar" user on this box, and I haven't got the faintest idea why would it want to be trying to put the log file there.
Well, it turned out that setting innodb_log_group_home_dir to the new MySQL data dir (where my log was before, and where it should default to anyway) did the trick.
The server now starts properly and all the data seems to be there.
I still don't know where it got the 'poplar' username as a good idea to try placing log files, but it could be some leftover (mis)configuration from AppArmor that wasn't cleanly re-set when I uninstalled it.