Due to some hardware issues one of our database faulted and even using force recovery is not able to start it back up. The error I see on our logs is:
Doing recovery: scanned up to log sequence number 40160211114
Error: trying to access page number 4293806975 in space 0, space name
./ibdata1, which is outside the tablespace bounds. Byte offset 0, len
16384, i/o type 10.
If you get this error at mysqld startup, please check that your my.cnf
matches the ibdata files that you have in the MySQL server.
Will reinstalling mysql lose my data?
Related
i have Windows 2016 Standard OS with installed MySQL 5.5.33.0.
This morning I found out that the mysql server is not running and cannot be turned on. When i try to turn it on, it crashes immediately, and there is only exception information 0xc0000005 in Event Viewer. I can't even run mysqlcheck because it doesn't work when mysqld isn't running.
I will be very grateful for any advice.
I was able to enable logging and set paths in MySQL settings. There are quite a few entries in the log with this text (numbers change):
210107 10:28:21 InnoDB: Error: page 33382 log sequence number 114951961648
InnoDB: is in the future! Current system log sequence number 111386529682.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files.
See InnoDB: dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: for more information.
210107 10:32:33 InnoDB: Error: Insert buffer insert fails; page free 43, dtuple size 50
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
0: len 40; hex 436f6d70757465722050726573732020202020202020202020202020202020202020202020202020;
asc Computer Press ;;
1: len 4; hex 800070da; asc p ;;
InnoDB: The table where this index record belongs
InnoDB: is now probably corrupt. Please run CHECK TABLE on
InnoDB: that table.
InnoDB: space 0, page 109621, zip_size 0, bitmap bits 1
Progress update:
I managed to start MySQL with innodb_force_recovery = 3 and it even runs for a while. If I run mysqlcheck.exe --all-databases --auto-repair, it reports corrupt in some tables of this type:
Warning: InnoDB: Index 'ix_linear_pole' contains 1683314 entries, should be 1683294.
However, the check does not run to the end and for one specific table the whole mysql server crashes again.
UPDATE: I found out that the error is in the loaded command in the InnoDB buffer, but I don't know how to delete it. Each time I turn on the server, a command in the buffer crash it.
After several hours of research, I found that MySQL was failing to load InnoDB Buffer. The easiest way was to add a command to my.ini
innodb_force_recovery = 4
However, the database is started only for SHOW data. Using mysqldump, I backed up the data without indexes. Then I deleted the database from the server, I also deleted the logs and database data from the system (C: \ ProgramData \ ... \ MySQL \ data \ ibdata and iblog0 and iblog1). Then I stopped MySQL and removed innodb_force_recovery. The database then ran in but without data.
I uploaded the original data back using normal recovery and created indexes.
From that moment on, everything works without problems.
I am running MySQL (5.7) on a Windows 2012 server. One of the table (> 1 million records) seems to be corrupted. The mysql error log shows
2019-01-16T09:41:09.256990Z 0 [Note] InnoDB: Buffer pool(s) load completed at 190116 14:41:09
2019-01-16T09:41:25.153201Z 2 [Warning] IP address 'ip' could not be resolved: No such host is known.
2019-01-16T09:41:25.620502Z 3 [ERROR] InnoDB: Trying to access page number 3 in space 470, space name shop/products, which is outside the tablespace bounds. Byte offset 0, len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server.
2019-01-16T09:41:25.623268Z 3 [ERROR] InnoDB: Server exits.
This issue raised when I was inserting records in this table. This table have 22 indexed columns and one of them is FULLTEXT.
I have also tried innodb_force_recovery = [1 - 6], but I am not able to dump the products table. Whenever I tried to access this table it crashes the DB server.
is there any way to get dump or recover this table. I have old dated backup so I can't use that one. Thanks :)
MYSQL Server Details:
Version 5.7 (32 bit)
Windows Server 2012
Database Engine: InnoDB
Updated
I am facing the strange issue. When I execute the table status command it shows the rows count is zero,
but products.ibd file size is 4GB in mysql data folder.
SHOW TABLE STATUS FROM shop LIKE 'products';
File size by OS
I have a Master - Slave setup with MySql v5.1.39 running ~10 db's on 12 core Linux machine. I had to move the bin-log files to a separate disk for performance issues. So I followed these steps:
Stop everything that's using the db
Stop Slave
Stop master
Change paths in /my.cfg to /mysql/log/* to /mysql/newlog/* on master and slave
Copy /mysql/log/* to /mysql/newlog/. on master and slave
Start Slave
All Ok!
Start Master
First problem! on the slave:
150113 12:21:22 [ERROR] Got fatal error 1236: 'Could not find first log file name in binary log index file' from master when reading data from binary log
150113 12:21:22 [Note] Slave I/O thread exiting, read up to log 'bin-log.005523', position 716864371
Now a quick Google didn't resolve anything and since downtime is an issue. I stopped the Master, changed the configuration back and restarted. Now the second "problem"!
...
150113 13:02:22 InnoDB: Error: page 182380 log sequence number 3407 300161079
InnoDB: is in the future! Current system log sequence number 3407 299353326.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: for more information.
...
I quote problem because everything works fine. Replication to the slave restarted and worked. I started the applications and those work fine. But when start MySql on the Master, I get the errors above, about 50 of them with different page- and sequence numbers.
How does moving files affect page- and sequence numbers and where do they come from? How big is my problem? everything seems to work fine.
Please ask if you need anymore information and thanks for your help.
First problem was caused by the file /mysql/log/bin-log.index. I forgot to change the contents of this file to point to the new directory of the log files:
/mysql/log/bin-log.000028 -> /mysql/newlog/bin-log.000028
/mysql/log/bin-log.000029 -> /mysql/newlog/bin-log.000029
/mysql/log/bin-log.000030 -> /mysql/newlog/bin-log.000030
/mysql/log/bin-log.000031 -> /mysql/newlog/bin-log.000031
The second problem was caused by the timestamps of some files. I should have preserved the timestamps with cp -p log/* newlog/. or rsync -avrx log/* newlog/..
I am running MySQL 5.1 on Ubuntu 10.04, and until recently, I used the default data dir location. Some other parts of the config file were tuned for performance, but the paths stayed default.
Recently, I started running out of space and decided to add another hard disk, mount it as /mysql and use it solely for MySQL data. So, I changed the paths, copied the old data dir into the new data dir and thought that would be the end of it.
Unfortunately, it wasn't - and it later turned out that apparmor was the issue, even though I updated the MySQL profile in apparmor to reflect the new path(s). After some messing about, disabling apparmor, the server would work and I was able to import the big database that is the original reason I needed more space.
Now, that was yesterday - the whole 200GB database was imported, keys were sorted and everything seemed fine until I tried to start the server today. Here's the error that I see in the log:
120913 13:53:38 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name /home/{my_username}/mysql/data/ibdata1
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.
Here's a few strange things with that:
a) I'm sudo-ed in as root, and I'm using the 'service mysql start' command to start it
b) There's no mention of /home/{my_username}... path ANYWHERE in any of the configs.
I couldn't find any info or bug reports regarding this type of a problem. I don't even know what I would search for, since the problem can't really be explained in less than 2 paragraphs.
Further information: Manually setting innodb_data_home_dir eliminates the earlier problem, however, now I get this instead:
120913 14:08:06 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name /home/poplar/mysql/innodb-logs/ib_logfile0
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.
Now, there's no "poplar" user on this box, and I haven't got the faintest idea why would it want to be trying to put the log file there.
Well, it turned out that setting innodb_log_group_home_dir to the new MySQL data dir (where my log was before, and where it should default to anyway) did the trick.
The server now starts properly and all the data seems to be there.
I still don't know where it got the 'poplar' username as a good idea to try placing log files, but it could be some leftover (mis)configuration from AppArmor that wasn't cleanly re-set when I uninstalled it.
I have problem starting the MySql server.
The log says:
InnoDB: Error in opening ./ibdata1
111220 16:16:43 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
/usr/libexec/mysqld: Disk is full writing './mysql-bin.000028' (Errcode: 28). Waiting for someone to free space... Retry
in 60 secs
After I check the disk - it says it's full.
So, after searching for solution - I found that I need to purge the binary log.
However, in order to purge - I need to start the MySql server, but all the spacein the disk is taken by the binary log, so I can't start...
It's also not advised to simply delete the binary logs.
So, I am kind of stuck.
Can't run the mysql to purge logs and can't purge logs because can't run server.
Any help? :)
Edit: The disk contains only the logs, there's nothing else.
If the disk is ext[2|3|4] you can use tune2fs to set the portion of the disk reserved for root to 0, giving you maybe enough breathing room to start the server
this would be tune2fs -m 0 /dev/whatever (after unmounting, ofcourse)
Try to start the mysql server with the option --expire_logs_days=, it should delete the log and older than days directory at startup.
bye
Gianluca