ibdata1 file is being deleted - mysql

Question

Last week after some previous database tuning, I stopped mysql and it failed to restart.
After a long period of troubleshooting I found that the ibdata1 file was not as big as it should have been, it had been deleted and recreated as new.
I retrieved the old 9.5gb file from backup, replaced it and mysql started again, happy days.
I've been having some more server trouble today, had a look in the mysql folder and the file has disappeared again.
I haven't stopped mysql yet so everything is still up and running, I will have to retrieve it from backup and restart with my fingers crossed.
So my question is, why is it disappearing?? My guess us I've made an accidental change in the my.cnf file and then not restarted. Unfortunately I don't have a backup of the file because I didn't know there was a change made.
(Untidy) My.cnf is as follows:
[mysqld]
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0
innodb_thread_concurrency= 4
innodb_buffer_pool_size = 2G
thread_concurrency = 3
thread_cache_size = 32
table_cache = 1024
query_cache_size = 64M
query_cache_limit = 2M
join_buffer_size = 8M
tmp_table_size = 256M
key_buffer = 32M
innodb_autoextend_increment=512
max_allowed_packet = 16M
max_heap_table_size = 256M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
innodb_log_file_size = 100M
innodb_additional_mem_pool_size = 20M
innodb_flush_log_at_trx_commit=2
innodb_lock_wait_timeout=1800
innodb_log_buffer_size=500K
log-error=/var/log/mysqld.log
slow_query_log = /var/log/mysql-slow.log
long_query_time = 5
pid-file=/var/run/mysqld/mysqld.pid
sort_buffer_size = 2M
read_buffer_size = 2M
wait_timeout = 120
key_buffer = 384M
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 1M
max_connections=50
query_cache_type = 1
Any help greatly appreciated!
Thanks

Answer 1

Don't stop MySQL! mysqld keeps it open while it's running, so it's still on the file system.
MySQL never deletes ibdata1, so it must be some external command.
To recover the database stop all writes to the database, wait till the main thread is in "waiting for server activity" or "sleeping" state:
mysql> pager grep Main
PAGER set to 'grep Main'
mysql> show engine innodb status\G
Main thread id 4994568192, state: sleeping
1 row in set (0.00 sec)
Take dump of all databases (this step is not necessary, but for extra safety do it);
mysqldump -A > mydb.sql
Find the deleted ibdata1 in /proc filesystem and copy it back to MySQL datadir
# ls -la /proc/`pidof mysqld`/fd/ | grep -e ibdata
lrwx------ 1 root root 64 May 26 02:41 3 -> /var/lib/mysql/ibdata1 (deleted)
Note 3 - it's a file descriptor of ibdata1.
Copy ibdata1 back:
# cp /proc/`pidof mysqld`/fd/3 /var/lib/mysql/ibdata1
Then restart MySQL

Answer 2

#Akuzminsky's answer is what you need to do to get your current ibdata1 back. And he is correct that MySQL never deletes ibdata1, regardless of your my.cnf configuration.
So something else is deleting the file. How can one find out? Try running the Linux Audit Daemon. You won't be able to find out what deleted the file last time (unless you were already running the Audit Daemon), but in case it happens again you're ready.
See this StackExchange answer for details: https://askubuntu.com/questions/48844/how-to-find-the-pid-of-the-process-which-has-deleted-a-file