I am using Mariadb (mysql Ver 15.1 Distrib 5.5.40-MariaDB) in CentOS (CentOS Linux release 7.0.1406). I tried to restart Mariadb.
systemctl start mariadb
Job for mariadb.service failed. See 'systemctl status mariadb.service' and 'journalctl -xn' for details.
When I checked : /var/log/mysql-error.log as suggested by 'systemctl status mariadb.service'
170523 10:20:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170523 10:20:10 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
170523 10:20:10 [Warning] Changed limits: max_open_files: 1024 max_connections: 214 table_cache: 400
170523 10:20:10 InnoDB: The InnoDB memory heap is disabled
170523 10:20:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170523 10:20:10 InnoDB: Compressed tables use zlib 1.2.7
170523 10:20:10 InnoDB: Using Linux native AIO
170523 10:20:11 InnoDB: Initializing buffer pool, size = 15.6G
170523 10:20:11 InnoDB: Completed initialization of buffer pool
170523 10:20:11 InnoDB: highest supported file format is Barracuda.
170523 10:20:12 InnoDB: Waiting for the background threads to start
170523 10:20:13 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 5336723441509
170523 10:20:13 [Note] Plugin 'FEEDBACK' is disabled.
170523 10:20:13 [ERROR] /usr/libexec/mysqld: unknown variable 'extra-lsndir=/var/tmp'
170523 10:20:13 [ERROR] Aborting
170523 10:20:13 InnoDB: Starting shutdown...
170523 10:20:15 InnoDB: Shutdown completed; log sequence number 5336723441509
170523 10:20:15 [Note] /usr/libexec/mysqld: Shutdown complete
170523 10:20:15 mysqld_safe mysqld from pid file /var/lib/mysql/master.compare-prices.repchecker.jp.pid ended
170523 10:21:10 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
170523 10:21:10 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
170523 10:21:10 [Warning] Changed limits: max_open_files: 1024 max_connections: 214 table_cache: 400
170523 10:21:10 InnoDB: The InnoDB memory heap is disabled
170523 10:21:10 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170523 10:21:10 InnoDB: Compressed tables use zlib 1.2.7
170523 10:21:10 InnoDB: Using Linux native AIO
170523 10:21:10 InnoDB: Initializing buffer pool, size = 15.6G
170523 10:21:11 InnoDB: Completed initialization of buffer pool
170523 10:21:11 InnoDB: highest supported file format is Barracuda.
170523 10:21:12 InnoDB: Waiting for the background threads to start
170523 10:21:13 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 5336723441509
170523 10:21:13 [Note] Plugin 'FEEDBACK' is disabled.
170523 10:21:13 [ERROR] /usr/libexec/mysqld: unknown variable 'extra-lsndir=/var/tmp'
170523 10:21:13 [ERROR] Aborting
.
.
.
.
PS: /var/lib/mysql directory is owned by mysql (there is one more directory mysql~ which is empty)
drwxr-xr-x 8 mysql mysql 4096 May 23 11:05 mysql
drwxr-xr-x. 3 mysql mysql 26 Feb 4 2015 mysql~
Please provide some suggestion. How can I restart my mariadb, I have important database under mariadb.
From the last line of error
[ERROR] /usr/libexec/mysqld: unknown variable 'extra-lsndir=/var/tmp'
It is clear that this variable is not known so try to comment it from /etc/my.cnf and try again.
# Point the following paths to different dedicated disks
tmpdir = /var/tmp
extra-lsndir = /var/tmp ------> Comment this line
target-dir=/var/tmp
I hope it will help.
Related
My Macbook crashed and now MAMP (latest version v6.6.3) is not working anymore.
I was able to log in to MYSQL only over command line with
/Applications/MAMP/bin/startMysql.sh
/Applications/MAMP/bin/startApache.sh
But my databases are gone.
However, I can see in the error log that there is an issue with one of my databases (drupal8). So the database is still somewhere? But why can't I see it?
2022-10-12T02:36:17.6NZ mysqld_safe Logging to '/Applications/MAMP/logs/mysql_error.log'.
2022-10-12T02:36:17.6NZ mysqld_safe Starting mysqld daemon with databases from /Library/Application Support/appsolute/MAMP PRO/db/mysql57
mysqld: [Warning] World-writable config file '/Applications/MAMP/tmp/mysql/my.cnf' is ignored.
2022-10-12T02:36:17.347806Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-10-12T02:36:17.349461Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2022-10-12T02:36:17.350939Z 0 [Note] /Applications/MAMP/Library/bin/mysqld (mysqld 5.7.34) starting as process 62124 ...
2022-10-12T02:36:17.363631Z 0 [Warning] Setting lower_case_table_names=2 because file system for /Library/Application Support/appsolute/MAMP PRO/db/mysql57/ is case insensitive
2022-10-12T02:36:17.373326Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-10-12T02:36:17.373353Z 0 [Note] InnoDB: Uses event mutexes
2022-10-12T02:36:17.373365Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-10-12T02:36:17.373375Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-10-12T02:36:17.375559Z 0 [Note] InnoDB: Number of pools: 1
2022-10-12T02:36:17.378292Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-10-12T02:36:17.380118Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2022-10-12T02:36:17.390613Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-10-12T02:36:17.415180Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2022-10-12T02:36:17.418235Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 73772091416
2022-10-12T02:36:17.418356Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 73772146375
2022-10-12T02:36:17.422286Z 0 [Note] InnoDB: Database was not shutdown normally!
2022-10-12T02:36:17.422329Z 0 [Note] InnoDB: Starting crash recovery.
2022-10-12T02:36:17.467479Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 2022-10-12T02:36:17.468325Z 0 [ERROR] InnoDB: Trying to access page number 32 in space 580, space name drupal8/cache_menu, which is outside the tablespace bounds. Byte offset 0, len 16384, i/o type read. If you get this error at mysqld startup, please check that your my.cnf matches the ibdata files that you have in the MySQL server.
2022-10-12T02:36:17.468359Z 0 [ERROR] InnoDB: Server exits.
2022-10-12T02:36:17.6NZ mysqld_safe mysqld from pid file /Applications/MAMP/tmp/mysql/mysql.pid ended
When I open PhpMyAdmin I get an error as well:
PHP 5.5+ is required.
Currently installed version is: 5.4.45
But I have PHP set to 7.4.21 in MAMP.
I have 4 containers using mgt-commerce/mgt-development-environment-7.2. Basically everything runs well until I cannot access the control panel. It returns 500.
After checking the services, logs, etc, it seems like mysql is unable to start. Here is the error log captured in /var/log/mysql/error.log during starting mysql service using sudo service mysql start.
2019-09-26T23:38:43.740283Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2019-09-26T23:38:43.921450Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.26-29) starting as process 1576 ...
2019-09-26T23:38:43.924613Z 0 [Note] InnoDB: PUNCH HOLE support available
2019-09-26T23:38:43.924635Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-09-26T23:38:43.924639Z 0 [Note] InnoDB: Uses event mutexes
2019-09-26T23:38:43.924642Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-09-26T23:38:43.924644Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-09-26T23:38:43.924647Z 0 [Note] InnoDB: Using Linux native AIO
2019-09-26T23:38:43.924859Z 0 [Note] InnoDB: Number of pools: 1
2019-09-26T23:38:43.924920Z 0 [Note] InnoDB: Using CPU crc32 instructions
2019-09-26T23:38:43.925849Z 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2019-09-26T23:38:43.946744Z 0 [Note] InnoDB: Completed initialization of buffer pool
2019-09-26T23:38:43.955252Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-09-26T23:38:43.966654Z 0 [Note] InnoDB: Recovering partial pages from the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
2019-09-26T23:38:44.033822Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2019-09-26T23:38:44.033986Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2019-09-26T23:38:44.034023Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2019-09-26T23:38:44.034040Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2019-09-26T23:38:45.135220Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2019-09-26T23:38:45.135270Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2019-09-26T23:38:45.135283Z 0 [ERROR] Failed to initialize builtin plugins.
2019-09-26T23:38:45.135291Z 0 [ERROR] Aborting
2019-09-26T23:38:45.135301Z 0 [Note] Binlog end
2019-09-26T23:38:45.135371Z 0 [Note] Shutting down plugin 'CSV'
2019-09-26T23:38:45.136207Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
Is there anyone using this image ever getting this kind of error?
Update
It seems like mysql doesn't have the access to a particular file(s) or directory(ies), but I'm not sure which file(s) or directory(ies) . Here is the permission of mysql data directory:
clp#magento2dev:/$ ls -la /var/lib | grep mysql
drwxr-x--- 1 mysql mysql 4096 Sep 26 23:38 mysql
drwxrwx--- 2 mysql mysql 4096 Jul 11 15:24 mysql-files
drwxr-x--- 2 mysql mysql 4096 Jul 11 15:24 mysql-keyring
My database crash with out of memory error ~1-2 times per day. Last crash was at 17:30 in logs and at 19:30 in metrics (different timezones).
Errors in mariadb.log:
181120 16:09:00 [Warning] IP address '185.156.177.144' could not be resolved: Name or service not known
181120 16:35:40 [Warning] IP address '58.221.58.248' could not be resolved: Name or service not known
181120 17:30:38 mysqld_safe Number of processes running now: 0
181120 17:30:38 mysqld_safe mysqld restarted
181120 17:30:41 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
181120 17:30:41 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 16636 ...
181120 17:30:41 [Warning] Changed limits: max_open_files: 1024 max_connections: 70 table_cache: 472
181120 17:30:41 InnoDB: The InnoDB memory heap is disabled
181120 17:30:41 InnoDB: Mutexes and rw_locks use GCC atomic builtins
181120 17:30:41 InnoDB: Compressed tables use zlib 1.2.7
181120 17:30:41 InnoDB: Using Linux native AIO
181120 17:30:41 InnoDB: Initializing buffer pool, size = 150.0M
181120 17:30:42 InnoDB: Completed initialization of buffer pool
181120 17:30:42 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
181120 17:30:42 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
181120 17:30:46 InnoDB: Waiting for the background threads to start
181120 17:30:47 Percona XtraDB (http://www.percona.com) 5.5.49-MariaDB-38.0 started; log sequence number 18741266161
181120 17:30:47 [Note] Plugin 'FEEDBACK' is disabled.
181120 17:30:48 [Note] Server socket created on IP: '0.0.0.0'.
181120 17:30:49 [Note] Event Scheduler: Loaded 0 events
181120 17:30:49 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.52-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
181120 17:30:53 [ERROR] mysqld: Table './admin_wikis/wp_posts' is marked as crashed and should be repaired
181120 17:30:53 [Warning] Checking table: './admin_wikis/wp_posts'
181120 17:31:58 mysqld_safe Number of processes running now: 0
181120 17:31:58 mysqld_safe mysqld restarted
181120 17:32:01 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.
181120 17:32:01 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 16899 ...
181120 17:32:01 [Warning] Changed limits: max_open_files: 1024 max_connections: 70 table_cache: 472
181120 17:32:01 InnoDB: The InnoDB memory heap is disabled
181120 17:32:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
181120 17:32:01 InnoDB: Compressed tables use zlib 1.2.7
181120 17:32:01 InnoDB: Using Linux native AIO
181120 17:32:01 InnoDB: Initializing buffer pool, size = 150.0M
InnoDB: mmap(161447936 bytes) failed; errno 12
181120 17:32:01 InnoDB: Completed initialization of buffer pool
181120 17:32:01 InnoDB: Fatal error: cannot allocate memory for the buffer pool
181120 17:32:01 [ERROR] Plugin 'InnoDB' init function returned error.
181120 17:32:01 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
181120 17:32:01 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
181120 17:32:01 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
181120 17:32:01 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
181120 17:32:01 [Note] Plugin 'FEEDBACK' is disabled.
181120 17:32:02 [ERROR] Unknown/unsupported storage engine: InnoDB
181120 17:32:02 [ERROR] Aborting
181120 17:32:02 [Note] /usr/libexec/mysqld: Shutdown complete
There is nothing interesing in apache log in this time. Event requests count
$ sudo grep -r "GET \/" /etc/httpd/logs/domains/ | cut -d' ' -f4 | sed 's/\[//;s/\]//;s/\"//' | cut -d':' -f1-2 | sort | uniq -c
459 19/Nov/2018:03
426 19/Nov/2018:04
239 19/Nov/2018:05
350 19/Nov/2018:06
381 19/Nov/2018:07
415 19/Nov/2018:08
778 19/Nov/2018:09
500 19/Nov/2018:10
450 19/Nov/2018:11
633 19/Nov/2018:12
458 19/Nov/2018:13
527 19/Nov/2018:14
713 19/Nov/2018:15
654 19/Nov/2018:16
573 19/Nov/2018:17
413 19/Nov/2018:18
471 19/Nov/2018:19
499 19/Nov/2018:20
661 19/Nov/2018:21
452 19/Nov/2018:22
773 19/Nov/2018:23
934 20/Nov/2018:00
295 20/Nov/2018:01
369 20/Nov/2018:02
441 20/Nov/2018:03
384 20/Nov/2018:04
927 20/Nov/2018:05
524 20/Nov/2018:06
972 20/Nov/2018:07
612 20/Nov/2018:08
609 20/Nov/2018:09
677 20/Nov/2018:10
753 20/Nov/2018:11
615 20/Nov/2018:12
717 20/Nov/2018:13
474 20/Nov/2018:14
973 20/Nov/2018:15
510 20/Nov/2018:16
534 20/Nov/2018:17
415 20/Nov/2018:18
90 20/Nov/2018:19
I get some metrics:
Processes before crash:
Processes in the moment of crash:
Memory and CPU usage:
Total RAM 3.6G. Does someone have any ideas?
httpd is the largest memory user; decrease its settings, especially the number of 'children' it is willing to spawn. (Under 10 should suffice for most small installations.)
max_connections: 70 table_cache: 472 -- Cut each of those in half; this will help a little with memory allocation.
Next time the CPU hits 100%, do SHOW FULL PROCESSLIST; in MySQL. That may give you a clue of what runaway queries you have. Then we can focus on them. If that fails, turn on the slowlog; it will catch some of them.
Consider moving to an 8GB server. (Is this a cloud server?)
I have a server running wordpress on it.
It is running well these days, but today I found it can not connect database.
So I checked server, found mysql service was shutdown.
Then I restart , but it only work for a while, after some seconds, it shutdown again.
This is mysql log:
160508 10:07:36 [Warning] IP address '222.186.59.174' could not be resolved: Name or service not known
160508 14:08:14 [Warning] IP address '210.209.127.33' could not be resolved: Name or service not known
160508 23:08:15 [Warning] IP address '183.63.89.196' could not be resolved: Name or service not known
160509 16:08:02 [Warning] IP address '198.55.114.190' has been resolved to the host name '198.55.114.190.static.quadranet.com', which resembles IPv4-address itself.
160510 8:30:34 [Warning] IP address '222.186.134.25' could not be resolved: Name or service not known
160510 12:39:02 [Warning] IP address '118.193.255.253' could not be resolved: Name or service not known
160510 16:06:36 [Warning] IP address '118.193.218.65' could not be resolved: Name or service not known
160511 0:02:03 [Warning] IP address '116.255.183.230' could not be resolved: Name or service not known
160511 00:39:09 mysqld_safe Number of processes running now: 0
160511 00:39:09 mysqld_safe mysqld restarted
160511 0:39:10 [Note] /usr/libexec/mysqld (mysqld 5.5.47-MariaDB) starting as process 24298 ...
160511 0:39:42 InnoDB: The InnoDB memory heap is disabled
160511 0:39:42 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160511 0:39:42 InnoDB: Compressed tables use zlib 1.2.7
160511 0:39:42 InnoDB: Using Linux native AIO
160511 0:39:42 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
160511 0:39:42 InnoDB: Completed initialization of buffer pool
160511 0:39:42 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160511 0:39:42 [ERROR] Plugin 'InnoDB' init function returned error.
160511 0:39:42 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160511 0:39:42 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
160511 0:39:42 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
160511 0:39:42 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
160511 00:39:44 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
160511 17:11:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
160511 17:11:08 [Note] /usr/libexec/mysqld (mysqld 5.5.47-MariaDB) starting as process 24894 ...
160511 17:11:08 InnoDB: The InnoDB memory heap is disabled
160511 17:11:08 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160511 17:11:08 InnoDB: Compressed tables use zlib 1.2.7
160511 17:11:08 InnoDB: Using Linux native AIO
160511 17:11:08 InnoDB: Initializing buffer pool, size = 128.0M
160511 17:11:08 InnoDB: Completed initialization of buffer pool
160511 17:11:08 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
160511 17:11:08 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
160511 17:11:08 InnoDB: Waiting for the background threads to start
160511 17:11:09 Percona XtraDB (http://www.percona.com) 5.5.46-MariaDB-37.6 started; log sequence number 8239814
160511 17:11:09 [Note] Plugin 'FEEDBACK' is disabled.
160511 17:11:09 [Note] Server socket created on IP: '0.0.0.0'.
160511 17:11:09 [Note] Event Scheduler: Loaded 0 events
160511 17:11:09 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.47-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
160511 17:12:21 mysqld_safe Number of processes running now: 0
160511 17:12:21 mysqld_safe mysqld restarted
160511 17:12:21 [Note] /usr/libexec/mysqld (mysqld 5.5.47-MariaDB) starting as process 25104 ...
160511 17:12:21 InnoDB: The InnoDB memory heap is disabled
160511 17:12:21 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160511 17:12:21 InnoDB: Compressed tables use zlib 1.2.7
160511 17:12:21 InnoDB: Using Linux native AIO
160511 17:12:21 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137756672 bytes) failed; errno 12
160511 17:12:21 InnoDB: Completed initialization of buffer pool
160511 17:12:21 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160511 17:12:21 [ERROR] Plugin 'InnoDB' init function returned error.
160511 17:12:21 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160511 17:12:21 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
160511 17:12:21 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
160511 17:12:21 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
160511 17:12:21 [Note] Plugin 'FEEDBACK' is disabled.
160511 17:12:21 [ERROR] Unknown/unsupported storage engine: InnoDB
160511 17:12:21 [ERROR] Aborting
160511 17:12:21 [Note] /usr/libexec/mysqld: Shutdown complete
160511 17:12:21 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
I always see there are a lot failed login attempt when I ssh to my server, and you can see there are many ip in log, is attackers caused this problem or my server has been attacked? ?
If not, what happend to my server?
Please check your memory allocation on server.
Real cause of this problem in error log lines below:
160511 17:12:21 [ERROR] mysqld: Out of memory (Needed 128917504 bytes)
160511 17:12:21 [ERROR] mysqld: Out of memory (Needed 96681984 bytes)
160511 17:12:21 [ERROR] mysqld: Out of memory (Needed 72499200 bytes)
Recently My production server was corrupted due to maintenance shutdown by digital ocean. When we trying to recover our database we couldn't access our droplet. Somehow we managed to recover these (ibdata1, ib_logfile0, ib_logfile1 and some .frm) files. When i copied this files to my new mysql server(in another machine), I could't start the mysql server. I got this output
sudo service mysql restart
mysql stop/waiting
start: Job failed to start
I checked the log file and i got this error,
151019 8:55:27 [Warning] Using unique option prefix myisam-recover instead of $
151019 8:55:27 [Note] Plugin 'FEDERATED' is disabled.
151019 8:55:27 InnoDB: The InnoDB memory heap is disabled
151019 8:55:27 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151019 8:55:27 InnoDB: Compressed tables use zlib 1.2.8
151019 8:55:27 InnoDB: Using Linux native AIO
151019 8:55:27 InnoDB: Initializing buffer pool, size = 128.0M
151019 8:55:27 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 50331648 bytes
InnoDB: than specified in the .cnf file 0 5242880 bytes!
151019 8:55:27 [ERROR] Plugin 'InnoDB' init function returned error.
151019 8:55:27 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151019 8:55:27 [ERROR] Unknown/unsupported storage engine: InnoDB
151019 8:55:27 [ERROR] Aborting
151019 8:55:27 [Note] /usr/sbin/mysqld: Shutdown complete
151019 8:55:28 [Warning] Using unique option prefix myisam-recover instead of $
151019 8:55:28 [Note] Plugin 'FEDERATED' is disabled.
151019 8:55:28 InnoDB: The InnoDB memory heap is disabled
151019 8:55:28 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151019 8:55:28 InnoDB: Compressed tables use zlib 1.2.8
151019 8:55:28 InnoDB: Using Linux native AIO
151019 8:55:28 InnoDB: Initializing buffer pool, size = 128.0M
151019 8:55:28 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 50331648 bytes
InnoDB: than specified in the .cnf file 0 5242880 bytes!
151019 8:55:28 [ERROR] Plugin 'InnoDB' init function returned error.
151019 8:55:28 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151019 8:55:28 [ERROR] Unknown/unsupported storage engine: InnoDB
151019 8:55:28 [ERROR] Aborting
151019 8:55:28 [Note] /usr/sbin/mysqld: Shutdown complete
151019 8:55:29 [Warning] Using unique option prefix myisam-recover instead of $
151019 8:55:29 [Note] Plugin 'FEDERATED' is disabled.
151019 8:55:29 InnoDB: The InnoDB memory heap is disabled
151019 8:55:29 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151019 8:55:29 InnoDB: Compressed tables use zlib 1.2.8
151019 8:55:29 InnoDB: Using Linux native AIO
151019 8:55:29 InnoDB: Initializing buffer pool, size = 128.0M
151019 8:55:29 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 50331648 bytes
InnoDB: than specified in the .cnf file 0 5242880 bytes!
151019 8:55:29 [ERROR] Plugin 'InnoDB' init function returned error.
151019 8:55:29 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151019 8:55:29 [ERROR] Unknown/unsupported storage engine: InnoDB
151019 8:55:29 [ERROR] Aborting
151019 8:55:29 [Note] /usr/sbin/mysqld: Shutdown complete
I tried adding innodb_log_file_size= 50331648 as suggested by #akuzminsky
Then I got this error log.
151022 6:54:13 [Warning] Using unique option prefix myisam-recover instead of $
151022 6:54:13 [Note] Plugin 'FEDERATED' is disabled.
151022 6:54:13 InnoDB: The InnoDB memory heap is disabled
151022 6:54:13 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151022 6:54:13 InnoDB: Compressed tables use zlib 1.2.8
151022 6:54:13 InnoDB: Using Linux native AIO
151022 6:54:13 InnoDB: Initializing buffer pool, size = 128.0M
151022 6:54:13 InnoDB: Completed initialization of buffer pool
151022 6:54:13 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 15379293
151022 6:54:13 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: space header page consists of zero bytes in file ./toucan/ormcli$
InnoDB: Error: space header page consists of zero bytes in file ./toucan/catego$
InnoDB: Error: space header page consists of zero bytes in file ./toucan/survey$
InnoDB: Error: space header page consists of zero bytes in file ./toucan/compla$
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 15382291
InnoDB: Page directory corruption: infimum not pointed to
151022 6:54:13 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex 10cfd0d80000000000000000000000000000000000e9e915000800000000001$
InnoDB: End of page dump
151022 6:54:13 InnoDB: Page checksum 282054872, prior-to-4.0.14-form checksum$
InnoDB: stored checksum 282054872, prior-to-4.0.14-form stored checksum 3468306$
InnoDB: Page lsn 0 15329557, low 4 bytes of lsn at page end 15329557
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a file space header page
10:54:13 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346701 K $
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x20)[0x7fa4d90e2f10]
/usr/sbin/mysqld(handle_fatal_signal+0x3d5)[0x7fa4d8fcd175]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fa4d7d5d340]
/usr/sbin/mysqld(+0x5f058a)[0x7fa4d91e258a]
/usr/sbin/mysqld(+0x56c44e)[0x7fa4d915e44e]
/usr/sbin/mysqld(+0x576137)[0x7fa4d9168137]
/usr/sbin/mysqld(+0x5a5632)[0x7fa4d9197632]
/usr/sbin/mysqld(+0x5a6699)[0x7fa4d9198699]
/usr/sbin/mysqld(+0x591ad3)[0x7fa4d9183ad3]
/usr/sbin/mysqld(+0x542a1a)[0x7fa4d9134a1a]
/usr/sbin/mysqld(+0x511d48)[0x7fa4d9103d48]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x41)[0x7fa4d8fcf$
/usr/sbin/mysqld(+0x302221)[0x7fa4d8ef4221]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x92a)[0x7fa4d8ef821a]
/usr/sbin/mysqld(+0x28ac5b)[0x7fa4d8e7cc5b]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x45b)[0x7fa4d8e815db]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fa4d739fec5]
/usr/sbin/mysqld(+0x286b18)[0x7fa4d8e78b18]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Please help me to resolve this.
Thanks in advance.
Note lines in the error log:
InnoDB: Error: log file ./ib_logfile0 is of different size 0 50331648 bytes
InnoDB: than specified in the .cnf file 0 5242880 bytes!
The actual size of the redo log file differs from the size in MySQL config.
Add this line to my.cnf
innodb_log_file_size= 50331648
If the tablespace is corrupted you'll still have to heal it. But at least MySQL won't stop at this step.