Less than 2 weeks ago I posted a question here about a problem in my OpenShift cartridge because I suddenly couldn't restart it anymore: Error restarting Openshift MySQL cartridge
It was working fine and I didn't change anything on it but it simply stopped working. The only way I could "solve" the problem was deleting the gear and creating a new one, but now I have the same problem again :-(
This is really frustrating. I thought OpenShift was a realiable PaaS, but it looks like there's still a long way to go. I considering to buy the Enterprise plan, but I don't to pay for something that stops working out of nowhere.
By the way, here is the error log in case someone knows this time why this is happening:
Starting MySQL 5.5 cartridge
MySQL server failed to start:
140316 18:02:14 mysqld_safe Logging to '/var/lib/openshift/53104e375973cad496000022/mysql//log/mysql_error.log'.
140316 18:02:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/53104e375973cad496000022/mysql/data/
140316 18:02:17 mysqld_safe mysqld from pid file /var/lib/openshift/53104e375973cad496000022/mysql/pid/mysql.pid ended
140316 18:02:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/53104e375973cad496000022/mysql/data/
140316 18:02:15 [Note] Plugin 'FEDERATED' is disabled.
140316 18:02:15 InnoDB: The InnoDB memory heap is disabled
140316 18:02:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140316 18:02:15 InnoDB: Compressed tables use zlib 1.2.3
140316 18:02:15 InnoDB: Using Linux native AIO
140316 18:02:15 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
InnoDB: Warning: io_setup() attempt 1 failed.
InnoDB: Warning: io_setup() attempt 2 failed.
InnoDB: Warning: io_setup() attempt 3 failed.
InnoDB: Warning: io_setup() attempt 4 failed.
InnoDB: Warning: io_setup() attempt 5 failed.
140316 18:02:17 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
140316 18:02:17 InnoDB: Fatal error: cannot initialize AIO sub-system
140316 18:02:17 [ERROR] Plugin 'InnoDB' init function returned error.
140316 18:02:17 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140316 18:02:17 [ERROR] Unknown/unsupported storage engine: InnoDB
140316 18:02:17 [ERROR] Aborting
140316 18:02:17 [Note] /opt/rh/mysql55/root/usr/libexec/mysqld: Shutdown complete
140316 18:02:17 mysqld_safe mysqld from pid file /var/lib/openshift/53104e375973cad496000022/mysql/pid/mysql.pid ended
Failed to execute: 'control restart' for /var/lib/openshift/53104e375973cad496000022/mysql
Thanks!
Looks like you are having the same issue as this user: Can't add a MySQL cartridge on OpenShift
It's a mysql bug we are working on a work around.
Related
I have installed mysql server on ec2 AMI. Suddenly stopped working mysql server and its not getting started.
Here is mysql log :
160629 4:28:37 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 3059 ...
160629 4:28:37 [Note] Plugin 'FEDERATED' is disabled.
160629 4:28:37 InnoDB: The InnoDB memory heap is disabled
160629 4:28:37 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160629 4:28:37 InnoDB: Compressed tables use zlib 1.2.8
160629 4:28:37 InnoDB: Using Linux native AIO
160629 4:28:37 InnoDB: Initializing buffer pool, size = 512.0M
160629 4:28:37 InnoDB: Completed initialization of buffer pool
160629 4:28:37 InnoDB: highest supported file format is Barracuda.
InnoDB: No valid checkpoint found.
InnoDB: If this error appears when you are creating an InnoDB database,
InnoDB: the problem may be that during an earlier attempt you managed
InnoDB: to create the InnoDB data files, but log file creation failed.
InnoDB: If that is the case, please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/error-creating-innodb.html
160629 4:28:37 [ERROR] Plugin 'InnoDB' init function returned error.
160629 4:28:37 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160629 4:28:37 [ERROR] Unknown/unsupported storage engine: InnoDB
160629 4:28:37 [ERROR] Aborting
160629 4:28:37 [Note] /usr/libexec/mysql55/mysqld: Shutdown complete
160629 04:28:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Here are approach what I have tried yet:
1) Upgraded EC2 t2.micro to t2.medium.
2) Increased innodb_buffer_pool_size in my.cnf
3) Rename ib_logfile0 and ib_logfile1
Please suggest what next step should I do to start mysql.
Edit:
Sudo free -m
total used free shared buffers cached
Mem: 3954 149 3805 0 9 86
-/+ buffers/cache: 53 3901
Edit:
Is there only solution to uninstall mysql and re-install ?
If yes, then how can retain already created databases in mysql? Please Suggest.
Please go through the below link which is being given in your error logs.
It has mentioned certain steps to solve runtime exception .You can cross check and proceed. If nothing helped than you can opt for uninstallation.
http://dev.mysql.com/doc/refman/5.5/en/error-creating-innodb.html
I have an Amazon ec2 micro instance that holds a WordPress site. Since yesterday, my mysql db keep crashing saying:
[ec2-user#ip-xxx-xx-xx-xxx ~]$ sudo service mysqld status
mysqld dead but subsys locked
I tried restarting the mysqld but after every 5 mins it keeps crashing with the same message - "Error establishing a database connection"
This is what my DB log file reads -
160123 05:44:41 mysqld_safe mysqld restarted
160123 5:44:43 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.46) starting as process 7928 ...
160123 5:44:44 [Note] Plugin 'FEDERATED' is disabled.
160123 5:44:46 InnoDB: The InnoDB memory heap is disabled
160123 5:44:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
160123 5:44:46 InnoDB: Compressed tables use zlib 1.2.8
160123 5:44:46 InnoDB: Using Linux native AIO
160123 5:44:46 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160123 5:44:46 InnoDB: Completed initialization of buffer pool
160123 5:44:46 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160123 5:44:46 [ERROR] Plugin 'InnoDB' init function returned error.
160123 5:44:46 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160123 5:44:46 [ERROR] Unknown/unsupported storage engine: InnoDB
160123 5:44:46 [ERROR] Aborting
160123 5:44:46 [Note] /usr/libexec/mysql55/mysqld: Shutdown complete
160123 05:44:46 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
The common solution that found out there when I Googled for a solution for the issue - mysqld dead but subsys locked, is to create a swapfile. I did exactly as most of the solution suggest but since I'm not well versed with UNIX, I don't know what I have done is right or not.
Or, Is there any other solution for this issue. Kindly help.
Have you tried this mysql is dead but subsys locked
Seems quite straightforward. Also please consider moving to RDS instead of running a mysql server in ec2. Saves you from troubles of this kind.
150505 16:57:01 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150505 16:58:01 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150505 16:58:01 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150505 16:58:01 [Note] Plugin 'FEDERATED' is disabled.
150505 16:58:01 InnoDB: The InnoDB memory heap is disabled
150505 16:58:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150505 16:58:01 InnoDB: Compressed tables use zlib 1.2.8
150505 16:58:01 InnoDB: Using Linux native AIO
150505 16:58:01 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150505 16:58:01 InnoDB: Completed initialization of buffer pool
150505 16:58:01 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150505 16:58:01 [ERROR] Plugin 'InnoDB' init function returned error.
150505 16:58:01 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150505 16:58:01 [ERROR] Unknown/unsupported storage engine: InnoDB
150505 16:58:01 [ERROR] Aborting
Above is mysql error log, sometime mysql will just crash, and I want to do a restart on the service, it will not able to start,the error is something like
the "unknown instance" error.
Anyone got any idea what the error above is about and how to solve it, I tried google but seems to not able find the answer.
Thanks!
I found this question searching for an answer myself, my server's /var/log/mysql/error.log had very similar statements to yours. Running dmesg | tail -20 revealed the culprit: the OS ran out of memory and killed mysqld:
[835661.446277] Out of memory: Kill process 1024 (mysqld) score 91 or sacrifice child
[835661.446912] Killed process 1024 (mysqld) total-vm:901544kB, anon-rss:92332kB, file-rss:0kB
[835661.488337] init: mysql main process (1024) killed by KILL signal
Now to figure out why memory ran out, on this newer server, with twice as much RAM as the one it's replacing, which never did this...
I went to spin up a new application today and I'm running into some issues. I was able to add the PHP 5.4 cartridge just fine. I went to add a MySQL 5.5 cartridge and I received the following:
Starting MySQL 5.5 cartridge
MySQL server failed to start:
140320 03:05:46 mysqld_safe Logging to '/var/lib/openshift/532a883a4382ec501600043b/mysql//log/mysql_error.log'.
140320 03:05:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/532a883a4382ec501600043b/mysql/data/
140320 03:05:51 mysqld_safe mysqld from pid file /var/lib/openshift/532a883a4382ec501600043b/mysql/pid/mysql.pid ended
140320 03:05:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/openshift/532a883a4382ec501600043b/mysql/data/
140320 3:05:48 [Note] Plugin 'FEDERATED' is disabled.
140320 3:05:48 InnoDB: The InnoDB memory heap is disabled
140320 3:05:48 InnoDB: Mutexes and rw_locks use GCC atomic builtins
140320 3:05:48 InnoDB: Compressed tables use zlib 1.2.3
140320 3:05:48 InnoDB: Using Linux native AIO
140320 3:05:48 InnoDB: Warning: io_setup() failed with EAGAIN. Will make 5 attempts before giving up.
InnoDB: Warning: io_setup() attempt 1 failed.
InnoDB: Warning: io_setup() attempt 2 failed.
InnoDB: Warning: io_setup() attempt 3 failed.
InnoDB: Warning: io_setup() attempt 4 failed.
InnoDB: Warning: io_setup() attempt 5 failed.
140320 3:05:51 InnoDB: Error: io_setup() failed with EAGAIN after 5 attempts.
InnoDB: You can disable Linux Native AIO by setting innodb_use_native_aio = 0 in my.cnf
140320 3:05:51 InnoDB: Fatal error: cannot initialize AIO sub-system
140320 3:05:51 [ERROR] Plugin 'InnoDB' init function returned error.
140320 3:05:51 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
140320 3:05:51 [ERROR] Unknown/unsupported storage engine: InnoDB
140320 3:05:51 [ERROR] Aborting
140320 3:05:51 [Note] /opt/rh/mysql55/root/usr/libexec/mysqld: Shutdown complete
140320 03:05:51 mysqld_safe mysqld from pid file /var/lib/openshif /532a883a4382ec501600043b/mysql/pid/mysql.pid ended
Failed to execute: 'control start' for /var/lib/openshift/532a883a4382ec501600043b/mysql
I've seen one other question that seems similar, but was when attempting to restart their app.
Any idea how to fix this problem?
Looks like there is a bug (http://elenst.ru/mariadb/aio-max-nr-in-general-and-innodb-error-io_setup-failed-with-eagain-in-particular/) (http://bugs.mysql.com/bug.php?id=61575) we are looking into work arounds
I tried to restart mysql. However it did not restart. Here is the content of mysql.log:
131217 17:01:09 InnoDB: Starting shutdown...
131217 17:01:11 InnoDB: Shutdown completed; log sequence number 153229469
131217 17:01:11 [Note] /usr/libexec/mysqld: Shutdown complete
131217 17:01:11 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
131217 17:01:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
131217 17:01:11 [Note] Plugin 'FEDERATED' is disabled.
131217 17:01:11 InnoDB: The InnoDB memory heap is disabled
131217 17:01:11 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131217 17:01:11 InnoDB: Compressed tables use zlib 1.2.3
131217 17:01:11 InnoDB: Using Linux native AIO
^G/usr/libexec/mysqld: Can't create/write to file '/tmp/ibsMvrkL' (Errcode: 13)
131217 17:01:11 InnoDB: Error: unable to create temporary file; errno: 13
131217 17:01:11 [ERROR] Plugin 'InnoDB' init function returned error.
131217 17:01:11 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
131217 17:01:11 [ERROR] Unknown/unsupported storage engine: InnoDB
131217 17:01:11 [ERROR] Aborting
Can any one help me to find the reason of this? In fact mysql was working before this time and it seems that it is not a regular error!!Please let me know if any other information is nedeed.
Thanks a lot
Have you checked if there is a problem about your disk partition. May be its out of space or have a problem. The error clearly shows the problem is caused by creating temp file failure. Rest is listed because InnoDB failed to initialize.
This error clearly mentioned that MySQL daemon unable to create a temporary file under /tmp directory.
Please check the permission of /tmp directory.
Your /tmp directory permission should have a sticky bit. below is the example for same.
drwxrwxrwt 3 root root 4.0K Feb 4 13:48 tmp