I'm using mysqld Ver 5.5.22-0ubuntu1 for debian-linux-gnu on x86_64 ((Ubuntu)) under Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-24-generic x86_64). Updates are current.
I had two events that were working fine, and then a power outage knocked the server out. Since then, mysql will not start if event_scheduler=ENABLED in /etc/mysql/my.cnf:
# service mysql restart
mysql stop/waiting
start: Job failed to start
I dropped both events and the event table is empty.
/var/log/mysql.err is completely empty.
There must be some events queued up somewhere but I can't seem to find them. Any assistance would be greatly appreciated.
Edit: /var/log/mysql/error.log doesn't give a whole lot more information:
130624 11:31:46 [Note] /usr/sbin/mysqld: Normal shutdown
130624 11:31:46 [Note] Event Scheduler: Purging the queue. 0 events
130624 11:31:48 [Warning] /usr/sbin/mysqld: Forcing close of thread 475 user: 'root'
130624 11:31:48 InnoDB: Starting shutdown...
130624 11:31:49 InnoDB: Shutdown completed; log sequence number 7222402477
130624 11:31:49 [Note] /usr/sbin/mysqld: Shutdown complete
Here's syslog for the same time period:
Jun 24 11:31:49 earth kernel: [4569993.677493] type=1400 audit(1372098709.303:19): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/run/mysqld/mysqld.pid" pid=20660 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=102 ouid=102
Jun 24 11:31:49 earth kernel: [4569993.697382] type=1400 audit(1372098709.323:20): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=20666 comm="apparmor_parser"
Jun 24 11:31:49 earth kernel: [4569993.722622] init: mysql main process (20670) terminated with status 1
Jun 24 11:31:49 earth kernel: [4569993.722650] init: mysql main process ended, respawning
Jun 24 11:31:50 earth kernel: [4569994.727981] init: mysql post-start process (20671) terminated with status 1
Jun 24 11:31:50 earth kernel: [4569994.737481] type=1400 audit(1372098710.363:21): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=20694 comm="apparmor_parser"
Jun 24 11:31:50 earth kernel: [4569994.762575] init: mysql main process (20698) terminated with status 1
Jun 24 11:31:50 earth kernel: [4569994.762602] init: mysql main process ended, respawning
Jun 24 11:31:51 earth kernel: [4569995.767984] init: mysql post-start process (20699) terminated with status 1
Jun 24 11:31:51 earth kernel: [4569995.777498] type=1400 audit(1372098711.403:22): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=20722 comm="apparmor_parser"
Jun 24 11:31:51 earth kernel: [4569995.802326] init: mysql main process (20726) terminated with status 1
Jun 24 11:31:51 earth kernel: [4569995.802354] init: mysql respawning too fast, stopped
On newer debian installs the mysql error log goes to syslog by default. Check in /var/log/ there should be mysqld startup logs and maybe some errors related to the mysql.event table.
Related
I am running a AWS Lightsail instance with Ubuntu 18.04 and latest MySQL. Since a few days, the MySQL service stops at random times and then forces me to perform a manual restart after which everything goes back to normal for 24-48 hours until I face another stop.
Terminal echoes this error when running sudo service mysql status:
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2020-04-26 17:54:44 UTC; 10h ago
Process: 30117 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Process: 30095 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 14182 (code=killed, signal=KILL)
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Failed to start MySQL Community Server.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Scheduled restart job, restart counter is at 8.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Stopped MySQL Community Server.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Start request repeated too quickly.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: mysql.service: Failed with result 'exit-code'.
Apr 26 17:54:44 ip-172-26-4-241 systemd[1]: Failed to start MySQL Community Server.
After digging through the /var/log/mysql/error.log file, I found dozens of lines like this:
2020-04-26T10:10:46.710146Z 2 [ERROR] InnoDB: innodb_force_recovery is on. We do not allow database modifications by the user. Shut down mysqld and edit my.cnf to$
Since the error message is truncated at exactly the wrong spot, I did some more digging through other Warning messages and found this one:
2020-04-26T17:54:44.312663Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2020-04-26T17:54:44.312671Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2020-04-26T17:54:44.312675Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-04-26T17:54:44.312680Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-04-26T17:54:44.312684Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-04-26T17:54:44.312688Z 0 [ERROR] Failed to initialize builtin plugins.
2020-04-26T17:54:44.312691Z 0 [ERROR] Aborting
I am by no means a MySQL expert, but I interpret that mmamp error as the system running out of space, which is a little weird since the instance uses around 10% space and has at most 2 users at the same time (hence little to no CPU usage). I run a single site Wordpress installation (just to divert users) and phpmyadmin plus some minor testing scripts.
Appreciate any help with this, thank you.
The most likely cause is that your configuration is broken in a way that makes mysqld try to allocate more memory that your server has.
I am using MySQL Percona XtraDB Cluster with 3 node and MySQL service in one node did not work because of disk full. after fix the issue we try to start MySQL on this node but the following error occur: (before this issue all node had been working fine)
$ sudo /etc/init.d/mysql start
[....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
failed!
$ sudo systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon
Loaded: loaded (/etc/init.d/mysql; generated)
Drop-In: /etc/systemd/system/mysql.service.d
└─override.conf
Active: failed (Result: exit-code) since Tue 2020-03-24 11:57:55 +0430; 1h 2min ago
Docs: man:systemd-sysv-generator(8)
Process: 19397 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
Mar 24 11:57:47 server-3 systemd[1]: Starting LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon...
Mar 24 11:57:47 server-3 mysql[19397]: * Starting MySQL (Percona XtraDB Cluster) database server mysqld
Mar 24 11:57:55 server-3 mysql[19397]: * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Mar 24 11:57:55 server-3 mysql[19397]: ...fail!
Mar 24 11:57:55 server-3 systemd[1]: mysql.service: Control process exited, code=exited status=1
Mar 24 11:57:55 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 11:57:55 server-3 systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
Operating System: Ubuntu 18.04.2 LTS
mysql Ver 14.14 Distrib 5.7.25-28, for debian-linux-gnu (x86_64) using 7.0
also please find journalctl log in below:
$ sudo journalctl -xe
Journal file /var/log/journal/7e072ef87bdf452f8a9684d905f87daf/user-1863206337.journal is truncated, ignoring file.
Mar 24 19:05:22 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:22.235+04:30] time="2020-03-24T19:05:22+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:22 server-3 mysql[6820]: * The server quit without updating PID file (/var/run/mysqld/mysqld.pid).
Mar 24 19:05:22 server-3 mysql[6820]: ...fail!
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Control process exited, code=exited status=1
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: mysql.service: Failed with result 'exit-code'.
Mar 24 19:05:22 server-3 systemd[1]: Failed to start LSB: Start and stop the mysql (Percona XtraDB Cluster) daemon.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mysql.service has failed.
--
-- The result is RESULT.
Mar 24 19:05:22 server-3 sudo[6779]: pam_unix(sudo:session): session closed for user root
Mar 24 19:05:22 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:22 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:22 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:22 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:22 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:23 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:54620->[192.168.8.3]:161
Mar 24 19:05:23 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:54647->[192.168.8.3]:161
Mar 24 19:05:25 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:25.550+04:30] time="2020-03-24T19:05:25+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:26 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:55485->[192.168.8.3]:161
Mar 24 19:05:26 server-3 snmpd[1805]: Connection from UDP: [192.168.2.10]:55500->[192.168.8.3]:161
Mar 24 19:05:27 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:27.235+04:30] time="2020-03-24T19:05:27+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:27 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:27 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:27 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:27 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:27 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:30 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:30.550+04:30] time="2020-03-24T19:05:30+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:31 server-3 pmm-agent[30366]: ERRO[2020-03-24T19:05:31.438+04:30] cannot select ##slow_query_log_file: dial tcp 127.0.0.1:3306: connect: connection
Mar 24 19:05:32 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:32.234+04:30] time="2020-03-24T19:05:32+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:32 server-3 kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20170831/exfield-427)
Mar 24 19:05:32 server-3 kernel: No Local Variables are initialized for Method [_PMM]
Mar 24 19:05:32 server-3 kernel: No Arguments are initialized for method [_PMM]
Mar 24 19:05:32 server-3 kernel: ACPI Error: Method parse/execution failed \_SB.PMI0._PMM, AE_AML_BUFFER_LIMIT (20170831/psparse-550)
Mar 24 19:05:32 server-3 kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20170831/power_meter-338)
Mar 24 19:05:35 server-3 pmm-agent[30366]: INFO[2020-03-24T19:05:35.550+04:30] time="2020-03-24T19:05:35+04:30" level=error msg="Error pinging mysqld: dial tcp 1
Mar 24 19:05:36 server-3 sudo[7805]: admin_user : problem with defaults entries ; TTY=pts/0 ; PWD=/etc/systemd/system/mysql.service.d ; USER=root ;
Mar 24 19:05:36 server-3 sudo[7805]: admin_user : TTY=pts/0 ; PWD=/etc/systemd/system/mysql.service.d ; USER=root ; COMMAND=/bin/journalctl -xe
Mar 24 19:05:36 server-3 sudo[7805]: pam_unix(sudo:session): session opened for user root by admin_user(uid=0)
and please find mysqld.log in below:
/var/log$ sudo tail -n50 mysqld.log
2020-03-23T20:44:32.844540Z 36 [ERROR] WSREP: Node consistency compromised, aborting...
2020-03-23T20:44:32.844642Z 13 [ERROR] WSREP: Failed to apply trx: source: 89d92e92-6638-11e9-bdcb-7f2c26717188 version: 4 local: 0 state: APPLYING flags: 1 conn_id: 26113114 trx_id: 5580550846 seqnos (l: 151018291, g: 793430465, s: 793430462, d: 793430355, ts: 33700460567224654)
2020-03-23T20:44:32.844671Z 13 [ERROR] WSREP: Failed to apply trx 793430465 4 times
2020-03-23T20:44:32.844683Z 13 [ERROR] WSREP: Node consistency compromised, aborting...
2020-03-23T20:44:33.255602Z 21 [Note] WSREP: (c0cb4c75, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.8.1:4567 tcp://192.168.8.2:4567
2020-03-23T20:44:37.817120Z 21 [Note] WSREP: declaring node with index 0 suspected, timeout PT5S (evs.suspect_timeout)
2020-03-23T20:44:37.817139Z 21 [Note] WSREP: declaring node with index 1 suspected, timeout PT5S (evs.suspect_timeout)
2020-03-23T20:44:37.817147Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspecting node: 82cab7cb
2020-03-23T20:44:37.817152Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspected node without join message, declaring inactive
2020-03-23T20:44:37.817158Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspecting node: 89d92e92
2020-03-23T20:44:37.817161Z 21 [Note] WSREP: evs::proto(c0cb4c75, LEAVING, view_id(REG,82cab7cb,31)) suspected node without join message, declaring inactive
2020-03-23T20:44:37.817182Z 21 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,82cab7cb,31)
memb {
c0cb4c75,0
}
joined {
}
left {
}
partitioned {
82cab7cb,0
89d92e92,0
}
)
2020-03-23T20:44:37.817203Z 21 [Note] WSREP: Current view of cluster as seen by this node
view ((empty))
2020-03-23T20:44:37.817358Z 21 [Note] WSREP: gcomm: closed
2020-03-23T20:44:37.817463Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2020-03-23T20:44:37.817504Z 0 [Note] WSREP: Flow-control interval: [10000, 10000]
2020-03-23T20:44:37.817515Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-03-23T20:44:37.817522Z 0 [Note] WSREP: Received NON-PRIMARY.
2020-03-23T20:44:37.817524Z 13 [ERROR] WSREP: non-standard exception
2020-03-23T20:44:37.817530Z 0 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 793430465)
2020-03-23T20:44:37.817568Z 13 [Note] WSREP: applier thread exiting (code:8)
2020-03-23T20:44:37.817584Z 13 [Note] WSREP: Starting Shutdown
2020-03-23T20:44:37.817587Z 0 [Note] WSREP: Received self-leave message.
2020-03-23T20:44:37.817529Z 36 [ERROR] WSREP: non-standard exception
2020-03-23T20:44:37.817643Z 36 [Note] WSREP: applier thread exiting (code:8)
2020-03-23T20:44:37.817656Z 36 [Note] WSREP: Starting Shutdown
2020-03-23T20:44:37.817759Z 0 [Note] WSREP: Received shutdown signal. Will sleep for 10 secs before initiating shutdown. pxc_maint_mode switched to SHUTDOWN
2020-03-23T20:44:37.817813Z 0 [Note] WSREP: Flow-control interval: [10000, 10000]
2020-03-23T20:44:37.817841Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-03-23T20:44:37.817852Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2020-03-23T20:44:37.817860Z 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 793430465)
2020-03-23T20:44:37.817870Z 0 [Note] WSREP: RECV thread exiting 0: Success
2020-03-23T20:44:37.818185Z 21 [Note] WSREP: recv_thread() joined.
2020-03-23T20:44:37.818192Z 21 [Note] WSREP: Closing replication queue.
2020-03-23T20:44:37.818195Z 21 [Note] WSREP: Closing slave action queue.
2020-03-23T20:44:37.818247Z 21 [Note] WSREP: /usr/sbin/mysqld: Terminated.
How to start MySQL in this node?
thanks in advance.
As your myslqd.log shows, the disk full condition has left your node in a state, where it can find no way to achieve consistency with the cluster.
This implies, that you have to remove the node from the cluster, empty it out and then re-add it like a new node.
I rebooted a digital ocean box and now I can't start mysql. When I run start command i get :
Redirecting to /bin/systemctl restart mysql.service
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Result of systemctl status mysql.service
● mysql.service - Percona XtraDB Cluster
Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2017-12-11 17:08:18 UTC; 5s ago
Process: 26300 ExecStopPost=/usr/bin/mysql-systemd stop-post (code=exited, status=0/SUCCESS)
Process: 26270 ExecStop=/usr/bin/mysql-systemd stop (code=exited, status=2)
Process: 25674 ExecStartPost=/usr/bin/mysql-systemd start-post $MAINPID (code=exited, status=1/FAILURE)
Process: 25673 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 25632 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=0/SUCCESS)
Main PID: 25673 (code=exited, status=0/SUCCESS)
Dec 11 17:08:18 server-name-hidden mysql-systemd[25674]: ERROR! mysqld_safe with PID 25673 has already exited: FAILURE
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service: control process exited, code=exited status=1
Dec 11 17:08:18 server-name-hidden mysql-systemd[26270]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:08:18 server-name-hidden mysql-systemd[26270]: ERROR! mysql already dead
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service: control process exited, code=exited status=2
Dec 11 17:08:18 server-name-hidden mysql-systemd[26300]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:08:18 server-name-hidden mysql-systemd[26300]: WARNING: mysql may be already dead
Dec 11 17:08:18 server-name-hidden systemd[1]: Failed to start Percona XtraDB Cluster.
Dec 11 17:08:18 server-name-hidden systemd[1]: Unit mysql.service entered failed state.
Dec 11 17:08:18 server-name-hidden systemd[1]: mysql.service failed.
/var/run/mysqld/ is owned by mysql user - but it's empty. If i add a mysqld.pid file it gets removed when i run mysql start.
Does anyone know why a reboot would cause this or give me any next steps. I have reviewed the mysqld.log file and can't see anything of use. here are the last 30 lines
017-12-11T15:14:42.473164Z 0 [Note] WSREP: Received shutdown signal. Will sleep for 10 secs before initiating shutdown. pxc_maint_mode switched to SHUTDOWN
2017-12-11T15:14:52.474584Z 0 [Note] WSREP: Stop replication
2017-12-11T15:14:52.474691Z 0 [Note] WSREP: Closing send monitor...
2017-12-11T15:14:52.475160Z 0 [Note] WSREP: Closed send monitor.
2017-12-11T15:14:52.475268Z 0 [Note] WSREP: gcomm: terminating thread
2017-12-11T15:14:52.475305Z 0 [Note] WSREP: gcomm: joining thread
2017-12-11T15:14:52.475886Z 0 [Note] WSREP: gcomm: closing backend
2017-12-11T15:14:52.476079Z 0 [Note] WSREP: Current view of cluster as seen by this node
2017-12-11T15:14:52.476491Z 0 [Note] WSREP: gcomm: closed
2017-12-11T15:14:52.476532Z 0 [Note] WSREP: Received self-leave message.
2017-12-11T15:14:52.476559Z 0 [Note] WSREP: Flow-control interval: [0, 0]
2017-12-11T15:14:52.476595Z 0 [Note] WSREP: Trying to continue unpaused monitor
2017-12-11T15:14:52.476602Z 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2017-12-11T15:14:52.476608Z 0 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 13842228)
2017-12-11T15:14:52.476632Z 0 [Note] WSREP: RECV thread exiting 0: Success
2017-12-11T15:14:52.477098Z 0 [Note] WSREP: recv_thread() joined.
2017-12-11T15:14:52.477110Z 0 [Note] WSREP: Closing replication queue.
2017-12-11T15:14:52.477116Z 0 [Note] WSREP: Closing slave action queue.
2017-12-11T15:14:52.477123Z 0 [Note] Giving 63 client threads a chance to die gracefully
2017-12-11T15:14:52.478905Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 21 user: ‘hidden’
2017-12-11T15:14:52.478957Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 22 user: 'hidden'
2017-12-11T15:14:52.478994Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6387 user: 'hidden'
2017-12-11T15:14:52.479034Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6367 user: 'hidden'
2017-12-11T15:14:52.479084Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6373 user: 'hidden'
2017-12-11T15:14:52.479130Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6368 user: 'hidden'
2017-12-11T15:14:54.479289Z 0 [Note] WSREP: Waiting for active wsrep applier to exit
I've also been told to try : systemctl start mysql#bootstrap however this fails with the same error. here is the result from journalctl -xe
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service: control process exited, code=exited status=1
Dec 11 17:24:58 dropletname mysql-systemd[28663]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:24:58 dropletname mysql-systemd[28663]: ERROR! mysql already dead
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service: control process exited, code=exited status=2
Dec 11 17:24:58 dropletname mysql-systemd[28694]: WARNING: mysql pid file /var/run/mysqld/mysqld.pid empty or not readable
Dec 11 17:24:58 dropletname mysql-systemd[28694]: WARNING: mysql may be already dead
Dec 11 17:24:58 dropletname systemd[1]: Failed to start Percona XtraDB Cluster with config /etc/sysconfig/mysql.bootstrap.
Dec 11 17:24:58 dropletname systemd[1]: Unit mysql#bootstrap.service entered failed state.
Dec 11 17:24:58 dropletname systemd[1]: mysql#bootstrap.service failed.
Dec 11 17:24:58 dropletname polkitd[510]: Unregistered Authentication Agent for unix-process:28009:649814 (system bus name :1.159, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)
Dec 11 17:25:02 dropletname sendmail[27970]: unable to qualify my own domain name (dropletname) -- using short name
Dec 11 17:25:02 dropletname sendmail[27970]: vBBHP2cx027970: from=hidden, size=1655, class=-60, nrcpts=1, msgid=<201712111725.vBBHP2cx027970#dropletname>, relay=hidden#localhost
Dec 11 17:25:02 dropletname sendmail[28728]: vBBHP26v028728: from=<hidden#dropletname>, size=1935, class=-60, nrcpts=1, msgid=<201712111725.vBBHP2cx027970#dropletname>, proto=ESMTP, daemon=MTA, relay=dropletname [127.0.0.1]
Dec 11 17:25:02 dropletname sendmail[27970]: vBBHP2cx027970: to=hidden, ctladdr=hidden (1000/1000), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=139655, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (vBBHP26v028728 Message accepted for del
Dec 11 17:25:02 dropletname sendmail[28729]: vBBHP26v028728: to=<hidden#dropletname>, ctladdr=<hidden#dropletname> (1000/1000), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=140148, dsn=2.0.0, stat=Sent
Dec 11 17:25:02 dropletname systemd[1]: Removed slice User Slice of hidden
ps aux|grep mysql
root 11590 0.0 0.0 107924 608 pts/0 T 15:43 0:00 cat /var/log/mysqld.log
root 24666 0.0 0.0 107924 612 pts/0 T 16:52 0:00 cat /var/log/mysqld.log
root 32182 0.0 0.0 112664 972 pts/2 S+ 17:59 0:00 grep --color=auto mysq
I managed to resolve this in the end however i'm not sure what 100% did it. I changed /etc/my.conf to look like this :
!includedir /etc/my.cnf.d/
!includedir /etc/percona-xtradb-cluster.conf.d/
[client]
socket=/var/lib/mysql/mysql.sock
[mysqld]
server-id=1
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-bin
log_slave_updates
expire_logs_days=7
innodb_strict_mode=OFF
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
I created a mysqld.pid and mysqld.lock file in /var/run/mysqld/ ensuring the owner and group was mysql:mysql
touch /var/run/mysqld/mysqld.pid;
touch /var/run/mysqld/mysqld.lock;
chown -R mysql:mysql /var/run/mysqld/;
I then ran the following two commands which seems to fail but may have helped
service mysql start --wsrep-cluster-address="gcomm://";
systemctl start mysql --wsrep-new-cluster;
Finally I ran
systemctl start mysql#bootstrap.service
service mysql start
This had previously failed every time I tried. Subsequently I then found that the /var/lib/mysql/mysql.sock had been created which previously did not exist.
I've installed Homestead via VirtualBox and Vagrant on macOS Sierra. Everything works, except for some mysterious reason MySQL shuts itself down after the box has been up for longer than 1 minute 42 seconds...
I'm able to use MySQL without issue for the first 1 minute 42 seconds, but then suddenly the service shuts down and refuses further connections.
I tailed /var/log/syslog after the server came up, and the following syslog entries appearing:
Dec 10 20:02:25 homestead systemd[1]: mariadb.service: Start operation timed out. Terminating.
Dec 10 20:02:25 homestead mysqld[1611]: 2016-12-10 20:02:25 140574075632384 [Note] /usr/sbin/mysqld: Normal shutdown
Dec 10 20:02:25 homestead mysqld[1611]: 2016-12-10 20:02:25 140574075632384 [Note] Event Scheduler: Purging the queue. 0 events
Dec 10 20:02:25 homestead kernel: [ 100.520787] audit_printk_skb: 15 callbacks suppressed
Dec 10 20:02:25 homestead kernel: [ 100.520844] audit: type=1400 audit(1481400145.309:17): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=1611 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
Dec 10 20:02:25 homestead mysqld[1611]: 2016-12-10 20:02:25 140573193234176 [Note] InnoDB: FTS optimize thread exiting.
Dec 10 20:02:25 homestead mysqld[1611]: 2016-12-10 20:02:25 140574075632384 [Note] InnoDB: Starting shutdown...
Dec 10 20:02:25 homestead systemd[1]: Started Session 4 of user vagrant.
Dec 10 20:02:26 homestead mysqld[1611]: 2016-12-10 20:02:26 140574075632384 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
Dec 10 20:02:27 homestead mysqld[1611]: 2016-12-10 20:02:27 140574075632384 [Note] InnoDB: Shutdown completed; log sequence number 1355249813
Dec 10 20:02:27 homestead mysqld[1611]: 2016-12-10 20:02:27 140574075632384 [Note] /usr/sbin/mysqld: Shutdown complete
Dec 10 20:02:27 homestead kernel: [ 102.937893] audit: type=1400 audit(1481400147.726:18): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=2030 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
Dec 10 20:02:27 homestead kernel: [ 102.938561] audit: type=1400 audit(1481400147.726:19): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="run/systemd/notify" pid=1611 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=113 ouid=0
Dec 10 20:02:27 homestead systemd[1]: Failed to start MariaDB database server.
Dec 10 20:02:27 homestead systemd[1]: mariadb.service: Unit entered failed state.
Dec 10 20:02:27 homestead systemd[1]: mariadb.service: Failed with result 'timeout'.
Dec 10 20:02:27 homestead systemd[1]: Reached target Multi-User System.
Dec 10 20:02:27 homestead systemd[1]: Starting Notify bootloader that boot was successful...
Dec 10 20:02:27 homestead systemd[1]: Reached target Graphical Interface.
Dec 10 20:02:27 homestead systemd[1]: Starting Update UTMP about System Runlevel Changes...
Dec 10 20:02:27 homestead systemd[1]: Started Update UTMP about System Runlevel Changes.
Dec 10 20:02:27 homestead snap[2063]: Ignoring 'booted' on classic
Dec 10 20:02:27 homestead systemd[1]: Started Notify bootloader that boot was successful.
Dec 10 20:02:27 homestead systemd[1]: Startup finished in 5.989s (kernel) + 1min 36.860s (userspace) = 1min 42.850s.
The first line, where it says mariadb.service, gets logged the moment I get kicked out of MySQL. But I have no idea why.
It looks like the logs are implying MySQL / MariaDB are still trying to start up after the box has been brought up - even though I've been able to connect to them by this point without issue. Then some mystery timeout is hit, and the whole thing shuts down.
Can anyone help me interpret this...? Thanks!
Finally figured it out - AppArmor seems to be causing the problem.
By following the steps outlined here, I was able to resolve the issue, and MySQL no longer shuts down unexpected.
Here is the error i am getting
mariadb.service - MariaDB database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled)
Active: failed (Result: exit-code) since Mon 2014-10-13 14:31:02 CEST; 10s ago
Process: 23609 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
Process: 23608 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
Process: 23506 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Main PID: 23608 (code=exited, status=0/SUCCESS)
Oct 13 14:30:59 localhost.localdomain mariadb-prepare-db-dir[23506]: Support MariaDB development by buying support/new features from
Oct 13 14:30:59 localhost.localdomain mariadb-prepare-db-dir[23506]: SkySQL Ab. You can contact us about this at sales#skysql.com.
Oct 13 14:30:59 localhost.localdomain mariadb-prepare-db-dir[23506]: Alternatively consider joining our community based development effort:
Oct 13 14:30:59 localhost.localdomain mariadb-prepare-db-dir[23506]: http://mariadb.com/kb/en/contributing-to-the-mariadb-project/
Oct 13 14:30:59 localhost.localdomain mysqld_safe[23608]: 141013 14:30:59 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Oct 13 14:30:59 localhost.localdomain mysqld_safe[23608]: 141013 14:30:59 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Oct 13 14:31:02 localhost.localdomain mysqld_safe[23608]: 141013 14:31:02 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Oct 13 14:31:02 localhost.localdomain systemd[1]: mariadb.service: control process exited, code=exited status=1
Oct 13 14:31:02 localhost.localdomain systemd[1]: Failed to start MariaDB database server.
Oct 13 14:31:02 localhost.localdomain systemd[1]: Unit mariadb.service entered failed state.
THis is the mariadb.log file:
Im not really understanding this error, because the sock file has not been created and my server hasn't succesfully started...
141013 14:37:28 [ERROR] Can't start server : Bind on unix socket: Permission denied
141013 14:37:28 [ERROR] Do you already have another mysqld server running on socket: /var/run/mysql/mysql.sock ?
141013 14:37:28 [ERROR] Aborting
141013 14:37:28 InnoDB: Starting shutdown...
141013 14:37:29 InnoDB: Shutdown completed; log sequence number 1597945
141013 14:37:29 [Note] /usr/libexec/mysqld: Shutdown complete
I installed it fresh several times and I can't seem to figure out what the error is in the logs. I am running everything as root. Thanks.