MySQL restarting every 30 minutes on Ubuntu 11.04 - mysql

Question

I'm having an issue where MySQL 5.1.54 is restarting every 30 minutes on Ubuntu 11.04. When this occurs, the following appears in the MySQL log:
111030 12:01:52 [Note] /usr/sbin/mysqld: Normal shutdown
111030 12:01:52 [Note] Event Scheduler: Purging the queue. 0 events
111030 12:01:52 InnoDB: Starting shutdown...
111030 12:01:54 InnoDB: Shutdown completed; log sequence number 0 875122
111030 12:01:54 [Note] /usr/sbin/mysqld: Shutdown complete
111030 12:01:55 [Note] Plugin 'FEDERATED' is disabled.
111030 12:01:55 InnoDB: Initializing buffer pool, size = 256.0M
111030 12:01:55 InnoDB: Completed initialization of buffer pool
111030 12:01:55 InnoDB: Started; log sequence number 0 875122
111030 12:01:55 [Note] Event Scheduler: Loaded 0 events
111030 12:01:55 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.54-1ubuntu4-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
This occurs like clockwork every 30 minutes, so it's obviously some service restarting it.
I have checked the crontab of every user on the system (including system users), and none of them have a crontab setup, as you can see in the output below:
# awk -F: '{print $1}' /etc/passwd | xargs -n 1 -i crontab -u {} -l
no crontab for root
no crontab for daemon
no crontab for bin
no crontab for sys
no crontab for sync
no crontab for games
no crontab for man
no crontab for lp
no crontab for mail
no crontab for news
no crontab for uucp
no crontab for proxy
no crontab for www-data
no crontab for backup
no crontab for list
no crontab for irc
no crontab for gnats
no crontab for nobody
no crontab for libuuid
no crontab for syslog
no crontab for sshd
no crontab for landscape
no crontab for ubuntu
no crontab for statd
no crontab for myproxy
no crontab for condor
no crontab for messagebus
no crontab for avahi
no crontab for joe
no crontab for smmta
no crontab for smmsp
no crontab for postfix
no crontab for deploy
no crontab for mysql
no crontab for redis
My dmesg contains the following each time it is restarted. I'm not an apparmor expert, but I believe this is a normal message obtained each time the MySQL service starts:
[1165328.780405] type=1400 audit(1319976114.984:74): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=31985 comm="apparmor_parser"
Also, here are the contents of the MySQL upstart configuration in /etc/init/mysql.conf:
# MySQL Service
description "MySQL Server"
author "Mario Limonciello <superm1#ubuntu.com>"
start on (net-device-up
and local-filesystems
and runlevel [2345])
stop on runlevel [016]
respawn
env HOME=/etc/mysql
umask 007
# The default of 5 seconds is too low for mysql which needs to flush buffers
kill timeout 300
pre-start script
#Sanity checks
[ -r $HOME/my.cnf ]
[ -d /var/run/mysqld ] || install -m 755 -o mysql -g root -d /var/run/mysqld
/lib/init/apparmor-profile-load usr.sbin.mysqld
LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<4096) }'
end script
exec /usr/sbin/mysqld
post-start script
for i in `seq 1 30` ; do
/usr/bin/mysqladmin --defaults-file="${HOME}"/debian.cnf ping && {
exec "${HOME}"/debian-start
# should not reach this line
exit 2
}
sleep 1
done
exit 1
end script
Any idea what might be causing this? It doesn't cause any problems, other than Monit alerts stating that "PID changed Service mysqld" (I have Monit monitoring mysqld -- but it reports no errors with the mysqld process, other than the fact that every 30 minutes, it has its PID changed since MySQL is restarted).
Thanks in advance.

Answer 1

are you using chef or puppet, which might be doing something that triggers the reboot?

Answer 2

can you check (and probably post) the definition of your mysql job in upstart? (/etc/init/mysql.conf). OK = try removing the "respawn". It does not work as documented in the upstart documentation. Generally it's used to respawn the process if it's killed by other process, but it seems it does not function as expected. You can see why the apparmour is always loading - because of the pre-start stanza in the script.
As upstart is very new, and is still evolving, better is to use the SysV way.

Answer 3

You should try to run it without AppArmor: just run /usr/bin/mysqld_safe or /usr/bin/mysqld without using upstart and wait 30 minutes. If mysql does not auto-restart, then disable AppArmor in the /etc/init/mysql.conf file, or configure it differently.
If the problem is still there, read mysql's log. If the log is not enabled by default, you can use the option --log-error=/tmp/mysql.log --log-warnings when launching mysqld.

Answer 4

Seems converting the /etc/init.d/mysql start up script to sysV style rather than as an upstart script seemed to correct the problem for me.

Answer 5

I had this same problem upgrading to Ubuntu 14.04. I found this question because it mentions the AppArmor log message, so thanks! I may not have realised MySQL was restarting otherwise.
Upon investigating /var/log/daemon.log, I found the output of /etc/mysql/debian-start appearing repeatedly. The relevant part was this:
May 18 06:48:18 tom /etc/mysql/debian-start[15525]: Upgrading MySQL tables if necessary.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysql' as: /usr/bin/mysql
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Error: Server version (5.5.35-1ubuntu1) does not match with the version of
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: the server (5.5.37) with which this program was built/distributed. You can
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: use --skip-version-check to skip this check.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: FATAL ERROR: Upgrade failed
I read through the /etc/mysql/debian-start script and tried running the upgrade command as in the script, hoping to debug it (MySQL server needs to be running at the time):
/usr/bin/mysql_upgrade --defaults-extra-file=/etc/mysql/debian.cnf
I then found that this worked without complaint and everything simply worked from that point onward. I don't know why it was failing in the first place, but that seemed to fix it. MySQL hasn't restarted itself since.