Kubuntu 17
When I login and do "ps -ef | grep mysqld" I see two copies running, one as the system service daemon, and one as a user process.
ps -ef | grep mysqld
mysql 1953 1 0 09:02 ? 00:00:13 /usr/sbin/mysqld
richard 3233 3220 0 09:13 ? 00:00:35 /usr/sbin/mysqld --
defaults-file=/home/richard/.local/share/akonadi/mysql.conf --
datadir=/home/richard/.local/share/akonadi/db_data/ --
socket=/tmp/akonadi-richard.EK3Z9U/mysql.socket
richard 13309 3138 0 18:05 pts/1 00:00:00 grep --color=auto
mysqld
I don't see any script or command with "mysqld" in ~/.profle, ~/.autostart, ~/.bashrc, /etc/profile, /etc/init.d (except for the mysql start script presumably used by the system and owned by root.
Where else should I look for the errant command?
Any great ideas on how to look for it effectively?
Related
I have mysql installed on Ubuntu18.04. I tried to kill the mysql process with kill -9 but it comes back immediately. Here's example from my terminal output:
root#mysql-image:~# ps aux | grep mysql
mysql 779 0.8 19.2 1166336 188064 ? Sl 20:06 0:02 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid
root 1046 0.0 0.1 13144 1060 pts/0 S+ 20:11 0:00 grep --color=auto mysql
root#mysql-image:~# kill -9 779
root#mysql-image:~# ps aux | grep mysql
mysql 1063 21.5 18.1 1165936 177556 ? Sl 20:11 0:00 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid
root 1094 0.0 0.1 13144 1032 pts/0 S+ 20:11 0:00 grep --color=auto mysql
root#mysql-image:~# kill -9 1063
root#mysql-image:~# ps aux | grep mysql
mysql 1142 21.5 18.1 1165936 177628 ? Sl 20:12 0:00 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid
root 1173 0.0 0.1 13144 1084 pts/0 S+ 20:12 0:00 grep --color=auto mysql
I'm trying to destroy the mysql process in an abrupt way and "unsafe way" as part of an experiment I'm doing. But the kill -9 doesn't seem to permanently shut down the mysql database.
What am I doing wrong?
Normally, systemd will be configured to restart mysql if it unexpectedly terminates, with Restart=on-abort in /usr/lib/systemd/system/mysql.service.
on-abort is explained in: https://unix.stackexchange.com/questions/564443/what-does-restart-on-abort-mean-in-a-systemd-service
You could try making a change to the mysql conf files that keep it from restarting (setting a non-existent user, maybe?). Or you can just do a
kill -15 instead, which is treated as an expected termination so won't cause a restart.
I'm learning linux core and I'm on the namepsaces topic now.
I tried to play with "unshare" command just to get hang of namespace and its essentials.
The problem is that it doesn't or, what is more probable, I'm doing something wrong.
I'd appreciate if you could help me to understand.
I try to execute busybox sh program as within its own PID namespace. That's what I do:
[ab#a ~]$ sudo unshare --pid busybox sh
/home/ab # ps
PID TTY TIME CMD
6014 pts/1 00:00:00 sudo
6016 pts/1 00:00:00 busybox
6026 pts/1 00:00:00 ps
So as I can see it from the output of the ps command all process are visible in the new env. And it gets confirmed when I check pid namespace id of newly created process and current one. See below
[ab#a ~]$ ps -p 6016,$$
PID TTY TIME CMD
4604 pts/0 00:00:00 bash
6016 pts/1 00:00:00 busybox
[ab#a ~]$ sudo ls -l /proc/4604/ns
total 0
lrwxrwxrwx. 1 ab ab 0 Aug 8 23:49 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 ab ab 0 Aug 8 23:49 mnt -> mnt:[4026531840]
lrwxrwxrwx. 1 ab ab 0 Aug 8 23:49 net -> net:[4026531968]
lrwxrwxrwx. 1 ab ab 0 Aug 8 23:49 pid -> pid:[4026531836]
lrwxrwxrwx. 1 ab ab 0 Aug 8 23:49 user -> user:[4026531837]
lrwxrwxrwx. 1 ab ab 0 Aug 8 23:49 uts -> uts:[4026531838]
[ab#a ~]$ sudo ls -l /proc/6016/ns
total 0
lrwxrwxrwx. 1 root root 0 Aug 9 00:07 ipc -> ipc:[4026531839]
lrwxrwxrwx. 1 root root 0 Aug 9 00:07 mnt -> mnt:[4026531840]
lrwxrwxrwx. 1 root root 0 Aug 9 00:07 net -> net:[4026531968]
lrwxrwxrwx. 1 root root 0 Aug 9 00:07 pid -> pid:[4026531836]
lrwxrwxrwx. 1 root root 0 Aug 9 00:07 user -> user:[4026531837]
lrwxrwxrwx. 1 root root 0 Aug 9 00:07 uts -> uts:[4026531838]
So, pid namespace stays the same eventhough I provided --pid argument to unshare call.
Could you please help me to understand why it happens.
Thanks
Solution
you should add --fork and --mount-proc switch to unshare as stated in the man page
-f, --fork
Fork the specified program as a child process of unshare rather than running it directly. This is useful
when creating a new PID namespace. Note that when unshare is waiting for the child process, then it
ignores SIGINT and SIGTERM and does not forward any signals to the child. It is necessary to send
signals to the child process.
Explanation (from man pid_namespaces)
a process's PID namespace membership is determined when the process is created and cannot be changed thereafter.
what unshare actually does when you supply --pid is setting the file descriptor at /proc/[PID]/ns/pid_for_children for the current process to the new PID namespace, causing children subsequently created by this process to be places in a different PID namespace (its children not itself!! important!).
So, when you supply --fork to unshare, it will fork your program (in this case busybox sh) as a child process of unshare and place it in the new PID namespace.
Why do I need --mount-proc ?
Try running unshare with only --pid and --fork and let's see what happen.
wendel#gentoo-grill ~ λ sudo unshare --pid --fork busybox sh
/home/wendel # echo $$
1
/home/wendel # ps
PID USER TIME COMMAND
12443 root 0:00 unshare --pid --fork busybox sh
12444 root 0:00 busybox sh
24370 root 0:00 {ps} busybox sh
.
.
. // bunch more
from echo $$ we can see that the pid is actually 1 so we know that we must be in the new PID namespace, but when we run ps we see other processes as if we are still in the parent PID namespace.
This is because of /proc is a special filesystem called procfs that kernel created in memory, and from the man page.
A /proc filesystem shows (in the /proc/[pid] directories) only processes visible in the PID namespace of the process that performed the mount, even if the /proc filesystem is viewed from processes in other namespaces.
So, in order for tools such as ps to work correctly, we need to re-mount /proc using a process in the new namespace.
But, assuming that your process is in the root mount namespace, if we re-mount /proc, this will mess up many things for other processes in the same mount namespace, because now they can't see anything (in /proc). So you should also put your process in new mount namespace too.
Good thing is unshare has --mount-proc.
--mount-proc[=mountpoint]
Just before running the program, mount the proc filesystem at mountpoint (default is /proc). This is useful when creating a new PID namespace. It also implies creating a new mount namespace since the /proc mount would
otherwise mess up existing programs on the system. The new proc filesystem is explicitly mounted as private (with MS_PRIVATE|MS_REC).
Let's verify that --mount-proc also put your process in new mount namespace.
bash outside:
wendel#gentoo-grill ~ λ ls -go /proc/$$/ns/{user,mnt,pid}
lrwxrwxrwx 1 0 Aug 9 10:05 /proc/17011/ns/mnt -> 'mnt:[4026531840]'
lrwxrwxrwx 1 0 Aug 9 10:10 /proc/17011/ns/pid -> 'pid:[4026531836]'
lrwxrwxrwx 1 0 Aug 9 10:10 /proc/17011/ns/user -> 'user:[4026531837]'
busybox:
wendel#gentoo-grill ~ λ doas ls -go /proc/16436/ns/{user,mnt,pid}
lrwxrwxrwx 1 0 Aug 9 10:05 /proc/16436/ns/mnt -> 'mnt:[4026533479]'
lrwxrwxrwx 1 0 Aug 9 10:04 /proc/16436/ns/pid -> 'pid:[4026533481]'
lrwxrwxrwx 1 0 Aug 9 10:17 /proc/16436/ns/user -> 'user:[4026531837]'
Notice that their user namespace is the same but mount and pid aren't.
Note: You can see that I cited a lot from man pages. If you want to learn more about linux namespaces (or anything unix really) first thing for you to do is to read the man page of each namespace. It is well written and really informative.
When I run apt-get install percona-server-server-5.6, the percona server gets installed and started. The processes running (ps -ef | grep mysql) look like this (HOSTNAME is obfuscated):
root 14309 1 0 23:54 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/HOSTNAME.pid
mysql 14413 14309 3 23:54 ? 00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/lib/mysql/HOSTNAME.err --pid-file=/var/lib/mysql/HOSTNAME.pid
My question is where is this default PID file coming from? There is no my.cnf file under /etc/mysql and the my.cnf file under /usr doesn't have any of this information. This is causing a problem ebcause when I deploy my configuration file and try to restart the server, the box obviously doesn't work. I believe the correct default for the PID file is /var/run/mysqld/mysqld.pid.
Looks like this is being set in the init mysql file.
server system: Ubuntu 10.04.2
Couchbase: 1.8.1
python: 2.7.3
back files: cbbackup ./default-data/default /data/Couchbase_backups/default_20120705
restore: cbrestore -a default_20120705/default*
root#945f14b6-3015-4dac-b486-a8914a3f553d:~# cbrestore -a /data/Couchbase_backups/default_20120706/defaults*
Error on key 'sk_deviceHistory_F118C61C-8B30-4667-A7E2-2959836734BD': Memcached error #32: Auth failure
Error on key 'sk_deviceHistory_F118C61C-8B30-4667-A7E2-2959836734BD': Memcached error #32: Auth failure
Error on key 'sk_deviceHistory_F118C61C-8B30-4667-A7E2-2959836734BD': Memcached error #32: Auth failure
Error on key 'sk_deviceHistory_F118C61C-8B30-4667-A7E2-2959836734BD': Memcached error #32: Auth failure
^Z
[1]+ Stopped cbrestore -a /data/Couchbase_backups/default_20120706/defaults*
root#945f14b6-3015-4dac-b486-a8914a3f553d:~# ps -ef|grep python
root 711 512 1 15:46 pts/2 00:00:00 python /opt/couchbase/lib/python/cbrestore -a /data/Couchbase_backups/default_20120706/defaults /data/Couchbase_backups/default_20120706/defaults-0.mb /data/Couchbase_backups/default_20120706/defaults-1.mb /data/Couchbase_backups/default_20120706/defaults-2.mb /data/Couchbase_backups/default_20120706/defaults-3.mb
root 726 512 0 15:46 pts/2 00:00:00 grep --color=auto python
root#945f14b6-3015-4dac-b486-a8914a3f553d:~# kill -9 711
root#945f14b6-3015-4dac-b486-a8914a3f553d:~# ps -ef|grep python
root 729 512 0 15:47 pts/2 00:00:00 grep --color=auto python
[1]+ Killed cbrestore -a /data/Couchbase_backups/default_20120706/defaults*
root#945f14b6-3015-4dac-b486-a8914a3f553d:~# ps -ef|grep python
root 731 512 0 15:47 pts/2 00:00:00 grep --color=auto python
root#945f14b6-3015-4dac-b486-a8914a3f553d:~#
have you tried the -u BUCKET_NAME and -p BUCKET_PASSWORD parameters? That way the cbrestore tool can successfully authenticate to the right bucket?
For more flag and usage, also, there's also the -h / --help parameter.
sudo ./cbbackup HOST:PORT ~/Documents/ -u ‘username’ -p ‘password‘ -b BUCKET_NAME
This works for me.
I'm having an issue where MySQL 5.1.54 is restarting every 30 minutes on Ubuntu 11.04. When this occurs, the following appears in the MySQL log:
111030 12:01:52 [Note] /usr/sbin/mysqld: Normal shutdown
111030 12:01:52 [Note] Event Scheduler: Purging the queue. 0 events
111030 12:01:52 InnoDB: Starting shutdown...
111030 12:01:54 InnoDB: Shutdown completed; log sequence number 0 875122
111030 12:01:54 [Note] /usr/sbin/mysqld: Shutdown complete
111030 12:01:55 [Note] Plugin 'FEDERATED' is disabled.
111030 12:01:55 InnoDB: Initializing buffer pool, size = 256.0M
111030 12:01:55 InnoDB: Completed initialization of buffer pool
111030 12:01:55 InnoDB: Started; log sequence number 0 875122
111030 12:01:55 [Note] Event Scheduler: Loaded 0 events
111030 12:01:55 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.1.54-1ubuntu4-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
This occurs like clockwork every 30 minutes, so it's obviously some service restarting it.
I have checked the crontab of every user on the system (including system users), and none of them have a crontab setup, as you can see in the output below:
# awk -F: '{print $1}' /etc/passwd | xargs -n 1 -i crontab -u {} -l
no crontab for root
no crontab for daemon
no crontab for bin
no crontab for sys
no crontab for sync
no crontab for games
no crontab for man
no crontab for lp
no crontab for mail
no crontab for news
no crontab for uucp
no crontab for proxy
no crontab for www-data
no crontab for backup
no crontab for list
no crontab for irc
no crontab for gnats
no crontab for nobody
no crontab for libuuid
no crontab for syslog
no crontab for sshd
no crontab for landscape
no crontab for ubuntu
no crontab for statd
no crontab for myproxy
no crontab for condor
no crontab for messagebus
no crontab for avahi
no crontab for joe
no crontab for smmta
no crontab for smmsp
no crontab for postfix
no crontab for deploy
no crontab for mysql
no crontab for redis
My dmesg contains the following each time it is restarted. I'm not an apparmor expert, but I believe this is a normal message obtained each time the MySQL service starts:
[1165328.780405] type=1400 audit(1319976114.984:74): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=31985 comm="apparmor_parser"
Also, here are the contents of the MySQL upstart configuration in /etc/init/mysql.conf:
# MySQL Service
description "MySQL Server"
author "Mario Limonciello <superm1#ubuntu.com>"
start on (net-device-up
and local-filesystems
and runlevel [2345])
stop on runlevel [016]
respawn
env HOME=/etc/mysql
umask 007
# The default of 5 seconds is too low for mysql which needs to flush buffers
kill timeout 300
pre-start script
#Sanity checks
[ -r $HOME/my.cnf ]
[ -d /var/run/mysqld ] || install -m 755 -o mysql -g root -d /var/run/mysqld
/lib/init/apparmor-profile-load usr.sbin.mysqld
LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<4096) }'
end script
exec /usr/sbin/mysqld
post-start script
for i in `seq 1 30` ; do
/usr/bin/mysqladmin --defaults-file="${HOME}"/debian.cnf ping && {
exec "${HOME}"/debian-start
# should not reach this line
exit 2
}
sleep 1
done
exit 1
end script
Any idea what might be causing this? It doesn't cause any problems, other than Monit alerts stating that "PID changed Service mysqld" (I have Monit monitoring mysqld -- but it reports no errors with the mysqld process, other than the fact that every 30 minutes, it has its PID changed since MySQL is restarted).
Thanks in advance.
are you using chef or puppet, which might be doing something that triggers the reboot?
can you check (and probably post) the definition of your mysql job in upstart? (/etc/init/mysql.conf). OK = try removing the "respawn". It does not work as documented in the upstart documentation. Generally it's used to respawn the process if it's killed by other process, but it seems it does not function as expected. You can see why the apparmour is always loading - because of the pre-start stanza in the script.
As upstart is very new, and is still evolving, better is to use the SysV way.
You should try to run it without AppArmor: just run /usr/bin/mysqld_safe or /usr/bin/mysqld without using upstart and wait 30 minutes. If mysql does not auto-restart, then disable AppArmor in the /etc/init/mysql.conf file, or configure it differently.
If the problem is still there, read mysql's log. If the log is not enabled by default, you can use the option --log-error=/tmp/mysql.log --log-warnings when launching mysqld.
Seems converting the /etc/init.d/mysql start up script to sysV style rather than as an upstart script seemed to correct the problem for me.
I had this same problem upgrading to Ubuntu 14.04. I found this question because it mentions the AppArmor log message, so thanks! I may not have realised MySQL was restarting otherwise.
Upon investigating /var/log/daemon.log, I found the output of /etc/mysql/debian-start appearing repeatedly. The relevant part was this:
May 18 06:48:18 tom /etc/mysql/debian-start[15525]: Upgrading MySQL tables if necessary.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysql' as: /usr/bin/mysql
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: Error: Server version (5.5.35-1ubuntu1) does not match with the version of
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: the server (5.5.37) with which this program was built/distributed. You can
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: use --skip-version-check to skip this check.
May 18 06:48:18 tom /etc/mysql/debian-start[15528]: FATAL ERROR: Upgrade failed
I read through the /etc/mysql/debian-start script and tried running the upgrade command as in the script, hoping to debug it (MySQL server needs to be running at the time):
/usr/bin/mysql_upgrade --defaults-extra-file=/etc/mysql/debian.cnf
I then found that this worked without complaint and everything simply worked from that point onward. I don't know why it was failing in the first place, but that seemed to fix it. MySQL hasn't restarted itself since.