Occasionally, without any identifiable pattern, MySQL restarts. There are no entries in the error.log or mysql.log (other than booting and checking tables).
Only thing that I could have identify is this exempt from dmesg:
[58129723.252413] type=1400 audit(1405923841.567:549): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/dev/tty" pid=12848 comm="mysqld" requested_mask="rw" denied_mask="rw" fsuid=105 ouid=0
[58129724.419314] init: mysql main process (26338) terminated with status 1
[58129724.419347] init: mysql main process ended, respawning
Does this mean that AppArmor is killing mysqld because it is try to access it's /dev/tty and the access is denied?
Why would mysqld need to access it's /dev/tty and more should it be allowed by mysqld profile? If it constantly needs access to /dev/tty shouldn't it be killed right away, and not occasionally like this? Period between two 'kills' can be a month sometimes.
Thanks in advance for quick answer
Related
I am running libvirt in user mode (qemu:///session).
I have /usr/libexec/qemu-bridge-helper with suid set and owner:group set as root:qemu. I have placed the qemu-bridge-helper entries present in /etc/apparmor.d/abstractions in /etc/apparmor.d/local/usr.sbin.libvirtd. Restarted apparmor after the change.
Creation of guest fails.
dmesg shows the following:
[ 4822.650928] audit: type=1400 audit(1519289598.015:142): apparmor="DENIED" operation="exec" profile="/usr/sbin/libvirtd" name="/usr/libexec/qemu-bridge-helper" pid=10051 comm="libvirtd" requested_mask="x" denied_mask="x" fsuid=6684 ouid=0
What am I missing here? And why do I still get the error?
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
I wanted to check out my logserver again, but all of sudden i get this message when i visit the url:
I am running on ubuntu xenial
FATAL: Cannot connect to MySQL server on 'localhost'. Please make sure you have specified a valid MySQL database name in 'include/config.php'
when i do journalctl -xe i get the following message:
-- Unit mysql.service has begun starting up.
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
Jun 15 19:26:12 arjanlog kernel: audit: type=1400 audit(1497547572.504:69): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld"
name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_m
Jun 15 19:26:12 arjanlog kernel: audit: type=1400 audit(1497547572.504:70): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=3665 comm="mysqld" requested_mask="r"
Jun 15 19:26:12 arjanlog kernel: audit: type=1400 audit(1497547572.504:71): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_m
Jun 15 19:26:12 arjanlog systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
I had this problem too and solved it by fixing the apparmor configuration file at /etc/apparmor.d/usr.sbin.mysqld. I added these lines:
/proc/*/status r,
/sys/devices/system/node/ r,
/sys/devices/system/node/node0/meminfo r,
Your journalctl -xe output shows the files MySQL needs permission to use. You can also look in /var/log/syslog:
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
That means that /usr/sbin/mysqld was DENIED trying to open /proc/3665/status and /sys/devices/system/node/ for reading (r).
Note that in the apparmor config file the trailing slash in /sys/devices/system/node/ is necessary, so don't leave it off!
It may not be necessary to stop apparmor entirely. It should be sufficient brute force to disable apparmor for mysqld, if you don't want to track down the core issue:
ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/usr.sbin.mysqld
For me the error was that the mysql database got corrupted. I could see that when I went to logs /var/log/mysql/error.log and it showed an error:
InnoDB: Database was not shutdown normally!
There were some solutions out there which someone could use like table corruption issue
But it didn't work for me as I wasn't able to start a server in WRITE mode in order to solve the issue.
I managed to solve the issue by creating a copy of directory /var/lib/mysql - I copied to mysql.old and then deleted the mysql directory.
Then I called a command in terminal:
sudo mysqld --initialize --initialize-insecure
To generate a new mysql directory
I have then changed permissions of new directory by changing the user and group of mysql directory to user: mysql and group: mysql. Use chown command:
sudo chown -R mysql:mysql /var/lib/mysql
Then I called command:
sudo mysql -u root
and changed password with:
SET PASSWORD FOR root#'localhost' = PASSWORD('newpassword');
And then restarted mysql server:
sudo service mysql restart
And everything worked.
Some of the commands that I have used along the way were (although not sure how much they contributed to fixing the issue):
sudo apt-get --reinstall install mysql-server-5.7
I read in another SO thread comment that the apparmor="DENIED" message probably isn't the reason that MySQL (or in my case MariaDB) wasn't starting, as it's only a warning.
In my case updating and upgrading apt-installed packages and rebooting the system solved the problem.
sudo apt-get dist-upgrade
sudo apt-get update
sudo apt-get upgrade
sudo reboot
Then you'll need to wait a few minutes while the system restarts and log back in and restart the service(s) with either service, systemctl and I think possibly /etc/system.d.
Then MariaDB was chugging away happily.
There's a good post on the various apt upgrade commands here, and I think that what update does is update the source lists.
I had found this command useful as well, when there were no log files in /var/log/mysql:
tail -f /var/log/syslog | grep mysql
I got this bug when I tried to move mysql file from /var/lib/mysql to somewhere else.
add config to /etc/apparmor.d/usr.sbin.mysqld didn't work for me
then i got this bug for ubuntu
it said:
sudo nano /etc/apparmor.d/usr.sbin.mysqld
line 25 after:
# Allow config access
/etc/mysql/** r,
-add line:
/etc/mysql/* r,
and it works for me:
add /etc/mysql/* r, to /etc/apparmor.d/usr.sbin.mysqld
run systemctl restart apparmor.service
and service mysql start
After I followed the instructions given in https://stackoverflow.com/a/45986591 I was still not able to make the MySql server run. I checked its logs via tail -30 /var/log/mysql/error.log . This line caught my attention 2019-05-21T04:46:03.462807Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool. By reading the previous line of code, I was that it was trying to allocate 128MB of memory. Then I checked my Memory status and it was only 111 Mb free. I stopped few programs (node servers) and restarted the MySql and it started working. Then i commented all the AppArmor entries I had entered and restarted AppArmor and MySql. MySql started without any errors. So I could conclude that MySql was not getting enough memory to get started.
solved by : sudo service apparmor teardown
This morning I noticed that my mysql server was not running. A look at the logs and I found the information below. While it is troubling that the mysqld service ran out of memory and was killed, it is more troubling that mysql could not restart.
Any ideas on why mysql could not respawn? How can I test to make sure that if the process is killed it will respawn?
Thank you.
387 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.475042] Out of memory: Kill process 810 (mysqld) score 232 or sacrifice child
388 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.475060] Killed process 810 (mysqld) total-vm:888108kB, anon-rss:139816kB, file-rss:0kB
389 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.655663] init: mysql main process (810) killed by KILL signal
390 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.655745] init: mysql main process ended, respawning
391 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.044805] type=1400 audit(1381408630.181:13): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=27754 comm="apparmor_parser"
392 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.676434] init: mysql main process (27763) terminated with status 1
393 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.676489] init: mysql main process ended, respawning
394 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.468923] init: mysql post-start process (27764) terminated with status 1
395 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.512363] type=1400 audit(1381408631.649:14): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=27800 comm="apparmor_parser"
396 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.681433] init: mysql main process (27804) terminated with status 1
397 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.681491] init: mysql respawning too fast, stopped
I would try running mysqld directly as a command, and looking at the output. It could be e.g. InnoDB corruption leading to stopping immediately after spawning, at which point upstart might try to respawn until apparmour stops it.
http://ubuntuforums.org/showthread.php?t=1475798
looping script issue probably
An old question but a recurrent issue. The question has two faces:
first, why the mysql process runs out of memory?
secondly, why the mysql process can't start again?
The first one is something addressed to an overkill in the configuration. An oversized buffer configuration can make mysql ask for more memory than the system can provide. Verify this question to get insights of how to find best fit for your environment.
The second issue could be very trickful. There are lot of possibilities for problems that can prevent mysql to start. The following steps can be performed to figure out what is the cause.
The first clue can be found in the mysql log error file, in most of cases found in /var/log/mysql/error.log
However, regardless the nature of the issue, the error log file can be empty. In this case try to:
look into syslog: in a terminal prompt
type tail -f /var/log/syslog
and in another terminal try to start mysql:
service start mysql
If this approach don't provide any useful clue try the following:
Start mysqld from raw in verbose mode:
su mysql
mysqld -v
as showed here.
The output messages can be helpful to find out the root cause that prevents mysql to start.
I got those logs from syslog regarding mysql, however they are too brief in that they didn't tell me (or at least I think) what caused mysql to terminate with status 1. Can someone help me get more detailed logs?
Best,
Yuan
May 25 18:32:21 barnard kernel: [1587749.902491] type=1400 audit(1337985141.215:263): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=22272 comm="apparmor_parser"
May 25 18:32:21 barnard kernel: [1587750.181772] init: mysql main process (22276) terminated with status 1
May 25 18:32:21 barnard kernel: [1587750.181819] init: mysql main process ended, respawning
May 25 18:32:22 barnard kernel: [1587750.939171] init: mysql post-start process (22277) terminated with status 1
May 25 18:32:22 barnard kernel: [1587750.951160] type=1400 audit(1337985142.263:264): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=22310 comm="apparmor_parser"
May 25 18:32:22 barnard kernel: [1587751.226853] init: mysql main process (22314) terminated with status 1
May 25 18:32:22 barnard kernel: [1587751.226899] init: mysql main process ended, respawning
May 25 18:32:23 barnard kernel: [1587751.996106] init: mysql post-start process (22315) terminated with status 1
May 25 18:32:23 barnard kernel: [1587752.008270] type=1400 audit(1337985143.323:265): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=22348 comm="apparmor_parser"
May 25 18:32:23 barnard kernel: [1587752.246873] init: mysql main process (22352) terminated with status 1
May 25 18:32:23 barnard kernel: [1587752.246919] init: mysql respawning too fast, stopped
Same problems here. There are some bugs in the Ubuntu installation/upgrade/configuration settings for mysql 5.5 from 5.1. Others claim success with the steps below, but none worked for me. This first step will remove apparmor (a security service), so only do this temporarily during debugging:
sudo service apparmor teardown
Add the following line to your /etc/mysql/my.cnf under a [server] section:
character_set_server = utf8
collation_server = utf8_general_ci
innodb_buffer_pool_size = 256M
But, like I said, this only worked for others, not me.
I'm running mysql off an SSD (/ mounted on SSD) with /var and /tmp symlinked to a HD, do you have a similarly hybrid drive configuration? Maybe this is causing the "respawing too fast" part of our problem.
i wanna use databases in D:\xampp\mysql\data in my linux mysql without moving it. so i can work on them from both linux and windows
here is what i did:
# mount -t ntfs -o uid=mysql,gid=mysql,dmask=002,fmask=113 /dev/sda5 /media/public/
# cd /var/lib/mysql
# ln /media/public/xampp/mysql/data/my_db -s
# chown -R mysql:mysql /var/lib/mysql/my_db
but mysql cannot read the database tables
use my_db
show tables
gives error: mysql cannot read the directory './my_db'
it seems this is a permission issue. what should i do?
PS: there is only one machine with two operating systems, Windows and Linux.
and i want to use the same directory as datadir when i am in Windows or Linux.
PS: i tried ntfs-3g. now mound partition and all it's content ownership is mysql:mysql. the /var/lib/mysql/my_db is mysql:mysql too. but i still got the following error on show tables
ERROR 1018 (HY000): Can't read dir of './my_db/' (errorno: 13)
First off, never allow more than one instance of MySQL to use the data files at the same time. It's a recipe for disaster.
Second, why do you need MySQL's server on both machines? Why not just use the client on the linux machine to connect via the network to the server on the windows machine?
Third, if you really must do this, I'd suggest enabling master-master replication between them. So each server would have its own storage, and would communicate the changes back and forth automatically.
Edit
Ahhh, wait... Are you dual booting? If so, you need to make sure the entire /var/lib/mysql directory is readable by the mysql user. It's not enough for you to just chmod the my_db folder.
Oh, and how did you link the mount point to the data directory for MySQL (That's not in your list of commands)...
ls /media/public/xampp/mysql/data/my_db
OK, have you pointed the datadir config in my.cnf to /media/public/xampp/mysql/data so MySQL knows where to look for it?
Re: edit:
ln /media/public/xampp/mysql/data/my_db -s
Oh... do you mean ln -s /media/public/xampp/mysql/data/my_db /var/lib/mysql/my_db?
mount -t ntfs
ntfs gives you the kernel NTFS filesystem support, which is read-only. If you want to be able to write to the files, you'll need ntfs-3g, assuming it is installed (most modern distros do).
You may need to add the new data directory to the list of directories in /etc/apparmor.d/usr.sbin.mysqld as per this blog post: http://www.lucasward.net/2011/05/moving-mysql-data-directory-beware-of.html
like so:
/new-mysql-datadir/ r,
/new-mysql-datadir/** rwk,
You can confirm that this is your problem by looking in /var/log/messages and seeing if you have error messages similar to this:
Aug 6 14:47:23 tinybeast kernel: [20254.145552] type=1400 audit(1344228443.102:54): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=24202 comm="apparmor_parser"
Aug 6 14:47:23 tinybeast kernel: [20254.165438] type=1400 audit(1344228443.122:55): apparmor="DENIED" operation="mknod" parent=1 profile="/usr/sbin/mysqld" name="/media/data/mysqldata/tinybeast.lower-test" pid=24206 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Aug 6 14:47:23 tinybeast kernel: [20254.165718] type=1400 audit(1344228443.122:56): apparmor="DENIED" operation="mknod" parent=1 profile="/usr/sbin/mysqld" name="/media/data/mysqldata/tinybeast.lower-test" pid=24206 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Aug 6 14:47:23 tinybeast kernel: [20254.167606] type=1400 audit(1344228443.126:57): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/media/data/mysqldata/mysql/plugin.frm" pid=24206 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Aug 6 14:47:23 tinybeast kernel: [20254.186759] type=1400 audit(1344228443.146:58): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/media/data/mysqldata/ibdata1" pid=24206 comm="mysqld" requested_mask="rw" denied_mask="rw" fsuid=0 ouid=0
To acknowledge my sources I learned all this from the aforementioned blog post, I'm just posting the particular parts here in case the link to the post ever goes stale.