I got those logs from syslog regarding mysql, however they are too brief in that they didn't tell me (or at least I think) what caused mysql to terminate with status 1. Can someone help me get more detailed logs?
Best,
Yuan
May 25 18:32:21 barnard kernel: [1587749.902491] type=1400 audit(1337985141.215:263): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=22272 comm="apparmor_parser"
May 25 18:32:21 barnard kernel: [1587750.181772] init: mysql main process (22276) terminated with status 1
May 25 18:32:21 barnard kernel: [1587750.181819] init: mysql main process ended, respawning
May 25 18:32:22 barnard kernel: [1587750.939171] init: mysql post-start process (22277) terminated with status 1
May 25 18:32:22 barnard kernel: [1587750.951160] type=1400 audit(1337985142.263:264): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=22310 comm="apparmor_parser"
May 25 18:32:22 barnard kernel: [1587751.226853] init: mysql main process (22314) terminated with status 1
May 25 18:32:22 barnard kernel: [1587751.226899] init: mysql main process ended, respawning
May 25 18:32:23 barnard kernel: [1587751.996106] init: mysql post-start process (22315) terminated with status 1
May 25 18:32:23 barnard kernel: [1587752.008270] type=1400 audit(1337985143.323:265): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=22348 comm="apparmor_parser"
May 25 18:32:23 barnard kernel: [1587752.246873] init: mysql main process (22352) terminated with status 1
May 25 18:32:23 barnard kernel: [1587752.246919] init: mysql respawning too fast, stopped
Same problems here. There are some bugs in the Ubuntu installation/upgrade/configuration settings for mysql 5.5 from 5.1. Others claim success with the steps below, but none worked for me. This first step will remove apparmor (a security service), so only do this temporarily during debugging:
sudo service apparmor teardown
Add the following line to your /etc/mysql/my.cnf under a [server] section:
character_set_server = utf8
collation_server = utf8_general_ci
innodb_buffer_pool_size = 256M
But, like I said, this only worked for others, not me.
I'm running mysql off an SSD (/ mounted on SSD) with /var and /tmp symlinked to a HD, do you have a similarly hybrid drive configuration? Maybe this is causing the "respawing too fast" part of our problem.
Related
This is awsome yesterday I got the same error and I debugged it but today I don't remember what to do since I did it to fast.
When I'm doing:
service mysql start
I get this message:
Job for mysql.service failed. See 'systemctl status mysql.service' and 'journalctl -xn' for details.
For this reason I did:
systemctl status mysql.service
and get:
Apr 16 18:11:25 darphus.xen.prgmr.com systemd[1]: Unit mysql.service entered failed state.
Apr 16 18:11:26 darphus.xen.prgmr.com systemd[1]: mysql.service holdoff time over, scheduling restart.
Apr 16 18:11:26 darphus.xen.prgmr.com systemd[1]: Stopping MySQL Community Server...
Apr 16 18:11:26 darphus.xen.prgmr.com systemd[1]: Starting MySQL Community Server...
Apr 16 18:11:26 darphus.xen.prgmr.com systemd[1]: mysql.service start request repeated too quickly, refusing to start.
Apr 16 18:11:26 darphus.xen.prgmr.com systemd[1]: Failed to start MySQL Community Server.
Apr 16 18:11:26 darphus.xen.prgmr.com systemd[1]: Unit mysql.service entered fai
journalctl -xn give:
Logs begin at Sun 2017-04-16 17:58:44 UTC, end at Sun 2017-04-16 18:16:35 UTC
Apr 16 18:16:34 darphus.xen.prgmr.com postfix/pickup[938]: warning: 868F625E66:
Apr 16 18:16:34 darphus.xen.prgmr.com postfix/pickup[938]: 868F625E66: uid=0 fro
Apr 16 18:16:34 darphus.xen.prgmr.com postfix/proxymap[941]: warning: mysql:/etc
Apr 16 18:16:34 darphus.xen.prgmr.com postfix/cleanup[1361]: warning: proxy:mysq
Apr 16 18:16:34 darphus.xen.prgmr.com postfix/cleanup[1361]: warning: 868F625E66
Apr 16 18:16:35 darphus.xen.prgmr.com postfix/pickup[938]: warning: 8744E25E66:
Apr 16 18:16:35 darphus.xen.prgmr.com postfix/pickup[938]: 8744E25E66: uid=0 fro
Apr 16 18:16:35 darphus.xen.prgmr.com postfix/proxymap[941]: warning: mysql:/etc
Apr 16 18:16:35 darphus.xen.prgmr.com postfix/cleanup[1362]: warning: proxy:mysq
Apr 16 18:16:35 darphus.xen.prgmr.com postfix/cleanup[1362]: warning: 8744E25E66
I think a tryed all the trick on the internet and nothing work for today.
Please help me since All my website are down for this reason.
The answer by #Lotfio Lakehal will fix your issues but should be considered a last resort - you certainly wouldn't want to run this on a database that had data you require as it will remove all your configuration values in the process. A better way would be to find out what is causing the issue in your my.cnf and fix it. MySQL has this ability built in.
mysqld --help --verbose
The above command will simulate starting your mysql server. If it can't it will give an error message and a description of what is wrong with the config files.
For instance I set up a test config file with some invalid data and when I ran the above command I got the following:
error: Found option without preceding group in config file: /etc/mysql/my.cnf at line: 22
This allows me to easily go to the configuration file and remove or edit the offending configuration, hopefully, allowing you to fix the said config file without having to do a full reinstall of the server.
Just to clarify this is in no way a put down on this answer, the previous answer will fix 99% of all issues but should be considered a last resort. Ideally, you want to fix the issue and understand why it occurred so you don't make the same mistakes in the future.
Usually this problem appears when you try to change something in the /etc/mysql/my.cnf.
WARNING
The following solution will delete all your database data! Try other solutions first.
Reinstalling MySQL may solve the problem, but should only be used as a last resort! To reinstall:
sudo apt purge mysql-server mysql-common
sudo apt autoremove mysql-server mysql-common
and reinstall it again
sudo apt-get update
sudo apt-get install mysql-server
check this out:
https://askubuntu.com/questions/392438/mysql-job-failed-to-start
note : This solution will delete all your database data ! So be aware of what you are doing .
I had the same problem running on ubuntu 18.04 with mysql 5.7. The first thing that I did was to go to mysql error log, it should be in /var/log/mysql/error.log.
There I found the following line:
[ERROR] InnoDB: Cannot allocate memory for the buffer pool
So looked like a memory issue, I checked the syslog for mentions of MySQL:
grep mysql syslog
Which displayed this:
kernel: [21270406.665399] Out of memory: Kill process 9674 (mysqld) score 251 or sacrifice child
I checked that I had ‘spare memory’ in the form of swap.
free -m
Which displayed this:
total used free shared buff/cache available
Mem: 985 522 69 45 393 276
Swap: 0 0 0
This meant that we have to setup swap space:
Swap: 0 0 0
In my case, mysql is on DigitalOcean and their guides are great so I defer you to the following:
How To Add Swap Space on Ubuntu 18.04
This answer should be read in conjunction with #"James Lendrem" answer.
If you changed or added anything to the mysql configuration files, try removing what you did.
First check the syslog file, which may give you a clue:
grep mysql /var/log/syslog | grep ERROR
This command searches the syslog for any reference to the term "mysql", and then limit those results to ones containing "ERROR".
Next look in the various config files. If you edited a specific config file, start there. Otherwise, look in all the config files in these directories:
ls -l /etc/mysql/*.cfg
ls -l /etc/mysql/mysql.conf.d/
ls -l /etc/mysql/conf.d/
If you change anything in any of the config files, (re)start the mysql server:
service mysql restart
# or else you could use for the first time:
service mysql start
I had the same problem running on ubuntu 18.04 with mysql 5.7
In log files I found the following lines:
Jul 13 01:25:35 r-lap audit[11196]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=11196 comm="mysqld" req
Jul 13 01:25:35 r-lap kernel: audit: type=1400 audit(1531425335.671:116): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/
On stopping apparmor, mysql start was successful with a message that ssl support could not be enabled. Stop the service.
To generate the certs,
Run
sudo mysql_ssl_rsa_setup --datadir /var/lib/mysql
The datadir may be different in your case.
Then do
sudo -i
cd /var/lib/mysql
chown mysql:mysql ./*
This will set the permissions for the generated cert and key files right.
Next you can start both apparmor and then mysql.
This fixed the issue in my case.
Since I did not try separately, I don't know whether it's the apparmor restart or generating the ssl key and certs that fixed the issue.
Note: I later found out this thread
For me, I ran some cache clearing operations to free up space using tools like Stacer, and then MySQL started showing problems.
Use this to get some clues first
grep mysql /var/log/syslog | grep ERROR
If the error says, Could not open file '/var/log/mysql/error.log' for error logging: No such file or directory
then clearly, the log files were cleaned up.
Use this to first create the MySQL directory and grant all permissions for MySQL to use that folder.
> sudo mkdir /var/log/mysql
> sudo chown -R mysql:mysql /var/log/mysql
> sudo service mysql restart
This solution has nothing to do with the my.cnf files and is completely safe on data.
Try running journalctl -u mysql.service to figure out why the error is happening.
Also, systemd cat mysql.service will show you the Service Unit file that describes your service - there might be errors.
Occasionally, without any identifiable pattern, MySQL restarts. There are no entries in the error.log or mysql.log (other than booting and checking tables).
Only thing that I could have identify is this exempt from dmesg:
[58129723.252413] type=1400 audit(1405923841.567:549): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/dev/tty" pid=12848 comm="mysqld" requested_mask="rw" denied_mask="rw" fsuid=105 ouid=0
[58129724.419314] init: mysql main process (26338) terminated with status 1
[58129724.419347] init: mysql main process ended, respawning
Does this mean that AppArmor is killing mysqld because it is try to access it's /dev/tty and the access is denied?
Why would mysqld need to access it's /dev/tty and more should it be allowed by mysqld profile? If it constantly needs access to /dev/tty shouldn't it be killed right away, and not occasionally like this? Period between two 'kills' can be a month sometimes.
Thanks in advance for quick answer
This morning I noticed that my mysql server was not running. A look at the logs and I found the information below. While it is troubling that the mysqld service ran out of memory and was killed, it is more troubling that mysql could not restart.
Any ideas on why mysql could not respawn? How can I test to make sure that if the process is killed it will respawn?
Thank you.
387 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.475042] Out of memory: Kill process 810 (mysqld) score 232 or sacrifice child
388 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.475060] Killed process 810 (mysqld) total-vm:888108kB, anon-rss:139816kB, file-rss:0kB
389 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.655663] init: mysql main process (810) killed by KILL signal
390 Oct 10 06:37:09 ip-xxx-xxx-xxx-xxx kernel: [12218775.655745] init: mysql main process ended, respawning
391 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.044805] type=1400 audit(1381408630.181:13): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=27754 comm="apparmor_parser"
392 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.676434] init: mysql main process (27763) terminated with status 1
393 Oct 10 06:37:10 ip-xxx-xxx-xxx-xxx kernel: [12218776.676489] init: mysql main process ended, respawning
394 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.468923] init: mysql post-start process (27764) terminated with status 1
395 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.512363] type=1400 audit(1381408631.649:14): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=27800 comm="apparmor_parser"
396 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.681433] init: mysql main process (27804) terminated with status 1
397 Oct 10 06:37:11 ip-xxx-xxx-xxx-xxx kernel: [12218777.681491] init: mysql respawning too fast, stopped
I would try running mysqld directly as a command, and looking at the output. It could be e.g. InnoDB corruption leading to stopping immediately after spawning, at which point upstart might try to respawn until apparmour stops it.
http://ubuntuforums.org/showthread.php?t=1475798
looping script issue probably
An old question but a recurrent issue. The question has two faces:
first, why the mysql process runs out of memory?
secondly, why the mysql process can't start again?
The first one is something addressed to an overkill in the configuration. An oversized buffer configuration can make mysql ask for more memory than the system can provide. Verify this question to get insights of how to find best fit for your environment.
The second issue could be very trickful. There are lot of possibilities for problems that can prevent mysql to start. The following steps can be performed to figure out what is the cause.
The first clue can be found in the mysql log error file, in most of cases found in /var/log/mysql/error.log
However, regardless the nature of the issue, the error log file can be empty. In this case try to:
look into syslog: in a terminal prompt
type tail -f /var/log/syslog
and in another terminal try to start mysql:
service start mysql
If this approach don't provide any useful clue try the following:
Start mysqld from raw in verbose mode:
su mysql
mysqld -v
as showed here.
The output messages can be helpful to find out the root cause that prevents mysql to start.
I recently discovered the following error in a log
Oct 14 10:21:43 squire-labs kernel: [4186256.684193] init: mysql main process (16331) terminated with status 1
Oct 14 10:21:43 squire-labs kernel: [4186256.684215] init: mysql main process ended, respawning
Oct 14 10:21:44 squire-labs kernel: [4186257.394474] type=1400 audit(1381738904.046:23): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=28201 comm="apparmor_parser"
Oct 14 10:21:47 squire-labs /etc/mysql/debian-start[28254]: Upgrading MySQL tables if necessary.
Oct 14 10:21:47 squire-labs /etc/mysql/debian-start[28257]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Oct 14 10:21:47 squire-labs /etc/mysql/debian-start[28257]: Looking for 'mysql' as: /usr/bin/mysql
Oct 14 10:21:47 squire-labs /etc/mysql/debian-start[28257]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Oct 14 10:21:47 squire-labs /etc/mysql/debian-start[28257]: This installation of MySQL is already upgraded to 5.5.29, use --force if you still need to run mysql_upgrade
Oct 14 10:21:47 squire-labs /etc/mysql/debian-start[28269]: Checking for insecure root accounts.
Oct 14 10:21:47 squire-labs /etc/mysql/debian-start[28274]: Triggering myisam-recover for all MyISAM tables
This happens often (at least 4 times a minute). I'm really not sure what might be causing this issue. Mysql is running on a production server, with enough disk space and memory. Up until yesterday we've never had any problems.
Any help will be much appreciated.
It turned out that one of my tables in one of the dbs was corrupted. This article proved to be very useful.
In the end, I simply made a mysqldump of the database and restored the dump to a new database. That solved the problem.
i wanna use databases in D:\xampp\mysql\data in my linux mysql without moving it. so i can work on them from both linux and windows
here is what i did:
# mount -t ntfs -o uid=mysql,gid=mysql,dmask=002,fmask=113 /dev/sda5 /media/public/
# cd /var/lib/mysql
# ln /media/public/xampp/mysql/data/my_db -s
# chown -R mysql:mysql /var/lib/mysql/my_db
but mysql cannot read the database tables
use my_db
show tables
gives error: mysql cannot read the directory './my_db'
it seems this is a permission issue. what should i do?
PS: there is only one machine with two operating systems, Windows and Linux.
and i want to use the same directory as datadir when i am in Windows or Linux.
PS: i tried ntfs-3g. now mound partition and all it's content ownership is mysql:mysql. the /var/lib/mysql/my_db is mysql:mysql too. but i still got the following error on show tables
ERROR 1018 (HY000): Can't read dir of './my_db/' (errorno: 13)
First off, never allow more than one instance of MySQL to use the data files at the same time. It's a recipe for disaster.
Second, why do you need MySQL's server on both machines? Why not just use the client on the linux machine to connect via the network to the server on the windows machine?
Third, if you really must do this, I'd suggest enabling master-master replication between them. So each server would have its own storage, and would communicate the changes back and forth automatically.
Edit
Ahhh, wait... Are you dual booting? If so, you need to make sure the entire /var/lib/mysql directory is readable by the mysql user. It's not enough for you to just chmod the my_db folder.
Oh, and how did you link the mount point to the data directory for MySQL (That's not in your list of commands)...
ls /media/public/xampp/mysql/data/my_db
OK, have you pointed the datadir config in my.cnf to /media/public/xampp/mysql/data so MySQL knows where to look for it?
Re: edit:
ln /media/public/xampp/mysql/data/my_db -s
Oh... do you mean ln -s /media/public/xampp/mysql/data/my_db /var/lib/mysql/my_db?
mount -t ntfs
ntfs gives you the kernel NTFS filesystem support, which is read-only. If you want to be able to write to the files, you'll need ntfs-3g, assuming it is installed (most modern distros do).
You may need to add the new data directory to the list of directories in /etc/apparmor.d/usr.sbin.mysqld as per this blog post: http://www.lucasward.net/2011/05/moving-mysql-data-directory-beware-of.html
like so:
/new-mysql-datadir/ r,
/new-mysql-datadir/** rwk,
You can confirm that this is your problem by looking in /var/log/messages and seeing if you have error messages similar to this:
Aug 6 14:47:23 tinybeast kernel: [20254.145552] type=1400 audit(1344228443.102:54): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=24202 comm="apparmor_parser"
Aug 6 14:47:23 tinybeast kernel: [20254.165438] type=1400 audit(1344228443.122:55): apparmor="DENIED" operation="mknod" parent=1 profile="/usr/sbin/mysqld" name="/media/data/mysqldata/tinybeast.lower-test" pid=24206 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Aug 6 14:47:23 tinybeast kernel: [20254.165718] type=1400 audit(1344228443.122:56): apparmor="DENIED" operation="mknod" parent=1 profile="/usr/sbin/mysqld" name="/media/data/mysqldata/tinybeast.lower-test" pid=24206 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Aug 6 14:47:23 tinybeast kernel: [20254.167606] type=1400 audit(1344228443.126:57): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/media/data/mysqldata/mysql/plugin.frm" pid=24206 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Aug 6 14:47:23 tinybeast kernel: [20254.186759] type=1400 audit(1344228443.146:58): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/media/data/mysqldata/ibdata1" pid=24206 comm="mysqld" requested_mask="rw" denied_mask="rw" fsuid=0 ouid=0
To acknowledge my sources I learned all this from the aforementioned blog post, I'm just posting the particular parts here in case the link to the post ever goes stale.