The MySQL service periodically stops. The error log is empty. The last info in system log before the last crash was
Nov 11 09:24:48 qwerty kernel: [2594467.725652] type=1400 audit(1447233888.936:26): apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=19376 comm="apparmor_parser"
Nov 11 09:24:50 qwerty kernel: [2594469.164305] type=1400 audit(1447233890.376:27): apparmor="ALLOWED" operation="unlink" parent=1 profile="/usr/sbin/mysqld" name="/run/mysqld/mysqld.sock" pid=19388 comm="mysqld" requested_mask="d" denied_mask="d" fsuid=104 ouid=104
Nov 11 09:24:50 qwerty kernel: [2594469.164330] type=1400 audit(1447233890.376:28): apparmor="ALLOWED" operation="mknod" parent=1 profile="/usr/sbin/mysqld" name="/run/mysqld/mysqld.sock" pid=19388 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=104 ouid=104
Nov 11 09:24:50 qwerty kernel: [2594469.164495] type=1400 audit(1447233890.376:29): apparmor="ALLOWED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/run/mysqld/mysql.pid" pid=19423 comm="mysqld" requested_mask="wc" denied_mask="wc" fsuid=104 ouid=104
Nov 11 09:24:50 qwerty kernel: [2594469.164504] type=1400 audit(1447233890.376:30): apparmor="ALLOWED" operation="truncate" parent=1 profile="/usr/sbin/mysqld" name="/run/mysqld/mysql.pid" pid=19423 comm="mysqld" requested_mask="w" denied_mask="w" fsuid=104 ouid=104
Nov 11 09:24:50 qwerty /etc/mysql/debian-start[19428]: Upgrading MySQL tables if necessary.
Nov 11 09:24:51 qwerty /etc/mysql/debian-start[19431]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Nov 11 09:24:51 qwerty /etc/mysql/debian-start[19431]: Looking for 'mysql' as: /usr/bin/mysql
Nov 11 09:24:51 qwerty /etc/mysql/debian-start[19431]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Nov 11 09:24:51 qwerty /etc/mysql/debian-start[19431]: This installation of MySQL is already upgraded to 5.5.41, use --force if you still need to run mysql_upgrade
Nov 11 09:24:51 qwerty /etc/mysql/debian-start[19443]: Checking for insecure root accounts.
Nov 11 09:24:51 qwerty /etc/mysql/debian-start[19449]: Triggering myisam-recover for all MyISAM tables
Could you help me to figure out what it might be?
Related
I don't know exactly what I did wrong, but it's likely some 'chown' operation that I did. I was trying to allow the user&group mysql:mysql access to a /media/usb drive, but may have inadvertently changed something else.
When I do sudo systemctl start mysql.service I get an error. Upon examining with sudo systemctl status mysqld, I get the following:
mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since Fri 2020-06-19 08:11:01 EDT; 19s ago
Process: 15459 ExecStart=/usr/sbin/mysqld (code=exited, status=203/EXEC)
Process: 15444 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 15459 (code=exited, status=203/EXEC); : 15460 (mysql-systemd-s)
Tasks: 2
Memory: 2.4M
CPU: 175ms
CGroup: /system.slice/mysql.service
└─control
├─15460 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─15687 sleep 1
Jun 19 08:11:01 apil-dlrig systemd[1]: Starting MySQL Community Server...
Jun 19 08:11:01 apil-dlrig systemd[1]: mysql.service: Main process exited, code=exited, status=203/EXEC
When I check ownership on /var/lib/mysql, I get the following, which seems reasonable. I.e. user mysql has full ownership on this folder.
apil#apil-dlrig:~$ sudo ls -la /var/lib/mysql
total 176212
drwx------ 7 mysql mysql 4096 Jun 19 07:34 .
drwxr-xr-x 79 root root 4096 Oct 30 2019 ..
-rw-r----- 1 mysql mysql 56 Oct 20 2019 auto.cnf
-rw------- 1 mysql mysql 1680 Nov 22 2019 ca-key.pem
-rw-r--r-- 1 mysql mysql 1112 Nov 22 2019 ca.pem
-rw-r--r-- 1 mysql mysql 1112 Nov 22 2019 client-cert.pem
-rw------- 1 mysql mysql 1676 Nov 22 2019 client-key.pem
-rw-r--r-- 1 mysql mysql 0 May 5 06:38 debian-5.7.flag
drwxr-x--- 2 mysql mysql 4096 Jun 6 13:44 foo
-rw-r----- 1 mysql mysql 665 Jun 19 07:34 ib_buffer_pool
-rw-r----- 1 mysql mysql 79691776 Jun 19 07:34 ibdata1
-rw-r----- 1 mysql mysql 50331648 Jun 19 07:34 ib_logfile0
-rw-r----- 1 mysql mysql 50331648 Oct 20 2019 ib_logfile1
-rw-r----- 1 mysql mysql 155 Jun 16 07:23 keyring_backup
drwxr-x--- 2 mysql mysql 4096 May 5 06:38 mysql
-rw-r--r-- 1 mysql mysql 6 May 5 06:38 mysql_upgrade_info
drwxr-x--- 2 mysql mysql 4096 May 5 06:38 performance_schema
-rw------- 1 mysql mysql 1680 Nov 22 2019 private_key.pem
drwxr-x--- 2 mysql mysql 4096 Jun 16 07:25 prod
-rw-r--r-- 1 mysql mysql 452 Nov 22 2019 public_key.pem
-rw-r--r-- 1 mysql mysql 1112 Nov 22 2019 server-cert.pem
-rw------- 1 mysql mysql 1680 Nov 22 2019 server-key.pem
drwxr-x--- 2 mysql mysql 12288 Nov 22 2019 sys
The /etc/systemd/system/multi.user.wants.targets/mysql.service looks as the following. Nothing should've changed here, i.e. it is as default as MySQL comes.
# MySQL systemd service file
[Unit]
Description=MySQL Community Server
After=network.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
PermissionsStartOnly=true
ExecStartPre=/usr/share/mysql/mysql-systemd-start pre
ExecStart=/usr/sbin/mysqld
ExecStartPost=/usr/share/mysql/mysql-systemd-start post
TimeoutSec=600
Restart=on-failure
RuntimeDirectory=mysqld
RuntimeDirectoryMode=755
Wondering what could be going wrong. Any help would be appreciated.
Thanks
If you look at the original error, the issue is: ExecStart=/usr/sbin/mysqld (code=exited, status=203/EXEC). Looks like some kind of execution privilege for the mysqld file? Checked it too:
ls -la /usr/sbin/mysqld which returned
-rw-r--r-- 1 root root 24585896 Apr 30 10:52 /usr/sbin/mysqld
So the issue (I thought) was that user root didn't have the execute permission. Looks at the first three letters rw-. The last dash means no execution privilege.
So I simply ran the following chmod 777 /usr/sbin/mysqld, after which the ownership returns as
-rwxrwxrwx 1 root root 24585896 Apr 30 10:52 /usr/sbin/mysqld
Now, systemctl start mysql.service runs just fine.
It's amazing how simply the process of writing a question on stackoverflow actually helps me solve a problem 80% of the time. Thanks again, folks.
Please help. All of a sudden around 2019-01-24 06:37:10 UTC I started receiving errors on my server that:
SQLSTATE[HY000] [1045] Access denied for user 'forge'#'localhost' (using password: YES)
I'm unable to access mysql. I did not change anything on my end, so I don't know what happened. When I look through the system logs, I do see some mysql activity around that time:
Jan 24 06:36:07 whispering-hurricane systemd[1]: Starting Daily apt upgrade and clean activities...
Jan 24 06:36:24 whispering-hurricane kernel: [1767450.357697] [UFW BLOCK] IN=eth0 OUT= MAC=32:f4:a6:e9:03:29:f0:4b:3a:4e:50:30:08:00 SRC=85.10.193.56 DST=206.189.227.234 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=37186 DF PROTO=TCP SPT=57178 DPT=22123 WINDOW=29200 RES=0x00 SYN URGP=0
Jan 24 06:36:32 whispering-hurricane systemd[1]: Reloading.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Reloading.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Stopped Daily apt upgrade and clean activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Stopping Daily apt upgrade and clean activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Started Daily apt upgrade and clean activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Stopped Daily apt download activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Stopping Daily apt download activities.
Jan 24 06:36:33 whispering-hurricane systemd[1]: Started Daily apt download activities.
Jan 24 06:36:38 whispering-hurricane systemd[1]: Reloading.
Jan 24 06:36:38 whispering-hurricane systemd[1]: Stopping MySQL Community Server...
Jan 24 06:36:42 whispering-hurricane systemd[1]: Stopped MySQL Community Server.
Jan 24 06:36:42 whispering-hurricane systemd[1]: Reloading.
Jan 24 06:36:46 whispering-hurricane kernel: [1767472.035497] audit: type=1400 audit(1548311806.292:39): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/sbin/mysqld" pid=27468 comm="apparmor_parser"
Jan 24 06:36:46 whispering-hurricane kernel: [1767472.069261] audit: type=1400 audit(1548311806.324:40): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27486 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 06:36:46 whispering-hurricane kernel: [1767472.132572] audit: type=1400 audit(1548311806.388:41): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27501 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 06:36:50 whispering-hurricane kernel: [1767476.078756] audit: type=1400 audit(1548311810.336:42): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27552 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 06:36:53 whispering-hurricane kernel: [1767478.885369] audit: type=1400 audit(1548311813.140:43): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/sbin/mysqld" pid=27601 comm="apparmor_parser"
Jan 24 06:36:53 whispering-hurricane systemd[1]: message repeated 5 times: [ Reloading.]
Jan 24 06:36:53 whispering-hurricane systemd[1]: Starting MySQL Community Server...
Jan 24 06:36:53 whispering-hurricane kernel: [1767479.454427] audit: type=1400 audit(1548311813.708:44): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27677 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 06:36:53 whispering-hurricane kernel: [1767479.499368] audit: type=1400 audit(1548311813.756:45): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=27682 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=113 ouid=0
Jan 24 06:36:54 whispering-hurricane systemd[1]: Started MySQL Community Server.
Can someone help me understand if something in that system log is what caused the problem? How can I fix it? This is our production server and I can not access the database. Neither can our application. Not good.
I see these upgrades occurred automatically at that time. These must be causing the issue:
Start-Date: 2019-01-24 06:36:21
Commandline: /usr/bin/unattended-upgrade
Upgrade: mysql-client-5.7:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2)
End-Date: 2019-01-24 06:36:22
Start-Date: 2019-01-24 06:36:25
Commandline: /usr/bin/unattended-upgrade
Upgrade: mysql-client-core-5.7:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2)
End-Date: 2019-01-24 06:36:26
Start-Date: 2019-01-24 06:36:38
Commandline: /usr/bin/unattended-upgrade
Upgrade: mysql-server-5.7:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2), mysql-server-core-5.7:amd64 (5.7.24-0ubuntu0.18.04.1, 5.7.25-0ubuntu0.18.04.2)
End-Date: 2019-01-24 06:36:54
I ran these commands as per one of the comments:
ufw status
ufw allow 3306
This did not seem to help. Here's the current output of ufw status:
Status: active
To Action From
-- ------ ----
22 ALLOW Anywhere
80 ALLOW Anywhere
443 ALLOW Anywhere
3306 ALLOW Anywhere
22 (v6) ALLOW Anywhere (v6)
80 (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
3306 (v6) ALLOW Anywhere (v6)
When I restart mysql on a working development server, I get this output in the syslog:
Jan 24 13:08:16 smooth-star systemd[1]: Stopping MySQL Community Server...
Jan 24 13:08:17 smooth-star systemd[1]: Stopped MySQL Community Server.
Jan 24 13:08:17 smooth-star systemd[1]: Starting MySQL Community Server...
Jan 24 13:08:17 smooth-star kernel: [4733737.407768] audit: type=1400 audit(1548335297.764:31): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=28433 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 13:08:17 smooth-star kernel: [4733737.456162] audit: type=1400 audit(1548335297.812:32): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=28438 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=113 ouid=0
Jan 24 13:08:18 smooth-star systemd[1]: Started MySQL Community Server.
On the production server, which has the issue, I get this when restarting mysql:
Jan 24 13:10:40 whispering-hurricane systemd[1]: Stopping MySQL Community Server...
Jan 24 13:10:41 whispering-hurricane systemd[1]: Stopped MySQL Community Server.
Jan 24 13:10:41 whispering-hurricane systemd[1]: Starting MySQL Community Server...
Jan 24 13:10:41 whispering-hurricane kernel: [ 8114.162705] kauditd_printk_skb: 6 callbacks suppressed
Jan 24 13:10:41 whispering-hurricane kernel: [ 8114.162707] audit: type=1400 audit(1548335441.898:130): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=6841 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 24 13:10:41 whispering-hurricane kernel: [ 8114.186619] audit: type=1400 audit(1548335441.922:131): apparmor="DENIED" operation="capable" profile="/usr/sbin/mysqld" pid=6841 comm="mysqld" capability=2 capname="dac_read_search"
Jan 24 13:10:41 whispering-hurricane kernel: [ 8114.217354] audit: type=1400 audit(1548335441.954:132): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=6850 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=113 ouid=0
Jan 24 13:10:42 whispering-hurricane systemd[1]: Started MySQL Community Server.
Is something in here causing the problem?
So, just to confirm:
You have generated a key using php artisan key:generate?
This key is in your .env file under a APP_KEY key?
Your config/app.php has a key 'key' => env('APP_KEY')?
If all of these are in place you should be right. Come back if there are any errors.
The reason you can't access the database might be due to UFW not allowing connection on your mysql port which I suppose is 3306. Could you SSH into your forge server and type:
ufw status
ufw allow 3306
You might need sudo permissions for this. The reason for this thinking is this line in your logs:
...
Jan 24 06:36:24 whispering-hurricane kernel: [1767450.357697] [UFW BLOCK] IN=eth0 OUT= MAC=32:f4:a6:e9:03:29:f0:4b:3a:4e:50:30:08:00 SRC=85.10.193.56 DST=206.189.227.234 LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=37186 DF PROTO=TCP SPT=57178 DPT=22123 WINDOW=29200 RES=0x00 SYN URGP=0
...
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
I wanted to check out my logserver again, but all of sudden i get this message when i visit the url:
I am running on ubuntu xenial
FATAL: Cannot connect to MySQL server on 'localhost'. Please make sure you have specified a valid MySQL database name in 'include/config.php'
when i do journalctl -xe i get the following message:
-- Unit mysql.service has begun starting up.
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
Jun 15 19:26:12 arjanlog kernel: audit: type=1400 audit(1497547572.504:69): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld"
name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_m
Jun 15 19:26:12 arjanlog kernel: audit: type=1400 audit(1497547572.504:70): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=3665 comm="mysqld" requested_mask="r"
Jun 15 19:26:12 arjanlog kernel: audit: type=1400 audit(1497547572.504:71): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_m
Jun 15 19:26:12 arjanlog systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
I had this problem too and solved it by fixing the apparmor configuration file at /etc/apparmor.d/usr.sbin.mysqld. I added these lines:
/proc/*/status r,
/sys/devices/system/node/ r,
/sys/devices/system/node/node0/meminfo r,
Your journalctl -xe output shows the files MySQL needs permission to use. You can also look in /var/log/syslog:
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
Jun 15 19:26:12 arjanlog audit[3665]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3665/status" pid=3665 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
That means that /usr/sbin/mysqld was DENIED trying to open /proc/3665/status and /sys/devices/system/node/ for reading (r).
Note that in the apparmor config file the trailing slash in /sys/devices/system/node/ is necessary, so don't leave it off!
It may not be necessary to stop apparmor entirely. It should be sufficient brute force to disable apparmor for mysqld, if you don't want to track down the core issue:
ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/usr.sbin.mysqld
For me the error was that the mysql database got corrupted. I could see that when I went to logs /var/log/mysql/error.log and it showed an error:
InnoDB: Database was not shutdown normally!
There were some solutions out there which someone could use like table corruption issue
But it didn't work for me as I wasn't able to start a server in WRITE mode in order to solve the issue.
I managed to solve the issue by creating a copy of directory /var/lib/mysql - I copied to mysql.old and then deleted the mysql directory.
Then I called a command in terminal:
sudo mysqld --initialize --initialize-insecure
To generate a new mysql directory
I have then changed permissions of new directory by changing the user and group of mysql directory to user: mysql and group: mysql. Use chown command:
sudo chown -R mysql:mysql /var/lib/mysql
Then I called command:
sudo mysql -u root
and changed password with:
SET PASSWORD FOR root#'localhost' = PASSWORD('newpassword');
And then restarted mysql server:
sudo service mysql restart
And everything worked.
Some of the commands that I have used along the way were (although not sure how much they contributed to fixing the issue):
sudo apt-get --reinstall install mysql-server-5.7
I read in another SO thread comment that the apparmor="DENIED" message probably isn't the reason that MySQL (or in my case MariaDB) wasn't starting, as it's only a warning.
In my case updating and upgrading apt-installed packages and rebooting the system solved the problem.
sudo apt-get dist-upgrade
sudo apt-get update
sudo apt-get upgrade
sudo reboot
Then you'll need to wait a few minutes while the system restarts and log back in and restart the service(s) with either service, systemctl and I think possibly /etc/system.d.
Then MariaDB was chugging away happily.
There's a good post on the various apt upgrade commands here, and I think that what update does is update the source lists.
I had found this command useful as well, when there were no log files in /var/log/mysql:
tail -f /var/log/syslog | grep mysql
I got this bug when I tried to move mysql file from /var/lib/mysql to somewhere else.
add config to /etc/apparmor.d/usr.sbin.mysqld didn't work for me
then i got this bug for ubuntu
it said:
sudo nano /etc/apparmor.d/usr.sbin.mysqld
line 25 after:
# Allow config access
/etc/mysql/** r,
-add line:
/etc/mysql/* r,
and it works for me:
add /etc/mysql/* r, to /etc/apparmor.d/usr.sbin.mysqld
run systemctl restart apparmor.service
and service mysql start
After I followed the instructions given in https://stackoverflow.com/a/45986591 I was still not able to make the MySql server run. I checked its logs via tail -30 /var/log/mysql/error.log . This line caught my attention 2019-05-21T04:46:03.462807Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool. By reading the previous line of code, I was that it was trying to allocate 128MB of memory. Then I checked my Memory status and it was only 111 Mb free. I stopped few programs (node servers) and restarted the MySql and it started working. Then i commented all the AppArmor entries I had entered and restarted AppArmor and MySql. MySql started without any errors. So I could conclude that MySql was not getting enough memory to get started.
solved by : sudo service apparmor teardown
I found this error on a fresh install of the Docker container provided by Mgt Development Environment:
root#mgt-dev-70:~# /etc/init.d/mysql start
/usr/sbin/mysqld: error while loading shared libraries: libpthread.so.0: cannot open shared object file: Permission denied
[....] Starting MySQL (Percona Server) database server: mysqld/usr/sbin/mysqld: error while loading shared libraries: libpthread.so.0: cannot open shared object file: Permission denied
/usr/sbin/mysqld: error while loading shared libraries: libpthread.so.0: cannot open shared object file: Permission denied
./usr/sbin/mysqld: error while loading shared libraries: libpthread.so.0: cannot open shared object file: Permission denied
./usr/sbin/mysqld: error while loading shared libraries: libpthread.so.0: cannot open shared object file: Permission denied
I used this tutorial https://www.mgt-commerce.com/documentation/mgt-development-environment-run-container and I pulled mgtcommerce/mgt-development-environment-7.0
As we see it is a problem with apparmor.
[192088.261031] audit: type=1400 audit(1516074899.710:1304): apparmor="DENIED" operation="open" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="var/lib/docker/aufs/diff/e5182def06a6a8b8d08c0a6d0c3a439585d2e7aea90507c245d421c319431d51/lib/x86_64-linux-gnu/libpthread-2.19.so" pid=18970 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[192088.261039] audit: type=1400 audit(1516074899.710:1305): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="var/lib/docker/aufs/diff/e5182def06a6a8b8d08c0a6d0c3a439585d2e7aea90507c245d421c319431d51/lib/x86_64-linux-gnu" pid=18970 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[192088.261066] audit: type=1400 audit(1516074899.710:1306): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="var/lib/docker/aufs/diff/e5182def06a6a8b8d08c0a6d0c3a439585d2e7aea90507c245d421c319431d51/usr/lib/x86_64-linux-gnu" pid=18970 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[192088.261083] audit: type=1400 audit(1516074899.710:1307): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="var/lib/docker/aufs/diff/e5182def06a6a8b8d08c0a6d0c3a439585d2e7aea90507c245d421c319431d51/lib" pid=18970 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[192088.261103] audit: type=1400 audit(1516074899.710:1308): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="var/lib/docker/aufs/diff/e5182def06a6a8b8d08c0a6d0c3a439585d2e7aea90507c245d421c319431d51/usr/lib" pid=18970 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
You can fix it by log in the container and run the commands below.
mv /usr/sbin/mysqld /usr/bin/mysqld
ln -nfs /usr/bin/mysqld /usr/sbin/mysqld
When I run command
mysql -u root -p
I got an error
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
When I try to run mysql service
service mysql start
it gets timed out with message
Job for mysql.service failed because a timeout was exceeded. See "systemctl status mysql.service" and "journalctl -xe" for details.
Logs
Jan 08 23:44:48 monal-Lenovo-IdeaPad-Z510 audit[30814]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=30814 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=123 ouid=0
Jan 08 23:44:48 monal-Lenovo-IdeaPad-Z510 audit[30814]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/30814/status" pid=30814 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=123 ouid=123
Jan 08 23:44:48 monal-Lenovo-IdeaPad-Z510 audit[30814]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/usr/my.cnf" pid=30814 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=123 ouid=0
Jan 08 23:44:48 monal-Lenovo-IdeaPad-Z510 kernel: audit: type=1400 audit(1483899288.523:210): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/30814/status" pid=30814 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=123 ouid=123
Jan 08 23:44:48 monal-Lenovo-IdeaPad-Z510 kernel: audit: type=1400 audit(1483899288.523:211): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=30814 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=123 ouid=0
Jan 08 23:44:48 monal-Lenovo-IdeaPad-Z510 kernel: audit: type=1400 audit(1483899288.523:212): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/30814/status" pid=30814 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=123 ouid=123
Jan 08 23:44:48 monal-Lenovo-IdeaPad-Z510 kernel: audit: type=1400 audit(1483899288.523:213): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/usr/my.cnf" pid=30814 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=123 ouid=0
Jan 08 23:44:48 monal-Lenovo-IdeaPad-Z510 mysqld_safe[30539]: 170108 23:44:48 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Jan 08 23:45:01 monal-Lenovo-IdeaPad-Z510 CRON[30867]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 08 23:45:01 monal-Lenovo-IdeaPad-Z510 CRON[30868]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jan 08 23:45:01 monal-Lenovo-IdeaPad-Z510 CRON[30867]: pam_unix(cron:session): session closed for user root
First stop the mysql instance. You will need to copy the exact /var/lib/mysql directory to your new data directory, preserving its permission. So use rsync with -a option
sudo rsync -av /var/lib/mysql /db/data/
Also change the data dir path in mysql.cnf file
You will need to add alias to apparmor so that it allows mysql to write to the new directory
To add alias:
sudo nano /etc/apparmor.d/tunables/alias
and then add this line to bottom
alias /var/lib/mysql/ -> /db/data/mysql
Restart the apparmor
sudo systemctl restart apparmor
and now start mysql. It should work
If your mysql is working and you want change your directory flow these steps:
change datadir from vim /etc/mysql/mysql.conf.d/mysqld.cnf
Change the owner or set high privilege for new directory
chown mysql.mysql /yourdir/mysql
You should change vim /etc/apparmor.d/tunables/alias
alias /var/lib/mysql/ -> /yourdb/mysql
finally add /db/data/mysql r and /db/data/mysql** rwk at the end of vim /etc/apparmor.d/usr.sbin.mysqld
/etc/init.d/apparmor reload
/etc/init.d/mysql start
But if your mysql doesn't work anymore remove it and the flow those steps.
If you need your data copy them to new directory.
remove steps:
apt-get autoremove mysql-server
apt-get remove --purge mysql-\*
apt-get install mysql-server
I add this answer to others who didn't find satisfying answer.
First, follow a simple tutorial to copy and set aliases and configs to the new desired data directory (datadir).
The tutorial shows how to:
Stopping the mysql service,
Backup & Copy existing database to the new location
Adding AppArmor rules.
Rebooting the service.
Then, I had this exact issue (AppArmor access denied) when moving my datdir to /media/data/mysql/, and the final step (restart mysql service) failed due to AVC apparmor="DENIED" operation="open" prof......
First, as #reihaneh mentioned, add AppArmor reading and writing rules:
Add /db/data/mysql r, and /db/data/mysql** rwk, (don't remove the commas) at the end of vim /etc/apparmor.d/usr.sbin.mysqld
Finally, the only thing left to do (not mentioned in tutorials) is to change the privileges for the new datadir to the MySQL-Server user and mysql group.
I used nautilus to do so:
Run nautilus as SU (sudo nautilus) -> right click on the NEW datadir -> Properties -> Permissions
Then choose owner: MySQL - MySQL Server
And group: mysql
Just run this command
sudo mysql_install_db --user=mysql --basedir=/usr/ --ldata=/var/lib/mysql/