Docker run command with -v flag puts container in Exited status - mysql

I am trying to map a local directory /home/ubuntu/data to /var/lib/mysql folder in container by using -v flag but container's status becomes Exited (0) 1. However, if I don't use -v flag at all, container is Up but this is not what I want. What could be the reason? I see volume mount line is missing in event logs opposed to working example.
$ docker -v
Docker version 17.09.0-ce, build afdb6d4
Dockerfile
FROM ubuntu:16.04
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y mysql-server \
&& sed -i "s/127.0.0.1/0.0.0.0/g" /etc/mysql/mysql.conf.d/mysqld.cnf \
&& mkdir /var/run/mysqld \
&& chown -R mysql:mysql /var/run/mysqld
VOLUME ["/var/lib/mysql"]
EXPOSE 3306
CMD ["mysqld_safe"]
This is the which doesn't work.
$ docker run -i -t -d -v /home/ubuntu/data:/var/lib/mysql --name mysql_container mysql_image
Event logs.
2017-11-... container create 08b44c094... (image=mysql_image, name=mysql_container)
2017-11-... network connect 62bb211934... (container=08b44c094..., name=bridge, type=bridge)
2017-11-... container start 08b44c094... (image=mysql_image, name=mysql_container)
2017-11-... container die 08b44c094... (exitCode=0, image=mysql_image, name=mysql_container)
2017-11-... network disconnect 62bb211934... (container=08b44c094..., name=bridge, type=bridge)
Container logs.
$ docker logs -t mysql_container
2017-11-... mysqld_safe Logging to syslog.
2017-11-... mysqld_safe Logging to '/var/log/mysql/error.log'.
2017-11-... mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
This works without -v
$ docker run -i -t -d --name mysql_container mysql_image
Event logs.
2017-11-... container create 84993141... (image=mysql_image, name=mysql_container)
2017-11-... network connect 62bb2119... (container=84993141..., name=bridge, type=bridge)
2017-11-... volume mount 8c36b53d33... (container=84993141...7, destination=/var/lib/mysql, driver=local, propagation=, read/write=true)
2017-11-... container start 84993141... (image=mysql_image, name=mysql_container)
Container logs.
$ docker logs -t mysql_container
2017-11-... mysqld_safe Logging to syslog.
2017-11-... mysqld_safe Logging to '/var/log/mysql/error.log'.
2017-11-... mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2017-11-... mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

It's a little complicated but interesting case.
So how you can check what's happening? Use following command:
docker run -i -t -v /tmp/data:/var/lib/mysql mysql_image bash
Now you are inside container so let's try command:
mysqld_safe
And it's ending but let's look into /var/log/mysql/error.log
We see there:
2017-11-25T17:22:24.006180Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2017-11-25T17:22:24.006211Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2017-11-25T17:22:24.006221Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2017-11-25T17:22:24.006229Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2017-11-25T17:22:24.006237Z 0 [ERROR] InnoDB: Cannot open datafile './ibdata1'
Ok let's see how /var/lib/mysql looks without volume mapping:
root#4474b1cd4300:/var/lib/mysql# ls -lah
total 109M
drwx------ 5 mysql mysql 4.0K Nov 25 17:24 .
drwxr-xr-x 1 root root 4.0K Nov 25 17:13 ..
-rw-r----- 1 mysql mysql 56 Nov 25 17:13 auto.cnf
-rw-r--r-- 1 root root 0 Nov 25 17:13 debian-5.7.flag
-rw-r----- 1 mysql mysql 419 Nov 25 17:13 ib_buffer_pool
-rw-r----- 1 mysql mysql 48M Nov 25 17:13 ib_logfile0
-rw-r----- 1 mysql mysql 48M Nov 25 17:13 ib_logfile1
-rw-r----- 1 mysql mysql 12M Nov 25 17:13 ibdata1
drwxr-x--- 2 mysql mysql 4.0K Nov 25 17:13 mysql
drwxr-x--- 2 mysql mysql 4.0K Nov 25 17:13 performance_schema
drwxr-x--- 2 mysql mysql 12K Nov 25 17:13 sys
mysql:mysql is owner of that directory
We have a lot mysql specific files there
Let's see what we've got with volume mapping:
root#fca45ee1e8fb:/var/lib/mysql# ls -lah
total 8.0K
drwxr-xr-x 2 root root 4.0K Nov 25 17:22 .
drwxr-xr-x 1 root root 4.0K Nov 25 17:13 ..
Docker is mapping this directory as root user
Docker is mapping this directory into host so all files disappear because on host machine that directory is empty
How to get this work?
Change your command to:
CMD chown -R mysql:mysql /var/lib/mysql && if [ ! -c /var/lib/mysql/ibdata1 ]; then mysqld --initialize-insecure; fi && mysqld_safe
What's happening there?
chown -R mysql:mysql /var/lib/mysql - get back mysql:mysql owner
if [ ! -c /var/lib/mysql/ibdata1 ]; then mysqld --initialize-insecure; fi - recreate mysql files with root user without pass but only if files not already exists (required for next runs)
mysqld_safe - run mysql

Related

Error with ibdata1 write when starting customised Docker MySQL container as non-root user

I need to start a MySQL container based on a MySQL image having an existing database as part of the image and set as the default database. It needs to run with non-root user of mysql since running as root is not permitted on our private Kubernetes cluster. Referencing another solution from SO for starting MySQL with a pre-existing database, created below Dockerfile. It started up a container successfully locally on Docker desktop UNTIL I made changes to try and make container runnable as user mysql.
The database schema was output from existing database on a VM using mysqdump and out to eddie_backup2.sql.
Dockerfile:
FROM containerregistry-na.foocompany/container-external/mysql:5.7.29 as builder
# That file does the DB initialization but also runs mysql daemon, by removing the last line it will only init
RUN ["sed", "-i", "s/exec \"$#\"/echo \"not running $#\"/", "/usr/local/bin/docker-entrypoint.sh"]
ENV MYSQL_ALLOW_EMPTY_PASSWORD="y"
ENV MYSQL_USER="eddie" MYSQL_PASSWORD="eddie_pwd" MYSQL_DATABASE="eddie"
ADD eddie_backup2.sql /tmp/eddie_backup2.sql
COPY setup.sql docker-entrypoint-initdb.d/
# Need to change the datadir to something else that /var/lib/mysql because the parent docker file defines it as a volume.
# https://docs.docker.com/engine/reference/builder/#volume :
# Changing the volume from within the Dockerfile: If any build steps change the data within the volume after
# it has been declared, those changes will be discarded.
RUN ["/usr/local/bin/docker-entrypoint.sh", "mysqld", "--datadir", "/initialized-db" ]
# added below line to change ownership
RUN ["/bin/bash", "-c", "chown -R mysql:mysql /initialized-db/"]
# starting with mysql image again and using the generated datadirectory from above interim image
FROM containerregistry.foocompany.net/container-external/mysql:5.7.29 as actual_base
COPY --from=builder /initialized-db /var/lib/mysql
# change owner to mysql and list immediately to verify it was done
RUN ["/bin/bash", "-c", "chown -R mysql:mysql ./var/lib/mysql/ -v && ls -lrt /var/lib/mysql"]
USER mysql
CMD mysqld --datadir=/var/lib/mysql --user=mysql
MySQL script setup.sql run at initialisation, as it is located in special directory where the process looks:
use eddie;
source /tmp/eddie_backup2.sql ;
However, the logs indicated an issue with permissions to write to Innodb* folders. I think these are or should be present under /var/lib/mysql . That is as far as I got.
docker build --no-cache -t eddie-mysql:0.3 .
Logs:
changed ownership of './var/lib/mysql/performance_schema/file_summary_by_event_n
ame.frm' from root:root to mysql:mysql
changed ownership of './var/lib/mysql/performance_schema/events_transactions_sum
mary_by_thread_by_event_name.frm' from root:root to mysql:mysql
changed ownership of './var/lib/mysql/performance_schema/hosts.frm' from root:ro
ot to mysql:mysql
changed ownership of './var/lib/mysql/performance_schema' from root:root to mysq
l:mysql
changed ownership of './var/lib/mysql/ib_buffer_pool' from root:root to mysql:my
sql
changed ownership of './var/lib/mysql/ca.pem' from root:root to mysql:mysql
changed ownership of './var/lib/mysql/private_key.pem' from root:root to mysql:m
ysql
changed ownership of './var/lib/mysql/ibdata1' from root:root to mysql:mysql
changed ownership of './var/lib/mysql/auto.cnf' from root:root to mysql:mysql
changed ownership of './var/lib/mysql/client-key.pem' from root:root to mysql:my
sql
ownership of './var/lib/mysql/' retained as mysql:mysql
total 176196
-rw------- 1 mysql mysql 1680 Oct 2 15:07 server-key.pem
-rw-r--r-- 1 mysql mysql 1112 Oct 2 15:07 server-cert.pem
-rw-r----- 1 mysql mysql 50331648 Oct 2 15:07 ib_logfile1
-rw-r--r-- 1 mysql mysql 1112 Oct 2 15:07 ca.pem
-rw------- 1 mysql mysql 1676 Oct 2 15:07 ca-key.pem
-rw-r----- 1 mysql mysql 56 Oct 2 15:07 auto.cnf
-rw------- 1 mysql mysql 1680 Oct 2 15:07 client-key.pem
-rw-r--r-- 1 mysql mysql 1112 Oct 2 15:07 client-cert.pem
-rw-r--r-- 1 mysql mysql 452 Oct 2 15:07 public_key.pem
-rw------- 1 mysql mysql 1680 Oct 2 15:07 private_key.pem
-rw-r----- 1 mysql mysql 79691776 Oct 2 15:07 ibdata1
-rw-r----- 1 mysql mysql 50331648 Oct 2 15:07 ib_logfile0
-rw-r----- 1 mysql mysql 1452 Oct 2 15:07 ib_buffer_pool
drwxr-x--- 2 mysql mysql 12288 Oct 2 15:07 sys
drwxr-x--- 2 mysql mysql 4096 Oct 2 15:07 performance_schema
drwxr-x--- 2 mysql mysql 4096 Oct 2 15:07 mysql
drwxr-x--- 2 mysql mysql 4096 Oct 2 15:07 eddie
Removing intermediate container 29e35ac511ea
---> ce46892514e4
Step 13/14 : USER mysql
---> Running in fd1831317581
Removing intermediate container fd1831317581
---> ae9d3e300cbf
Step 14/14 : CMD mysqld --datadir=/var/lib/mysql --user=mysql
---> Running in 17143095e06f
Removing intermediate container 17143095e06f
---> 9712fc738c4c
Successfully built 9712fc738c4c
Successfully tagged eddie-mysql:0.3
It can be seen above ibdata1 ownership changed to mysql. This is relevant later . .
docker run -d --name abc eddie-mysql:0.3
docker logs 746a210065840
Below log indicates ibdata is not writeable by user mysql even though according to image build log it is owned by mysql !
2020-10-02T15:13:08.264040Z 0 [Note] InnoDB: Completed initialization of buffer
pool
2020-10-02T15:13:08.265201Z 0 [Note] InnoDB: If the mysqld execution user is aut
horized, page cleaner thread priority can be changed. See the man page of setpri
ority().
2020-10-02T15:13:08.275162Z 0 [ERROR] InnoDB: The innodb_system data file 'ibdat
a1' must be writable
2020-10-02T15:13:08.275231Z 0 [ERROR] InnoDB: The innodb_system data file 'ibdat
a1' must be writable
2020-10-02T15:13:08.275263Z 0 [ERROR] InnoDB: Plugin initialization aborted with
error Generic error
2020-10-02T15:13:08.876474Z 0 [ERROR] Plugin 'InnoDB' init function returned err
or.
2020-10-02T15:13:08.876491Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE
ENGINE failed.
2020-10-02T15:13:08.876494Z 0 [ERROR] Failed to initialize builtin plugins.
2020-10-02T15:13:08.876496Z 0 [ERROR] Aborting
2020-10-02T15:13:08.876500Z 0 [Note] Binlog end
2020-10-02T15:13:08.876723Z 0 [Note] Shutting down plugin 'CSV'
2020-10-02T15:13:08.877008Z 0 [Note] mysqld: Shutdown complete
This may not be the most elegant solution but as mentioned earlier I could see that user mysql owns the file as a result of the chown added to my dockerfile. However discovered, it did not have write permission to it (confimred that after temporarily adding RUN ls -lrt /var/lib/mysql -v to list folder perms for debugging purposes ) which makes sense given the error message. Seems there is no publicly available image that takes care of this use case of starting a mySQL container as non root user.
So amended my Dockerfile to give most priveleged permissions to file ibdata1 (as well as the containing folder for good measure) right after mysqld initialisation with no-default data directory:
RUN ["/usr/local/bin/docker-entrypoint.sh", "mysqld", "--datadir", "/initialized-db" ]
RUN ["/bin/bash", "-c", "chown -R mysql:mysql /initialized-db/"]
RUN ["/bin/bash", "-c", "chmod ugo=rwx -R /initialized-db/"]
RUN chmod -R ugo+rwx /initialized-db/ibdata1
Here is the pertinent part of the build log:
Step 9/13 : RUN ["/bin/bash", "-c", "chown -R mysql:mysql /initialized-db/"]
---> Running in 973c96b0f535
Removing intermediate container 973c96b0f535
---> f190deb49406
Step 10/13 : RUN ["/bin/bash", "-c", "chmod ugo=rwx -R /initialized-db/"]
---> Running in 2e4612d7674c
Removing intermediate container 2e4612d7674c
---> efa6715342e2
Step 11/13 : RUN chmod -R ugo+rwx /initialized-db/ibdata1
---> Running in 3c2e288c19b7
Removing intermediate container 3c2e288c19b7
---> 1c0e7a32b2a4
Step 12/13 : FROM some-private-registry.net/container-external/mysql:5.7
.29 as actual_base
---> 5d9483f9a7b2
Step 13/13 : COPY --from=builder /initialized-db /var/lib/mysql
---> 19f51e56ae40
I could then run the image as user mysql:
docker container run -d --user mysql --name foo_name --user mysql foo-mysql:1.0

MySQL in docker won't persist changes to volume configured

I am trying to run the offical docker image of Mysql 5.7.28 on MacOS, but I cannot manage to make it persistent.
I used mount point on docker host as well volume created with docker volume, but it doesn't work.
I use the following to create the container:
docker run \
--detach \
--name=dockMysql \
--env="MYSQL_ROOT_PASSWORD=password" \
--publish 127.0.0.1:3307:3306 \
--volume=/Users/myuser/docker/dockMysql/data:/var/lib/mysql \
mysql:5.7.28
The data path: /Users/myuser/docker/dockMysql/data has all per
$ ls -l
drwxrwxrwx 6 root admin 192 May 24 2019 Users
$ pwd
/Users/myuser/docker/dockMysql/data
$ ls
auto.cnf ca.pem client-key.pem ib_logfile0 ibdata1 mysql private_key.pem server-cert.pem sys
ca-key.pem client-cert.pem ib_buffer_pool ib_logfile1 ibtmp1 performance_schema public_key.pem server-key.pem
It seems mysql writes data in the host directory provided but doesn't save the data upon restart of the container.
Anyone has any idea?
Thanks,
Ionut

mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid prevent from server restart

My mySQL server on CentOS has been working correctly,
but, I am unable to restart mysqld suddenly today.
1)
# /etc/rc.d/init.d/mysqld start
shows [failed]
2) see the log
tail /var/log/mysqld.log
.
.
mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
3) manually put
/usr/bin/mysqld_safe
151129 15:54:36 mysqld_safe Logging to '/var/log/mysqld.log'.
151129 15:54:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151129 15:54:37 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
4)check config
less /etc/my.cnf
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Recommended in standard MySQL setup
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
5)check mysql directory,every owner ship is belonging to mysql. However I can't fine mysql.sock (this is set in my.cnf)
cd /var/lib/mysql 
ls -la
drwxr-xr-x 5 mysql mysql 4096 11月 29 15:54 2015 .
drwxr-xr-x 21 root root 4096 6月 2 06:09 2015 ..
-rw-rw---- 1 mysql mysql 56 6月 2 05:42 2015 auto.cnf
drwx------ 2 mysql mysql 4096 11月 24 11:12 2015 myapp
-rw-rw---- 1 mysql mysql 50331648 11月 29 12:30 2015 ib_logfile0
-rw-rw---- 1 mysql mysql 50331648 11月 29 12:30 2015 ib_logfile1
-rw-rw---- 1 mysql mysql 79691776 11月 29 12:30 2015 ibdata1
drwx------ 2 mysql mysql 4096 6月 2 05:42 2015 mysql
drwx------ 2 mysql mysql 4096 6月 2 05:42 2015 performance_schema
6) I found there is no mysql.sock in /var/lib/mysql directory, then I try this for test purpose.
touch /var/lib/mysql/mysql.sock
try to restart
/etc/rc.d/init.d/mysqld start
somehow mysql.sock is deleted.
Is there any other things I can try??
In my case the problem was that mysql tried to create temporary files in /var/run/mysqld and that directory did not exist. I solved the problem by creating the directory manually and setting permissions for it:
mkdir -p /var/run/mysqld
chown mysql:mysql /var/run/mysqld
Actually, Mairadb locate pid file in /var/run/mariadb.
Then,you needn't any other operation just set pid-file=/var/run/mariadb/mysqld.pid
I googled around and try.
restorecon -r /var/lib/mysql
it works for me.
Thanks.
Just reinstall mariadb. This will correct all the permissions.

Mounting container volume from the hosts' drive?

im setting up a mysql container like so:
docker run -v /srv/information-db:/var/lib/mysql tutum/mysql /bin/bash -c "/usr/bin/mysql_install_db"
now, this works when nothing is mounted on /srv on the host, but when i mount my drive, docker seems to write to the underlying filesystem (/), eg:
/]# ls -l /srv
total 0
/]# mount /dev/xvdc1 /srv
/]# mount
...
/dev/xvdc1 on /srv type ext4 (rw,relatime,seclabel,data=ordered)
/]# docker run -v /srv/information-db:/var/lib/mysql tutum/mysql /bin/bash -c "/usr/bin/mysql_install_db"
/]# ls -l /srv
total 16
drwx------. 2 root root 16384 Apr 22 18:05 lost+found
/]# umount /dev/xvdc1
/]# ls -l /srv
total 4
drwxr-xr-x. 4 102 root 4096 Apr 22 18:24 information-db
Anyone seen this behaviour / have a solution?
Cheers
I've seen something like that. Try to perform stat -c %i checks both inside the host and container before and after mount event (in order to get inode values of the target dirs). I guess they're mismatched for a some reason when you mount external device.

MySQL installation from deb, permissions issues

I am trying to install MySQL on ubuntu 14.04 from the deb packages. I am having trouble starting the mysql server, it looks like a permissions problem. I followed the steps outlined here. Downloaded/untarred/installed deb tar bundle.
sudo apt-get install libaio1
tar -xvf mysql-server_5.7.4-m14-2ubuntu14.04_amd64.deb-bundle.tar
md5sum mysql-server_5.7.4-m14-2ubuntu14.04_amd64.deb-bundle.tar
sudo dpkg -i mysql-common_5.7.4-m14-2ubuntu14.04_amd64.deb
sudo dpkg -i mysql-community-server_5.7.4-m14-2ubuntu14.04_amd64.deb
sudo dpkg -i mysql-community-client_5.7.4-m14-2ubuntu14.04_amd64.deb
sudo dpkg -i libmysqlclient18_5.7.4-m14-2ubuntu14.04_amd64.deb
Here is where the files are installed on my system:
All configuration files (like my.cnf) are under /etc.
All binaries, libraries, headers, etc., are under /usr.
The data directory is under /var.
Following these instructions
I create a mysql group and user:
groupadd mysql
useradd -r -g mysql mysql
I change the ownership of mysql scripts to mysql (as per the instructions, but doubt this is necesssary)
cd /usr/bin
sudo chown mysql mysq*
sudo chgrp mysql mysq*
I run mysql_install_db to set up grant tables
sudo mysql_install_db --user=mysql
I switch back to root the ownership and group of mysql scripts.
cd /usr/bin
sudo chown root mysq*
sudo chgrp root mysq*
I change the ownership and group of /data to mysql. Location is /var/lib/mysql
cd /var/lib
ls -l mysql
total 122896
-rw-rw-rw- 1 mysql mysql 56 Jul 26 10:17 auto.cnf
-rw-rw-rw- 1 mysql mysql 12582912 Jul 26 10:17 ibdata1
-rw-rw-rw- 1 mysql mysql 50331648 Jul 26 10:17 ib_logfile0
-rw-rw-rw- 1 mysql mysql 50331648 Jul 24 17:36 ib_logfile1
-rw-rw---- 1 mysql mysql 12582912 Jul 26 10:17 ibtmp1
drwxrwxrw- 2 mysql mysql 4096 Jul 24 17:36 mysql
drwxrw-rw- 2 mysql mysql 4096 Jul 24 17:36 performance_schema
drwxrw-rw- 2 mysql mysql 4096 Jul 24 17:36 test
Now, when I try to start the mysql server, I get permission errors:
mysqld_safe --user=mysql &
[4] 5680
/var/lib$ 140727 00:42:17 mysqld_safe Logging to '/var/log/mysql/error.log'.
cat: /var/run/mysqld/mysqld.pid: Permission denied
rm: cannot remove ‘/var/run/mysqld/mysqld.pid’: Permission denied
140727 00:42:17 mysqld_safe Fatal error: Can't remove the pid file:
/var/run/mysqld/mysqld.pid
Please remove it manually and start /usr/bin/mysqld_safe again;
mysqld daemon not started
/usr/bin/mysqld_safe: 129: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
Similarly, if I try to start the server as root:
mysqld_safe -p -u root
140727 00:54:08 mysqld_safe Logging to '/var/log/mysql/error.log'.
140727 00:54:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/bin/mysqld_safe: 129: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
/usr/bin/mysqld_safe: 1: eval: cannot create /var/log/mysql/error.log: Permission denied
140727 00:54:08 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
/usr/bin/mysqld_safe: 129: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
Apparently there are many places where I should be changing permissions, which does not look like the way to go. Three questions:
Is there some evident point where I went wrong, or should I just uninstall and apt-get everything?
Should I have owner mysql and group mysql for all files that the server needs to update?
Is there a comprehensive list of locations where these files are?
You cannot create /var/log/mysql/error.log. First create the directory if it doesn't exist
sudo mkdir -p /var/log/mysql
Next, change ownership
sudo chown -R mysql /var/log/mysql
and then try again to start the server as root:
sudo mysqld_safe --user=mysql &
If the server does not start paste here the error messages