How can I enable MySQL binary logging using the official Docker image? - mysql

What would be the best way to enable binary logging using the official mysql image?
I have tried using the mysql:5.7 image, overriding the command when running it to also pass through the startup options to enable binary logging to mysqld (see below). The problem with this approach is that the mysql user does not have permission to write to the /var/log/mysql directory.
The run command:
docker run -d \
--name mysql \
-v /var/lib/mysql:/var/lib/mysql \
mysql:5.7 \
mysqld \
--datadir=/var/lib/mysql \
--user=mysql \
--server-id=1 \
--log-bin=/var/log/mysql/mysql-bin.log \
--binlog_do_db=test
The output:
mysqld: File '/var/log/mysql/mysql-bin.index' not found (Errcode: 2 - No such file or directory)
Should I fork the repository and add a volume for /var/log/mysql which the mysql user can write to and create a custom image, or is there a better way to do it? Is this possible using only the official mysql image?

The problem with this approach is that the mysql user does not have permission to write to the /var/log/mysql directory
The problem actually is that the directory /var/log/mysql does not exists on the mysql:5.7 Docker image. You can make sure of it running the following container:
$ docker run --rm mysql:5.7 ls /var/log/
alternatives.log
apt
bootstrap.log
btmp
dmesg
dpkg.log
faillog
fsck
lastlog
wtmp
Furthermore, MySQL binary logs aren't logs meant for following your MySQL server activity or errors ; they are logs meant to give your MySQL server a chance to recover data in case of a server crash.
As a consequence, you want those binary logs:
to stay close to your data
to be written on a fast file system
In most cases, Docker container file system is slow and that's why the MySQL data folder for the container is declared as a VOLUME.
So you also want your binary logs to be written on a Docker data volume and not the Docker container file system.
long story short, start your container with:
docker run -d \
--name mysql \
-v /var/lib/mysql:/var/lib/mysql \
mysql:5.7 \
mysqld \
--datadir=/var/lib/mysql \
--user=mysql \
--server-id=1 \
--log-bin=/var/lib/mysql/mysql-bin.log \
--binlog_do_db=test

Related

Docker Using a custom MySQL configuration file Always fail

Follow the documentation here,Always fail
Using a custom MySQL configuration file
The default configuration for MySQL can be found in /etc/mysql/my.cnf, which may !includedir additional directories such as /etc/mysql/conf.d or /etc/mysql/mysql.conf.d. Please inspect the relevant files and directories within the mysql image itself for more details.
If /my/custom/config-file.cnf is the path and name of your custom configuration file, you can start your mysql container like this (note that only the directory path of the custom config file is used in this command):
$ docker run --name some-mysql -v /my/custom:/etc/mysql/conf.d -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag
This will start a new container some-mysql where the MySQL instance uses the combined startup settings from /etc/mysql/my.cnf and /etc/mysql/conf.d/config-file.cnf, with settings from the latter taking precedence.
Configuration without a cnf file
Many configuration options can be passed as flags to mysqld. This will give you the flexibility to customize the container without needing a cnf file. For example, if you want to change the default encoding and collation for all tables to use UTF-8 (utf8mb4) just run the following:
$ docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
If you would like to see a complete list of available options, just run:
My version Docker version 18.09.7, build 2d0083d
The commands I run
docker run --name mysql2 -v /my/custom:/etc/mysql/conf.d -e MYSQL_ROOT_PASSWORD=root -d mysql:8.0.16
My custom configuration file
[mysqld]
sql_mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"
I found the reason. Because I set the mysql.cnf permissions to 777.

How to redirect mysql log to /dev/stdout inside container?

I am trying to write my own mariadb-alpine docker image. Everything works file but while I am trying to collect the mariadb logs I am getting nothing. I tried to follow a lot of related issue like this and tried those but in vain.
FROM alpine:edge
COPY my.cnf /etc/mysql/my.cnf
RUN set -ex \
&& apk add mariadb mariadb-client shadow \
&& ln -snf /usr/lib/mariadb /usr/lib/mysql \
&& mysql_install_db --user=mysql --skip-name-resolve --auth-root-authentication-method=socket --auth-root-socket-user=root --force --rpm --skip-test-db \
&& usermod -a -G tty mysql \
&& ln -sf /dev/stdout /var/log/mysqld.err \
&& chown -h mysql:mysql /var/log/mysqld.err
CMD ["mysqld_safe"]
EXPOSE 3306
Is it required to mysqld take pid=1 to work stdout ? In my case it is some like below.
# ps aux
PID USER TIME COMMAND
1 root 0:00 {mysqld_safe} /bin/sh /usr/bin/mysqld_safe
134 mysql 0:00 /usr/bin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mariadb/plugin --user=mysql --log-error=/var/log/mysqld.err --pid-file=49ea99ae9348.p
166 root 0:00 sh
171 root 0:00 ps aux
You're likely running into these issues:
https://github.com/moby/moby/issues/31243
https://github.com/moby/moby/issues/31106
Something with alpine breaks the ability to access /dev/stdout when you change user accounts. The workaround I've used involves:
Running the container with a tty
Adding the user inside the container to the tty group
Starting the command with a gosu/exec to replace the pid 1 shell script with your app
I'm not sure if the last part was required, and you may not have access to do this with the mysql command. You're already doing the second item. That just leaves the first item that you can implement with:
docker run -t your_image
or in a compose file:
services:
mysql:
image: your_image
tty: true
....
The only other option is to run your application directly as mysql instead of starting it as root with user: mysql in the compose file, but that may not be supported by mysql itself.
If none of those work, the option used by the official image is to pick a debian base image instead of the alpine image. You can see their Dockerfile here:
https://github.com/docker-library/mysql/blob/696fc899126ae00771b5d87bdadae836e704ae7d/8.0/Dockerfile

How to pre-configure and prefill official MySQL docker container?

I want to build a new MySQL image based on the official MySQL docker container image. I want to reduce the number of parameters I need to add when running the image. (eg. -e MYSQL_USER, -e MYSQL_DATABASE and even -e MYSQL_ROOT_PASSWORD='rootsecret')
that already includes my settings for the global variables and my Create Database SQL file in the docker-entrypoint-initdb.d folder.
How do I add all my settings and create a new image to simply docker run mysql:config1 docker run mysql:config2 and so on?
You could build your own mysql docker image using a docker file, configure username, password and everything else you might need, build that image, upload it to the docker hub and then when you launch a new docker container you just use the previously built container.
An example of a Docker file to build an ubuntu image with a mysql server inside would be something like bellow (save it to a file called Dockerfile):
FROM ubuntu:latest
RUN apt-get update \
&& apt-get install -y apt-utils \
&& { \
echo debconf debconf/frontend select Noninteractive; \
echo mysql-community-server mysql-community-server/data-dir \
select ''; \
echo mysql-community-server mysql-community-server/root-pass \
password 'Desired-Password'; \
echo mysql-community-server mysql-community-server/re-root-pass \
password 'Desired-Password'; \
echo mysql-community-server mysql-community-server/remove-test-db \
select true; \
} | debconf-set-selections \
&& apt-get install -y mysql-server mysql-client
Then build your mysql docker container like this (you have to be in the folder where the Docker file was/is saved):
docker build my-ubuntu-mysql-docker
Then you have to push it to the docker hub and then you can use it to start a new docker container like this:
docker run -d -p 2222:22 -p 3306:3306 --name my-ubuntu-mysql-docker ...
Where 2222 is local ssh port mapped to ssh port 22 of the docker container and 3306 is local mysql port mapped to the mysql port of the docker container.
I hope this helps!
The following has to be written into the Dockerfile:
FROM mysql:latest
LABEL Name=mylabel Version=0.0.1
COPY path/to/sh/sql/sql.gz/files /docker-entrypoint-initdb.d
ENV MYSQL_ROOT_PASSWORD='rootpassword'
As stated in the documentation on the official docker website:
When a container is started for the first time, a new database with
the specified name will be created and initialized with the provided
configuration variables. Furthermore, it will execute files with
extensions .sh, .sql and .sql.gz that are found in
/docker-entrypoint-initdb.d. Files will be executed in alphabetical
order.
What you you would want to do is to modify Mysql's image entry point
and please note that you do not need to pass all the parameters, most of them are optional

How do I run mysqld as root in the official MySql docker image?

First off, I know not to run as root normally. I have an abnormal situation: I need to use mysqldump with the --tab argument, which requires permission to write to disk, and I want to use those files outside the Docker container. I could explain why running mysqld as root makes this easier, but isn't this question long enough? Running as root is safe in this case because the container will be used only for running tests and for updating DB backup scripts based on SQL migration scripts, and it will be started to do 1 job and then taken back down again.
When I google for how to run mysqld as root, I find the answer indirectly given in instructions on how to NOT run as root. Among other things in order to run mysqld as user_name:
Start the server as user user_name. Another alternative is to start mysqld as the Unix root user and use the --user=user_name option.
To start the server as the given user automatically at system startup time, specify the user name by adding a user option to the [mysqld] group of the /etc/my.cnf option file or the my.cnf option file in the server's data directory.
Do we do one of those? Both of those? I'll assume both just in case. But do they really mean /etc/my.cnf, or does that depend on the installation (e.g. what Linux distribution)? E.g. Docker image mysql:5.6 has /etc/mysql/my.cnf. The directions for the MySql Docker image advise mounting a volume at /etc/mysql/conf.d which is referenced in the aforementioned my.cnf. (Doing so overwrites 2 configuration files that are there by default, so I used a COPY command in my Dockerfile instead to merely add a config file.) The file does make it into the container:
root#4f612d10a690:/etc/mysql/conf.d# cat my.cnf
[mysqld]
user=root
One further requirement from the MySql manual is to add the --user=root argument to mysqld. The official MySql image calls mysqld via its CMD, so I override that in my Dockerfile. My CMD command does indeed run (it is run in 2 places in official MySql image's entrypoint script):
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mysql 1 0.1 2.8 1452788 472756 ? Ssl 14:24 0:01 mysqld --user=root
Note that mysqld has the --user=root command I provided, but is running as the mysql user, not as root.
Here's my full Dockerfile:
FROM mysql:5.6
VOLUME ["/var/lib/mysql-files"]
COPY ["my.cnf", "/etc/mysql/conf.d"]
CMD ["mysqld", "--user=root"]
My only guess as to why it's not running as root is that they mysql image's entrypoint script changes to the mysql user before running:
# allow the container to be started with `--user`
if [ ...blah... -a "$(id -u)" = '0' ]; then
...blah...
exec gosu mysql "$BASH_SOURCE" "$#"
fi
The above snippet basically says, if the user is root, then run the supplied arguments (the CMD + args in this case) as the mysql user.
Is running mysqld as root simply not supported by the official MySql Docker image?
Note: this is how to run mysqld process as SO's root user, and not how to get the root MySQL user.
I don't know whether exists a better approach but this works.
Viewing the official entrypoint.sh, it seems that it has no support of chaging the default mysql user
I realized how to run mysql as root but you need to have already initialized the data directory.
Step 1) Start a normal mysql in order to initialize a volume (the mysql entrypoint.sh will do that job):
docker run \
--rm \
-v $(pwd)/mysql/:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD="abc" \
mysql:5.6
Step 2) Stop and remove that container:
docker stop <container-id>
Step 3) Start again a new mysql process based on the data dir that has been created, but this time avoid to run the official mysql entrypoint:
docker run \
-v $(pwd)/mysql/:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD="abc" \
--entrypoint mysqld \
mysql:5.6 \
--user root
Step 4) Check it:
▶ docker exec -it 4add4d065c3e bash
root#4add4d065c3e:/# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 2.8 23.0 1314864 471104 ? Ssl 15:12 0:00 mysqld --user root
root 28 3.0 0.1 20248 3040 ? Ss 15:12 0:00 bash
root 34 0.0 0.1 17500 2068 ? R+ 15:12 0:00 ps aux

docker cp permissions are wrong when you mount the directory back into container

I am trying to grab the /var/lib/mysql directory out of a running mysql container in order to keep it for persistence. Yet when I then mount the directory as a volume mysql complains that it does not have permission:
mysqld: Table 'mysql.plugin' doesn't exist
2016-02-29 13:34:19 1 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
and
2016-02-29 13:34:20 1 [Note] Server socket created on IP: '::'.
2016-02-29 13:34:20 1 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist
docker cp `cat mysqlinitCID`:/var/lib/mysql datadir/
at this point I'm even using the tar form of docker cp in combination with -p flag to preserve permissions, yet it still comes in with privilege issues.
docker cp `cat mysqlinitCID`:/var/lib/mysql - |sudo tar -C datadir/mysql/ -pxf -
It seems like the only way to do this properly is go around docker cp and mount say /tmp and tar up /var/lib/mysql yourself and copying the tarball to the mounted point and get it out that way.
Is there a proper way to utilize docker cp that I am missing out on?
I initialize the mysql container like this to grab from:
docker run \
--name=$(NAME)-mysql-init \
-d \
--env='DB_NAME=$(DB_NAME)' \
--cidfile="mysqlinitCID" \
--env='MYSQL_USER=$(DB_USER)' --env="MYSQL_ROOT_PASSWORD=$(DB_PASS)" \
--env="MYSQL_PASSWORD=$(DB_PASS)" \
--env="MYSQL_DATABASE=$(DB_NAME)" \
mysql:5.6
and here is how I start the container with volume attached:
docker run \
--name=$(NAME)-mysql \
-d \
--env='DB_NAME=$(DB_NAME)' \
--cidfile="mysqlCID" \
--env='MYSQL_USER=$(DB_USER)' --env="MYSQL_ROOT_PASSWORD=$(DB_PASS)" \
--env="MYSQL_PASSWORD=$(DB_PASS)" \
--volume=$(MYSQL_DATADIR):/var/lib/mysql \
mysql:5.6
Ok, my issue here was a simple typo where I put a mysql directory inside another mysql directory, so the correct answer is to use the tar form with one slight modification above:
docker cp `cat mysqlinitCID`:/var/lib/mysql - |sudo tar -C datadir/ -pxf -