I want to mount a directory from host inside mysql container so that mysql would write its logs to that directory and I would be able to access those logs from host.
For this I am using the following volume configuration:
volumes:
- ./logs/mysql:/var/log/mysql
But as pointed out in this answer, there are permission issues between host user and container user. The solution there was to use named volumes, but what I want is to access those logs on host in a convenient directory. Not inside internal directories of docker.
So I ran the default image and made few observations
By default the log files are not created at all in /var/log/mysql. This is because the default my.cnf has the error-log settings commented
You need to create your own config file to add these settings and map them inside /etc/mysql/mysql.conf.d
The /entrypoint.sh does change the permissions on /var/lib/mysql but not on /var/log/mysql
So to fix the issue you add a test.cnf file with below content
[mysqld]
log-error = /var/log/mysql/error.log
general_log = /var/log/mysql/log_output.log
And update your docker-compose with below settings
version: '2'
services:
mysql:
image: mysql:latest
volumes:
- ./logs:/var/log/mysql
- ./test.cnf:/etc/mysql/mysql.conf.d/test.cnf
environment:
MYSQL_ROOT_PASSWORD: root
entrypoint: ""
command: bash -c "chown -R mysql:mysql /var/log/mysql && exec /entrypoint.sh mysqld"
This would make sure that before running the entrypoint script the proper permissions are set
Related
I have a mysql docker container that has its data and logs dirs separately mapped to host folders for performance reasons.
I'm using docker-compose to start the container with a group of other related services.
--datadir=/var/lib/mysql/innodb-data
--innodb_log_group_home_dir=/var/lib/mysql/innodb-logs
The container dirs are mapped to the host files system via:
volumes:
- /db/mysql-innodb-data:/var/lib/mysql/innodb-data
- /db/mysql-innodb-logs:/var/lib/mysql/innodb-logs
My problem is that the MySQL container is setting the owner uid to 999.
On the host system this maps to the user 'systemd-coredump'.
Instead I want the container to apply the uid for the hosts 'mysql' user.
I've looked at the MySQL docker container and it has the following logic:
docker_create_db_directories() {
local user; user="$(id -u)"
# TODO other directories that are used by default? like /var/lib/mysql-files
# see https://github.com/docker-library/mysql/issues/562
mkdir -p "$DATADIR"
if [ "$user" = "0" ]; then
# this will cause less disk access than `chown -R`
find "$DATADIR" \! -user mysql -exec chown mysql '{}' +
fi
}
We can see that the above script applies the uid user the container runs under to the data directory. By default the container runs as root.
Given that root is uid 0 I don't actually see how this code is change the data-dirs directory to 999 and as such I suspect this code isn't actually the problem.
So I tried changing the user the container runs as to 'mysql'
mysql:
container_name: mysql
image: mysql:8.0
user: mysql
This changes the container user as expected but then MySQL couldn't start up as there are a number of config files that it can no longer read as it's not running as root.
Here is the full service section from my docker-compose:
mysql:
container_name: mysql
image: mysql:8.0
restart: on-failure
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ADMIN_PASSWORD}
MYSQL_DATABASE: ${MYSQL_SCHEMA}
command: >
--user=mysql
--lower-case-table-names=1
--datadir=/var/lib/mysql/innodb-data
--innodb_log_group_home_dir=/var/lib/mysql/innodb-logs
--default-authentication-plugin=mysql_native_password
--max-allowed-packet=512M
--innodb_buffer_pool_instances=${MYSQL_INNODB_BUFFER_POOL_INSTANCES-32}
--innodb_buffer_pool_chunk_size=${MYSQL_INNODB_BUFFER_POOL_CHUNK_SIZE-8M}
--innodb_buffer_pool_size=${MYSQL_INNODB_BUFFER_POOL_SIZE-512M}
--table_open_cache=${MYSQL_TABLE_OPEN_CACHE-512}
--max_connections=${MYSQL_MAX_CONNECTIONS-98}
--innodb_flush_neighbors=0
--innodb_fast_shutdown=2
--innodb_flush_log_at_trx_commit=1
--innodb_flush_method=fsync
--innodb_doublewrite=0
--innodb_use_native_aio=0
--innodb_read_io_threads=10
--innodb_write_io_threads=10
--slow_query_log_file=/tmp/mysql-slow.log --long-query-time=1
--slow_query_log
# mem_limit: ${MYSQL_MEMORY}
volumes:
- /db/mysql-innodb-data:/var/lib/mysql/innodb-data
- /db/mysql-innodb-logs:/var/lib/mysql/innodb-logs
network_mode: "host"
logging:
driver: "journald"
I'm using docker-compose v 1.27 and Docker v 19.03. I have this in my docker-compose.yml file ...
version: '3'
services:
mysql:
restart: always
image: mysql:8.0
cap_add:
- SYS_NICE # CAP_SYS_NICE
environment:
MYSQL_DATABASE: 'directory_data'
# So you don't have to use root, but you can if you like
MYSQL_USER: 'root'
# You can use whatever password you like
MYSQL_PASSWORD: 'password'
# Password for root access
MYSQL_ROOT_PASSWORD: 'password'
MYSQL_ROOT_HOST: '%'
ports:
- "3406:3306"
volumes:
- my-db:/var/lib/mysql
- ./mysql/mysqlconf:/etc/mysql/conf.d
command: ['mysqld', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
Note that I have no Dockerfile (didn't think I needed it). My "my.cnf" file, is below
davea$ cat mysql/mysqlconf/my.cnf
bind-address = 0.0.0.0
From Docker, how do I set the permissions of the my.cnf file to be read-only? This comes into play on Windows 10 in which running "docker-compose up" results in this warning
mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/my.cnf' is ignored.
Note, this answer -- https://stackoverflow.com/questions/64327260/in-docker-compose-how-do-i-set-perms-on-a-my-cnf-file-if-i-dont-have-a-dockerf, doesn't cut it, because it relies on setting th
I think the underlying problem here is that you are mounting a NTFS directory volume inside of an ext filesystem. Below are some possible solutions that may be helpful.
Docker-level: Use Read-only Volume Mounts
You can use a read-only volume mount instead of the default read-write setting.
For example, add :ro (read-only) to the end of the volume specification:
volumes:
- ...
- ./mysql/mysqlconf:/etc/mysql/conf.d:ro
Container-level: chmod the configuration file
If you want to suppress the warning, you can try setting the permissions of the files at run-time to read-only by expanding the command configuration to several commands. I think this is what you are referring to as the client's OS level, though. The mount would not be read-only.
For example:
command: bash -c "
chmod -R 0444 /etc/mysql/conf.d/ &&
mysqld --user=root --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
"
Note that this is incompatible with read-only mounts, as you cannot adjust the permissions since it is a read-only filesystem.
In your docker-compose yaml file, you can define the read-only access to the mounted volume
by adding :ro at the end of the volume definition.
version: '3'
services:
mysql:
restart: always
image: mysql:8.0
cap_add:
- SYS_NICE # CAP_SYS_NICE
environment:
MYSQL_DATABASE: 'directory_data'
# So you don't have to use root, but you can if you like
MYSQL_USER: 'root'
# You can use whatever password you like
MYSQL_PASSWORD: 'password'
# Password for root access
MYSQL_ROOT_PASSWORD: 'password'
MYSQL_ROOT_HOST: '%'
ports:
- "3406:3306"
volumes:
- my-db:/var/lib/mysql
- ./mysql/mysqlconf:/etc/mysql/conf.d:ro
command: ['mysqld', '--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
I suggest you set the permissions in a custom entrypoint script. This ensures they are adjusted on every container start, plays nicely with the "official" mysql image (it has custom entry point script support baked in) and does not collide with docker best practices (keeps mysqld running as pid 1).
It's three steps.
Create a script that makes all necessary adjustments and make it executable:
cat <<EOF > ./adjust-permissions.sh
#!/bin/sh
set -ex
chown -R root:root /etc/mysql/conf.d/
chmod -R 0644 /etc/mysql/conf.d/
EOF
chmod +x ./adjust-permissions.sh
You might want to leave out chown, personally, I like to ensure there are no surprises with mounted files.
Mount it into /docker-entrypoint-initdb.d/ inside the container (see docker-entrypoint.sh):
volumes:
[...]
- ./adjust-permissions.sh:/docker-entrypoint-initdb.d/adjust-permissions.sh
Enjoy.
This is what I see when I am in the container created by docker-compose:
mysql> SELECT user FROM mysql.user;
+------+
| user |
+------+
| root |
+------+
1 row in set (0.00 sec)
root#541e4d686184:/# echo $MYSQL_USER
dbuser
So dbuser is not present in the users table even though the $MYSQL_USER is set properly .
In docker-compose.yml I have this:
version: '2'
services:
db:
image: mysql:latest
environment:
MYSQL_DATABASE: mydb
MYSQL_USER: dbuser
MYSQL_PASSWORD: userpass
MYSQL_ROOT_PASSWORD: password
ports:
- "3306"
volumes:
- ./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
- my-datavolume:/var/lib/mysql
volumes:
my-datavolume:
I expected dbuser to be created automatically, but that didn't happen.
I also have a sql file to create my database and tables if they don't already exist, but right now tomcat can't connect to my database.
Same symptoms as this question, but I am already using a dictionary for my usernames/passwords.
UPDATE:
I am getting close. When inside container I manually did:
/docker-entrypoint-initdb.d/create_users.sh
Then the user was created inside MySQL table and I was able to deploy my application to my tomcat server and I didn't get an error about dbuser being denied access.
So, why did I have to run this command myself, it should be run by docker-compose, according to the mysql docker docs under Initializing a fresh instance.
How about:
docker-compose down -v
From the documentation:
-v - Remove volumes declared in the volumes section of the Compose file.
Your database has been already created inside a volume, so any changes of initial settings in docker-compose.yml won't be reflected.
In case you want to remove just a single volume, you may use docker volume ls to list all existing volumes and then docker volume rm <VOLUME NAME> to remove it.
Note: Bind mounts are not removed with the -v flag, so in case you are using them instead of volumes, you'll have to manually delete folders containing MySQL data. In docker-compose bind mounts are created whenever you provide a source path in your volumes section (eg. /my-path:/var/lib/mysql).
Worked for me : stop docker and remove manually all the folder containing MySQL data from previous builds.
Also : don't forget to add a MYSQL_DATABASE environment var or it won't create the user you specified.
Github issue
Important to note that the image entrypoint script will never make
changes to an existing database. If you mount an existing data
directory into var/lib/mysql, options like MYSQL_ROOT_PASSWORD will
have no effect
I met the same issue, you may try to remove everything under 'my-datavolume' because the environment works only in the initial stage that means there should not any data in '/var/lib/mysql'. This approach worked for me.
What worked for me is:
docker-compose down
docker volume ls
docker volume rm <volume-name>
docker-compose up -d
In the newly created volume, my user was there.
after my testing,
create init.sql and links to /docker-entrypoint-initdb.d
docker-compose down
docker volume ls
docker volume rm
docker-compose up -d
then everythi is ok
I'm trying to get familiar with the docker ecosystem and tried to setup a mysql database container. With docker-compose this looks like:
version: '2'
services:
db:
image: mysql:5.6.33#sha256:31ad2efd094a1336ef1f8efaf40b88a5019778e7d9b8a8579a4f95a6be88eaba
volumes:
- "./db/data:/var/lib/mysql"
- "./db/log:/var/log/mysql"
- "./db/conf:/etc/mysql/conf.d"
restart: "yes"
environment:
MYSQL_ROOT_PASSWORD: rootpw
MYSQL_DATABASE: db
MYSQL_USER: db
MYSQL_PASSWORD: dbpw
My conf directory contains one file:
[mysqld]
log_error =/var/log/mysql/mysql_error.log
general_log_file=/var/log/mysql/mysql.log
general_log =1
slow_query_log =1
slow_query_log_file=/var/log/mysql/mysql_slow.log
long_query_time =2
log_queries_not_using_indexes = 1
Unfortunately I don't get any log files that way. The setup itself is correct and the cnf file is used. After connecting to the container and creating the 3 files, chown them to mysql and restarting the container, the logging is working as expected.
I'm pretty sure that this is a common scenario, and my current way to get it running seems really stupid. What is the correct way to do it?
I could improve my approach by moving all this stuff in a Dockerfile, but this still seem strange to me.
I was looking for the exact same thing, and now, there is a better way to do it.
The docker mysql writes:
Many configuration options can be passed as flags to mysqld. This will
give you the flexibility to customize the container without needing a
cnf file. For example, if you want to change the default encoding and
collation for all tables to use UTF-8 (utf8mb4) just run the
following:
$ docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
In a docker-compose world, one could pass these arguments through the "command" section of the service:
command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
In my use case I just wanted to turn on the logs and specify the path to the log file :
command: mysqld --general-log=1 --general-log-file=/var/lib/mysql/general-log.log
With the adequate volumes (e.g. - ./logs/mysql.log:/var/lib/mysql/general-log.log), it becomes easy to reach them.
This is pretty straight forward and avoid dealing with a local configuration. It will works with any MySQL Docker images and will keep the my.cnf as shipped by the image.
Edit: change path from /var/log/mysql/ to /var/lib/mysql/ to ensure a MySQL writable folder.
After connecting to the container and creating the 3 files, chown them to mysql and restarting the container, the logging is working as expected.
That points to a host volume permission issue. When you map from a container to the host, no mappings are made on user id's, and the name attached to the uid inside the container may be very different from outside. You need to initialize the directory permissions with something the container user can write to. One simple method is to create a group that has access to write to the files on both the host and container, and then add the various users to this group on both your image and host OS. Another option is to use a named filesystem that you don't access directly from your host and initialize it with the image's directory permissions.
Edit: An example of a named volume with your docker-compose.yml is as simple as:
version: '2'
volumes:
mysql-data:
driver: local
mysql-log:
driver: local
mysql-conf:
driver: local
services:
db:
image: mysql:5.6.33
volumes:
- "mysql-data:/var/lib/mysql"
- "mysql-log:/var/log/mysql"
- "mysql-conf:/etc/mysql/conf.d"
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: rootpw
MYSQL_DATABASE: db
MYSQL_USER: db
MYSQL_PASSWORD: dbpw
Note that I also removed the sha256 from your image name, this reference would block you from being able to pull patched versions of the image. I also prefer the "unless-stopped" restart policy so that Docker does expected things on a reboot.
I needed to temporarily enable logging due to a weird PDO binding issue and I wanted to see the actual query being executed. This question was the top search result and I wasn't satisfied with any of the answers. Assuming you already have volumes setup for the container, I got it working the following way:
Run the following queries on the database:
SET global general_log = on;
SET global general_log_file='/var/log/mysql/mysql.log';
SET global log_output = 'file';
Get the container ID using docker ps.
Run docker exec -it <id> /usr/bin/tail -f /var/log/mysql/mysql.log
Run the following query once you're done: SET global general_log = off;
If you're having problems setting the general_log_file variable, you probably need to /bin/bash into the container then manually create the log file with the correct permissions.
In addition to ponsfrilus answer
you can do the same from docker run command:
docker run --name some-mysql --restart always -v //d/MySqlDockerData:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:latest --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --log_error=/var/lib/mysql/mysql_error.log --general_log_file=/var/lib/mysql/mysql.log --general_log=1 --slow_query_log=1 --slow_query_log_file=/var/lib/mysql/mysql_slow.log --long_query_time=2 --log_queries_not_using_indexes=1
I'm having a baffling time with docker-compose and mysql. When I try to start my mysql container with docker-compose, I get an uninformative mkdir error. My Dockerfile and docker-compose.yml don't even seem to contain a mkdir command for mysql.
Here is configuration:
docker-compose.yml
db:
build: docker/mysql
volumes:
- ./database:/var/lib/mysql
environment:
- MYSQL_DATABASE=xxxx
- MYSQL_USER=xxxx
- MYSQL_PASSWORD=xxxx
- MYSQL_ROOT_PASSWORD=xxxx
privileged: true
hostname: "docker.mysql"
Dockerfile
FROM mysql:5.6
MAINTAINER "ABC" <developers#abc.com>
ADD my.cnf /etc/mysql/my.cnf
RUN usermod -u 1000 mysql
RUN usermod -G staff mysql
RUN chown mysql /var/run/mysqld
EXPOSE 3306
my.cnf pastebin
http://pastebin.com/iVSGxhGV
sudo docker --version
Docker version 1.10.0, build 590d5108
This is a docker setup built by my team mates, so I am not sure what the priviledged:true line is doing, but when I remove it, everything appears to work.
with privileged:true
sudo docker-compose up db
Creating appname_db_1
Attaching to appname_db_1
db_1 | mkdir: cannot create directory '': No such file or directory
appname_db_1 exited with code 1
Once privileged: true is removed, it seems to work.
Does anyone know why this might be?