This is what I see when I am in the container created by docker-compose:
mysql> SELECT user FROM mysql.user;
+------+
| user |
+------+
| root |
+------+
1 row in set (0.00 sec)
root#541e4d686184:/# echo $MYSQL_USER
dbuser
So dbuser is not present in the users table even though the $MYSQL_USER is set properly .
In docker-compose.yml I have this:
version: '2'
services:
db:
image: mysql:latest
environment:
MYSQL_DATABASE: mydb
MYSQL_USER: dbuser
MYSQL_PASSWORD: userpass
MYSQL_ROOT_PASSWORD: password
ports:
- "3306"
volumes:
- ./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
- my-datavolume:/var/lib/mysql
volumes:
my-datavolume:
I expected dbuser to be created automatically, but that didn't happen.
I also have a sql file to create my database and tables if they don't already exist, but right now tomcat can't connect to my database.
Same symptoms as this question, but I am already using a dictionary for my usernames/passwords.
UPDATE:
I am getting close. When inside container I manually did:
/docker-entrypoint-initdb.d/create_users.sh
Then the user was created inside MySQL table and I was able to deploy my application to my tomcat server and I didn't get an error about dbuser being denied access.
So, why did I have to run this command myself, it should be run by docker-compose, according to the mysql docker docs under Initializing a fresh instance.
How about:
docker-compose down -v
From the documentation:
-v - Remove volumes declared in the volumes section of the Compose file.
Your database has been already created inside a volume, so any changes of initial settings in docker-compose.yml won't be reflected.
In case you want to remove just a single volume, you may use docker volume ls to list all existing volumes and then docker volume rm <VOLUME NAME> to remove it.
Note: Bind mounts are not removed with the -v flag, so in case you are using them instead of volumes, you'll have to manually delete folders containing MySQL data. In docker-compose bind mounts are created whenever you provide a source path in your volumes section (eg. /my-path:/var/lib/mysql).
Worked for me : stop docker and remove manually all the folder containing MySQL data from previous builds.
Also : don't forget to add a MYSQL_DATABASE environment var or it won't create the user you specified.
Github issue
Important to note that the image entrypoint script will never make
changes to an existing database. If you mount an existing data
directory into var/lib/mysql, options like MYSQL_ROOT_PASSWORD will
have no effect
I met the same issue, you may try to remove everything under 'my-datavolume' because the environment works only in the initial stage that means there should not any data in '/var/lib/mysql'. This approach worked for me.
What worked for me is:
docker-compose down
docker volume ls
docker volume rm <volume-name>
docker-compose up -d
In the newly created volume, my user was there.
after my testing,
create init.sql and links to /docker-entrypoint-initdb.d
docker-compose down
docker volume ls
docker volume rm
docker-compose up -d
then everythi is ok
Related
I have a mysql docker container that has its data and logs dirs separately mapped to host folders for performance reasons.
I'm using docker-compose to start the container with a group of other related services.
--datadir=/var/lib/mysql/innodb-data
--innodb_log_group_home_dir=/var/lib/mysql/innodb-logs
The container dirs are mapped to the host files system via:
volumes:
- /db/mysql-innodb-data:/var/lib/mysql/innodb-data
- /db/mysql-innodb-logs:/var/lib/mysql/innodb-logs
My problem is that the MySQL container is setting the owner uid to 999.
On the host system this maps to the user 'systemd-coredump'.
Instead I want the container to apply the uid for the hosts 'mysql' user.
I've looked at the MySQL docker container and it has the following logic:
docker_create_db_directories() {
local user; user="$(id -u)"
# TODO other directories that are used by default? like /var/lib/mysql-files
# see https://github.com/docker-library/mysql/issues/562
mkdir -p "$DATADIR"
if [ "$user" = "0" ]; then
# this will cause less disk access than `chown -R`
find "$DATADIR" \! -user mysql -exec chown mysql '{}' +
fi
}
We can see that the above script applies the uid user the container runs under to the data directory. By default the container runs as root.
Given that root is uid 0 I don't actually see how this code is change the data-dirs directory to 999 and as such I suspect this code isn't actually the problem.
So I tried changing the user the container runs as to 'mysql'
mysql:
container_name: mysql
image: mysql:8.0
user: mysql
This changes the container user as expected but then MySQL couldn't start up as there are a number of config files that it can no longer read as it's not running as root.
Here is the full service section from my docker-compose:
mysql:
container_name: mysql
image: mysql:8.0
restart: on-failure
environment:
MYSQL_ROOT_PASSWORD: ${MYSQL_ADMIN_PASSWORD}
MYSQL_DATABASE: ${MYSQL_SCHEMA}
command: >
--user=mysql
--lower-case-table-names=1
--datadir=/var/lib/mysql/innodb-data
--innodb_log_group_home_dir=/var/lib/mysql/innodb-logs
--default-authentication-plugin=mysql_native_password
--max-allowed-packet=512M
--innodb_buffer_pool_instances=${MYSQL_INNODB_BUFFER_POOL_INSTANCES-32}
--innodb_buffer_pool_chunk_size=${MYSQL_INNODB_BUFFER_POOL_CHUNK_SIZE-8M}
--innodb_buffer_pool_size=${MYSQL_INNODB_BUFFER_POOL_SIZE-512M}
--table_open_cache=${MYSQL_TABLE_OPEN_CACHE-512}
--max_connections=${MYSQL_MAX_CONNECTIONS-98}
--innodb_flush_neighbors=0
--innodb_fast_shutdown=2
--innodb_flush_log_at_trx_commit=1
--innodb_flush_method=fsync
--innodb_doublewrite=0
--innodb_use_native_aio=0
--innodb_read_io_threads=10
--innodb_write_io_threads=10
--slow_query_log_file=/tmp/mysql-slow.log --long-query-time=1
--slow_query_log
# mem_limit: ${MYSQL_MEMORY}
volumes:
- /db/mysql-innodb-data:/var/lib/mysql/innodb-data
- /db/mysql-innodb-logs:/var/lib/mysql/innodb-logs
network_mode: "host"
logging:
driver: "journald"
Hi I am building a service in which I need a Mysql/MariaDB database. I have been googling different solutions and I got the db started with a database created thanks to a guide a was following (never found the link again unfortunately).
Problem
The problem I am having is that the tables are not being created. I added the sql-scema file to /docker-entrypoint-initdb.d/ (you can check it down in the docker file) but it doesnt seem to be executing it (I have tried with both copy and ADD commands).
Current output
This is my current console output from the container:
[![image][1]][1]
The database is created but the SOW TABLES; command returns Empty Set.
Desired output
Since this db is going to be a service differents scripts connect to (currently python), I need to be able to create the db and the sql schema (tables, triggers, etc...) so my team can work with same configuration.
Some of the solutions I have tried (I cant find all the links i have visited only a few)
How to import a mysql dump file into a Docker mysql container
mysql:5.7 docker allow access from all hosts and create DB
Can't connect to mariadb outside of docker container
Mariadb tables are deleted when use volume in docker-compose
Project structure
The structure is pretty simple I am using the following docker-compose.yml
Docker-compose
I still have to try if the MARIADB_ enviroment variables are necessary here.
version: '3'
services:
db-mysql:
#image: mysql/mysql-server:latest
build: ./mysql-db
restart: always
container_name : db-music
ports:
- '3306:3306'
environment:
MYSQL_ROOT_PASSWORD: pwd
MYSQL_DATABASE : audio_service
MYSQL_USER : user
MYSQL_PASSWORD : password
environment:
MARIADB_ROOT_PASSWORD: pwd
MARIADB_DATABASE : audio_service
MARIADB_USER : user
MARIADB_PASSWORD : password
#https://stackoverflow.com/questions/29145370/how-can-i-initialize-a-mysql-database-with-schema-in-a-docker-container?rq=1
expose:
- '3306:3306'
volumes:
- type: bind
source : E:\python-code\Rockstar\volume\mysql
target : /var/lib/mysql
#- type: bind
#source : E:\python-code\Rockstar\mysql-db\sql_scripts\tables.sql
#target : /docker-entrypoint-initdb.d/init.sql
networks:
net:
ipam:
driver: default
config:
- subnet: 212.172.1.0/30
host:
name: host
external: true
Dockerfile
FROM mariadb:latest as builder
# That file does the DB initialization but also runs mysql daemon, by removing the last line it will only init
RUN ["sed", "-i", "s/exec \"$#\"/echo \"not running $#\"/", "/usr/local/bin/docker-entrypoint.sh"]
# needed for intialization
ENV MYSQL_ROOT_PASSWORD=root
ENV MYSQL_ROOT_PASSWORD = pwd
ENV MYSQL_DATABASE = audio_service
ENV MYSQL_USER = user
ENV MYSQL_PASSWORD = password
COPY sql_scripts/tables.sql /docker-entrypoint-initdb.d/
# Need to change the datadir to something else that /var/lib/mysql because the parent docker file defines it as a volume.
# https://docs.docker.com/engine/reference/builder/#volume :
# Changing the volume from within the Dockerfile: If any build steps change the data within the volume after
# it has been declared, those changes will be discarded.
RUN ["/usr/local/bin/docker-entrypoint.sh", "mysqld", "--datadir", "/initialized-db", "--aria-log-dir-path", "/initialized-db"]
FROM mariadb:latest
# needed for intialization
ENV MARIADB_ROOT_PASSWORD=root
ENV MARIADB_ROOT_PASSWORD = pwd
ENV MARIADB_DATABASE = audio_service
ENV MARIADB_USER = user
ENV MARIADB_PASSWORD = password
COPY --from=builder /initialized-db /var/lib/mysql
EXPOSE 3306
SQL schema file
create database audio_service;
use audio_service;
CREATE TABLE audio
(
audio_id BINARY(16),
title TEXT NOT NULL UNIQUE,
content MEDIUMBLOB NOT NULL,
PRIMARY KEY (audio_id)
) COMMENT='this table stores sons';
DELIMITER ;;
CREATE TRIGGER `audio_before_insert`
BEFORE INSERT ON `audio` FOR EACH ROW
BEGIN
IF new.audio_id IS NULL THEN
SET new.audio_id = UUID_TO_BIN(UUID(), TRUE);
END IF;
END;;
DELIMITER ;
There is no need to build your own image since the official mysql / mariadb images are already well suited. You only need to run them with the following as explained in their image documentations:
environment variables to initialize an new database with a respective user on the first run
a volume at /var/lib/mysql to persist the data
any initialization/sql scripts mounted into /docker-entrypoint-initdb.d
So storing your SQL* into a schema.sql file right next to the docker-compose.yml the following is enough to achieve what you want:
# docker-compose.yml
services:
db:
image: mariadb
environment:
MARIADB_ROOT_PASSWORD: pwd
MARIADB_DATABASE: audio_service
MARIADB_USER: user
MARIADB_PASSWORD: password
volumes:
# persist data files into `datadir` volume managed by docker
- datadir:/var/lib/mysql
# bind-mount any sql files that should be run while initializing
- ./schema.sql:/docker-entrypoint-initdb.d/schema.sql
volumes:
datadir:
*note that you can remove the CREATE DATABASE and USE statements from your schema.sql since these will be automatically done by the init script for you anyway
There are two reasons that your own setup isn't working as expected:
the line COPY --from=builder /initialized-db /var/lib/mysql won't work as expected for the same reason you described in your comment a bit above it: /var/lib/mysql is a volume and thus no new files a stored in it in the build steps after it was defined.
you are bind-mounting E:\python-code\Rockstar\volume\mysql to /var/lib/mysql in your docker-compose.yml.
But this will effectively override any contents of /var/lib/mysql of the image, i.e. although your own image built from your Dockerfile does include an initialized database this is overwritten by the contents of E:\python-code\Rockstar\volume\mysql when starting the service.
I am able to run a SQL file as follows:
Setup Dockerfile as:
FROM mysql
ENV MYSQL_DATABASE stock_app
COPY ./sql-scripts/ /docker-entrypoint-initdb.d/
Building the image:
docker build -t db.
Run the following command:
docker run -d -p 3306:3306 --name db -e MYSQL_ROOT_PASSWORD=password db
This is working and able to verify that SQL commands inside the script are executed and the table is created.
What changes should I make so that all three steps are done when I do the following command to bring up all my images?
docker-compose up -d
Instead of manually building the image for db and running the command to execute the SQL file, is there an option to place the run SQL file within Dockerfile? That is to include step 3 mentioned above within Dockerfile.
You probably don't even need the Dockerfile unless you are doing something else that isn't listed above. You should be able to do what you want with a simple docker-compose like this:
version: '3.3'
services:
database:
image: mysql
volumes:
- "./sql-scripts:/docker-entrypoint-initdb.d"
environment:
MYSQL_DATABASE: stock_app
MYSQL_ROOT_PASSWORD: password
ports:
- '3306:3306'
Whenever you update your sql script you would need to recreate the containers with docker-compose up -d --force-recreate. By design the mysql image will run any sql files found in the "/docker-entrypoint-initdb.d" folder so you should not have to manually run those every time you recreate the containers.
The mysql image has the a volume mounted that stores the database. If the folder does not exists, it is created and your scripts are applied.
Instead of trying to write a script that destroys the way the base image is built, just change your base image and make it run scripts every times as it is suggested in this question : docker-compose: reinitializing MySQL db every time
I'm trying to get familiar with the docker ecosystem and tried to setup a mysql database container. With docker-compose this looks like:
version: '2'
services:
db:
image: mysql:5.6.33#sha256:31ad2efd094a1336ef1f8efaf40b88a5019778e7d9b8a8579a4f95a6be88eaba
volumes:
- "./db/data:/var/lib/mysql"
- "./db/log:/var/log/mysql"
- "./db/conf:/etc/mysql/conf.d"
restart: "yes"
environment:
MYSQL_ROOT_PASSWORD: rootpw
MYSQL_DATABASE: db
MYSQL_USER: db
MYSQL_PASSWORD: dbpw
My conf directory contains one file:
[mysqld]
log_error =/var/log/mysql/mysql_error.log
general_log_file=/var/log/mysql/mysql.log
general_log =1
slow_query_log =1
slow_query_log_file=/var/log/mysql/mysql_slow.log
long_query_time =2
log_queries_not_using_indexes = 1
Unfortunately I don't get any log files that way. The setup itself is correct and the cnf file is used. After connecting to the container and creating the 3 files, chown them to mysql and restarting the container, the logging is working as expected.
I'm pretty sure that this is a common scenario, and my current way to get it running seems really stupid. What is the correct way to do it?
I could improve my approach by moving all this stuff in a Dockerfile, but this still seem strange to me.
I was looking for the exact same thing, and now, there is a better way to do it.
The docker mysql writes:
Many configuration options can be passed as flags to mysqld. This will
give you the flexibility to customize the container without needing a
cnf file. For example, if you want to change the default encoding and
collation for all tables to use UTF-8 (utf8mb4) just run the
following:
$ docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
In a docker-compose world, one could pass these arguments through the "command" section of the service:
command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
In my use case I just wanted to turn on the logs and specify the path to the log file :
command: mysqld --general-log=1 --general-log-file=/var/lib/mysql/general-log.log
With the adequate volumes (e.g. - ./logs/mysql.log:/var/lib/mysql/general-log.log), it becomes easy to reach them.
This is pretty straight forward and avoid dealing with a local configuration. It will works with any MySQL Docker images and will keep the my.cnf as shipped by the image.
Edit: change path from /var/log/mysql/ to /var/lib/mysql/ to ensure a MySQL writable folder.
After connecting to the container and creating the 3 files, chown them to mysql and restarting the container, the logging is working as expected.
That points to a host volume permission issue. When you map from a container to the host, no mappings are made on user id's, and the name attached to the uid inside the container may be very different from outside. You need to initialize the directory permissions with something the container user can write to. One simple method is to create a group that has access to write to the files on both the host and container, and then add the various users to this group on both your image and host OS. Another option is to use a named filesystem that you don't access directly from your host and initialize it with the image's directory permissions.
Edit: An example of a named volume with your docker-compose.yml is as simple as:
version: '2'
volumes:
mysql-data:
driver: local
mysql-log:
driver: local
mysql-conf:
driver: local
services:
db:
image: mysql:5.6.33
volumes:
- "mysql-data:/var/lib/mysql"
- "mysql-log:/var/log/mysql"
- "mysql-conf:/etc/mysql/conf.d"
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: rootpw
MYSQL_DATABASE: db
MYSQL_USER: db
MYSQL_PASSWORD: dbpw
Note that I also removed the sha256 from your image name, this reference would block you from being able to pull patched versions of the image. I also prefer the "unless-stopped" restart policy so that Docker does expected things on a reboot.
I needed to temporarily enable logging due to a weird PDO binding issue and I wanted to see the actual query being executed. This question was the top search result and I wasn't satisfied with any of the answers. Assuming you already have volumes setup for the container, I got it working the following way:
Run the following queries on the database:
SET global general_log = on;
SET global general_log_file='/var/log/mysql/mysql.log';
SET global log_output = 'file';
Get the container ID using docker ps.
Run docker exec -it <id> /usr/bin/tail -f /var/log/mysql/mysql.log
Run the following query once you're done: SET global general_log = off;
If you're having problems setting the general_log_file variable, you probably need to /bin/bash into the container then manually create the log file with the correct permissions.
In addition to ponsfrilus answer
you can do the same from docker run command:
docker run --name some-mysql --restart always -v //d/MySqlDockerData:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:latest --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --log_error=/var/lib/mysql/mysql_error.log --general_log_file=/var/lib/mysql/mysql.log --general_log=1 --slow_query_log=1 --slow_query_log_file=/var/lib/mysql/mysql_slow.log --long_query_time=2 --log_queries_not_using_indexes=1
I am trying to create a mysql database/schema if it doesn't already exist.
Here is what I have tried:
docker-compose.yml
mysql:
image: mysql:5.6.26
environment:
- MYSQL_ROOT_PASSWORD=root
command: "mysql -uroot -proot < createDB.sql"
ports:
- "3306:3306"
createDB.sql
CREATE DATABASE IF NOT EXISTS bignibou;
It does not work. What would be the best way to use docker/docker-compose in order to create a schema if it does not exist?
I finally found the beginning of a solution.
The MySQL image takes an environment variable i.e. MYSQL_DATABASE that initialize the container with the name of the database on image startup See here for full documentation.
Or read the excerpt below:
MYSQL_DATABASE
This variable is optional and allows you to specify the name of a
database to be created on image startup. If a user/password was
supplied (see below) then that user will be granted superuser access
(corresponding to GRANT ALL) to this database.
Here is what I came up with:
mysql:
image: mysql:5.6.26
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=bignibou
ports:
- "3306:3306"
I now need a way to specify the default collation but that is another story...
edit: For those interested in specifying a different collation from the default, here are the instructions to use another config file that will override the default one. See below:
Using a custom MySQL configuration file The MySQL startup
configuration is specified in the file /etc/mysql/my.cnf, and that
file in turn includes any files found in the /etc/mysql/conf.d
directory that end with .cnf. Settings in files in this directory will
augment and/or override settings in /etc/mysql/my.cnf. If you want to
use a customized MySQL configuration, you can create your alternative
configuration file in a directory on the host machine and then mount
that directory location as /etc/mysql/conf.d inside the mysql
container.
If /my/custom/config-file.cnf is the path and name of your custom
configuration file, you can start your mysql container like this (note
that only the directory path of the custom config file is used in this
command):
$ docker run --name some-mysql -v /my/custom:/etc/mysql/conf.d -e
MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag This will start a new
container some-mysql where the MySQL instance uses the combined
startup settings from /etc/mysql/my.cnf and
/etc/mysql/conf.d/config-file.cnf, with settings from the latter
taking precedence.
To not lost your data better use volumes as well:
version: '3'
services:
db:
image: mysql:5.7
volumes:
- mysql-db:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: my_db_name
ports:
- "3307:3306"
volumes:
mysql-db:
probably what you are trying to do needs an additional script. So if building an image instead of directly using a prebuilt image is an option for you, you need to use a Dockerfile and use a script file which first imports the script in MySql and then runs the service itself.
take a look at this answer: Docker - Initialize mysql database with schema
From the docker-compose documentation - see Define Services - you can tell which Dockerfile it will use to build the image. Therefore you can create a Dockerfile based on the mysql image and create the database inside it using standard Dockerfile commands.
This might be useful in case someone lands here in future. The real issue appears to be the "command" statement in the docker-compose file. Once the command finishes successfully the container will get destroyed. This sql script must be run only after docker-compose has run and containers have been created. docker-compose "command" is really to start a service in the container. In this case you overrode the mysql service with your command.