Using docker-compose in order to create a MySQL schema/database - mysql

I am trying to create a mysql database/schema if it doesn't already exist.
Here is what I have tried:
docker-compose.yml
mysql:
image: mysql:5.6.26
environment:
- MYSQL_ROOT_PASSWORD=root
command: "mysql -uroot -proot < createDB.sql"
ports:
- "3306:3306"
createDB.sql
CREATE DATABASE IF NOT EXISTS bignibou;
It does not work. What would be the best way to use docker/docker-compose in order to create a schema if it does not exist?

I finally found the beginning of a solution.
The MySQL image takes an environment variable i.e. MYSQL_DATABASE that initialize the container with the name of the database on image startup See here for full documentation.
Or read the excerpt below:
MYSQL_DATABASE
This variable is optional and allows you to specify the name of a
database to be created on image startup. If a user/password was
supplied (see below) then that user will be granted superuser access
(corresponding to GRANT ALL) to this database.
Here is what I came up with:
mysql:
image: mysql:5.6.26
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=bignibou
ports:
- "3306:3306"
I now need a way to specify the default collation but that is another story...
edit: For those interested in specifying a different collation from the default, here are the instructions to use another config file that will override the default one. See below:
Using a custom MySQL configuration file The MySQL startup
configuration is specified in the file /etc/mysql/my.cnf, and that
file in turn includes any files found in the /etc/mysql/conf.d
directory that end with .cnf. Settings in files in this directory will
augment and/or override settings in /etc/mysql/my.cnf. If you want to
use a customized MySQL configuration, you can create your alternative
configuration file in a directory on the host machine and then mount
that directory location as /etc/mysql/conf.d inside the mysql
container.
If /my/custom/config-file.cnf is the path and name of your custom
configuration file, you can start your mysql container like this (note
that only the directory path of the custom config file is used in this
command):
$ docker run --name some-mysql -v /my/custom:/etc/mysql/conf.d -e
MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag This will start a new
container some-mysql where the MySQL instance uses the combined
startup settings from /etc/mysql/my.cnf and
/etc/mysql/conf.d/config-file.cnf, with settings from the latter
taking precedence.

To not lost your data better use volumes as well:
version: '3'
services:
db:
image: mysql:5.7
volumes:
- mysql-db:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: my_db_name
ports:
- "3307:3306"
volumes:
mysql-db:

probably what you are trying to do needs an additional script. So if building an image instead of directly using a prebuilt image is an option for you, you need to use a Dockerfile and use a script file which first imports the script in MySql and then runs the service itself.
take a look at this answer: Docker - Initialize mysql database with schema

From the docker-compose documentation - see Define Services - you can tell which Dockerfile it will use to build the image. Therefore you can create a Dockerfile based on the mysql image and create the database inside it using standard Dockerfile commands.

This might be useful in case someone lands here in future. The real issue appears to be the "command" statement in the docker-compose file. Once the command finishes successfully the container will get destroyed. This sql script must be run only after docker-compose has run and containers have been created. docker-compose "command" is really to start a service in the container. In this case you overrode the mysql service with your command.

Related

Passing extra commands to a docker image from bitbucket-piplines.yml

Trying to set the default charset and collation of a mysql:5.7 docker image using Bitbucket Pipelines, the documentation is a little vague mentioning:
If you need to configure the underlying database engine further, refer to the official Docker Hub image for details.
This page that the bitbucket documentation sends you to suggests that it is possible... at least via docker:
docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
So my question is how do I pass these parameters in: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
I have seen people use command: parameter in the YML for bitbucket-pipelines however the pipeline config editor on bitbucket says it's not valid there:
definitions:
services:
mysql:
image: mysql:5.7
command: ['--character-set-server=utf8mb4', '--collation-server=utf8mb4_unicode_ci']
ports:
- "3306:3306"
variables:
MYSQL_DATABASE: $MY_DATABASE
MYSQL_ROOT_PASSWORD: $MY_PW
It seems that it is not possible to pass commands to containers that run as services at this point. I was able to find the schema of the YAML file that defines the pipelines (check line 365). Not only you can't set the command, but you also can't set the ports. Fortunately, 3306 is the default one.
As as workaround I'd suggest you build your own Docker image, based on the mysql:5.7 and change the CMD statement to mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci (you can see how the mysql image's CMD look's like from here). After that, you have to push the image to a registry to which your Bitbucket runner has access to and use this image for your pipeline.
The following Dockerfile might do the job for you:
FROM mysql:5.7
CMD ["mysqld", "--character-set-server=utf8mb4", "--collation-server=utf8mb4_unicode_ci"]
At the end, your definition will look like this:
definitions:
services:
mysql:
image: your-custom-mysql-image:5.7
variables:
MYSQL_DATABASE: $MY_DATABASE
MYSQL_ROOT_PASSWORD: $MY_PW

Dockerfile - Run a init sql file for setting up tables

I am able to run a SQL file as follows:
Setup Dockerfile as:
FROM mysql
ENV MYSQL_DATABASE stock_app
COPY ./sql-scripts/ /docker-entrypoint-initdb.d/
Building the image:
docker build -t db.
Run the following command:
docker run -d -p 3306:3306 --name db -e MYSQL_ROOT_PASSWORD=password db
This is working and able to verify that SQL commands inside the script are executed and the table is created.
What changes should I make so that all three steps are done when I do the following command to bring up all my images?
docker-compose up -d
Instead of manually building the image for db and running the command to execute the SQL file, is there an option to place the run SQL file within Dockerfile? That is to include step 3 mentioned above within Dockerfile.
You probably don't even need the Dockerfile unless you are doing something else that isn't listed above. You should be able to do what you want with a simple docker-compose like this:
version: '3.3'
services:
database:
image: mysql
volumes:
- "./sql-scripts:/docker-entrypoint-initdb.d"
environment:
MYSQL_DATABASE: stock_app
MYSQL_ROOT_PASSWORD: password
ports:
- '3306:3306'
Whenever you update your sql script you would need to recreate the containers with docker-compose up -d --force-recreate. By design the mysql image will run any sql files found in the "/docker-entrypoint-initdb.d" folder so you should not have to manually run those every time you recreate the containers.
The mysql image has the a volume mounted that stores the database. If the folder does not exists, it is created and your scripts are applied.
Instead of trying to write a script that destroys the way the base image is built, just change your base image and make it run scripts every times as it is suggested in this question : docker-compose: reinitializing MySQL db every time

User not created in MySQL when using docker-compose

This is what I see when I am in the container created by docker-compose:
mysql> SELECT user FROM mysql.user;
+------+
| user |
+------+
| root |
+------+
1 row in set (0.00 sec)
root#541e4d686184:/# echo $MYSQL_USER
dbuser
So dbuser is not present in the users table even though the $MYSQL_USER is set properly .
In docker-compose.yml I have this:
version: '2'
services:
db:
image: mysql:latest
environment:
MYSQL_DATABASE: mydb
MYSQL_USER: dbuser
MYSQL_PASSWORD: userpass
MYSQL_ROOT_PASSWORD: password
ports:
- "3306"
volumes:
- ./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
- my-datavolume:/var/lib/mysql
volumes:
my-datavolume:
I expected dbuser to be created automatically, but that didn't happen.
I also have a sql file to create my database and tables if they don't already exist, but right now tomcat can't connect to my database.
Same symptoms as this question, but I am already using a dictionary for my usernames/passwords.
UPDATE:
I am getting close. When inside container I manually did:
/docker-entrypoint-initdb.d/create_users.sh
Then the user was created inside MySQL table and I was able to deploy my application to my tomcat server and I didn't get an error about dbuser being denied access.
So, why did I have to run this command myself, it should be run by docker-compose, according to the mysql docker docs under Initializing a fresh instance.
How about:
docker-compose down -v
From the documentation:
-v - Remove volumes declared in the volumes section of the Compose file.
Your database has been already created inside a volume, so any changes of initial settings in docker-compose.yml won't be reflected.
In case you want to remove just a single volume, you may use docker volume ls to list all existing volumes and then docker volume rm <VOLUME NAME> to remove it.
Note: Bind mounts are not removed with the -v flag, so in case you are using them instead of volumes, you'll have to manually delete folders containing MySQL data. In docker-compose bind mounts are created whenever you provide a source path in your volumes section (eg. /my-path:/var/lib/mysql).
Worked for me : stop docker and remove manually all the folder containing MySQL data from previous builds.
Also : don't forget to add a MYSQL_DATABASE environment var or it won't create the user you specified.
Github issue
Important to note that the image entrypoint script will never make
changes to an existing database. If you mount an existing data
directory into var/lib/mysql, options like MYSQL_ROOT_PASSWORD will
have no effect
I met the same issue, you may try to remove everything under 'my-datavolume' because the environment works only in the initial stage that means there should not any data in '/var/lib/mysql'. This approach worked for me.
What worked for me is:
docker-compose down
docker volume ls
docker volume rm <volume-name>
docker-compose up -d
In the newly created volume, my user was there.
after my testing,
create init.sql and links to /docker-entrypoint-initdb.d
docker-compose down
docker volume ls
docker volume rm
docker-compose up -d
then everythi is ok

Enable logging in docker mysql container

I'm trying to get familiar with the docker ecosystem and tried to setup a mysql database container. With docker-compose this looks like:
version: '2'
services:
db:
image: mysql:5.6.33#sha256:31ad2efd094a1336ef1f8efaf40b88a5019778e7d9b8a8579a4f95a6be88eaba
volumes:
- "./db/data:/var/lib/mysql"
- "./db/log:/var/log/mysql"
- "./db/conf:/etc/mysql/conf.d"
restart: "yes"
environment:
MYSQL_ROOT_PASSWORD: rootpw
MYSQL_DATABASE: db
MYSQL_USER: db
MYSQL_PASSWORD: dbpw
My conf directory contains one file:
[mysqld]
log_error =/var/log/mysql/mysql_error.log
general_log_file=/var/log/mysql/mysql.log
general_log =1
slow_query_log =1
slow_query_log_file=/var/log/mysql/mysql_slow.log
long_query_time =2
log_queries_not_using_indexes = 1
Unfortunately I don't get any log files that way. The setup itself is correct and the cnf file is used. After connecting to the container and creating the 3 files, chown them to mysql and restarting the container, the logging is working as expected.
I'm pretty sure that this is a common scenario, and my current way to get it running seems really stupid. What is the correct way to do it?
I could improve my approach by moving all this stuff in a Dockerfile, but this still seem strange to me.
I was looking for the exact same thing, and now, there is a better way to do it.
The docker mysql writes:
Many configuration options can be passed as flags to mysqld. This will
give you the flexibility to customize the container without needing a
cnf file. For example, if you want to change the default encoding and
collation for all tables to use UTF-8 (utf8mb4) just run the
following:
$ docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
In a docker-compose world, one could pass these arguments through the "command" section of the service:
command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
In my use case I just wanted to turn on the logs and specify the path to the log file :
command: mysqld --general-log=1 --general-log-file=/var/lib/mysql/general-log.log
With the adequate volumes (e.g. - ./logs/mysql.log:/var/lib/mysql/general-log.log), it becomes easy to reach them.
This is pretty straight forward and avoid dealing with a local configuration. It will works with any MySQL Docker images and will keep the my.cnf as shipped by the image.
Edit: change path from /var/log/mysql/ to /var/lib/mysql/ to ensure a MySQL writable folder.
After connecting to the container and creating the 3 files, chown them to mysql and restarting the container, the logging is working as expected.
That points to a host volume permission issue. When you map from a container to the host, no mappings are made on user id's, and the name attached to the uid inside the container may be very different from outside. You need to initialize the directory permissions with something the container user can write to. One simple method is to create a group that has access to write to the files on both the host and container, and then add the various users to this group on both your image and host OS. Another option is to use a named filesystem that you don't access directly from your host and initialize it with the image's directory permissions.
Edit: An example of a named volume with your docker-compose.yml is as simple as:
version: '2'
volumes:
mysql-data:
driver: local
mysql-log:
driver: local
mysql-conf:
driver: local
services:
db:
image: mysql:5.6.33
volumes:
- "mysql-data:/var/lib/mysql"
- "mysql-log:/var/log/mysql"
- "mysql-conf:/etc/mysql/conf.d"
restart: unless-stopped
environment:
MYSQL_ROOT_PASSWORD: rootpw
MYSQL_DATABASE: db
MYSQL_USER: db
MYSQL_PASSWORD: dbpw
Note that I also removed the sha256 from your image name, this reference would block you from being able to pull patched versions of the image. I also prefer the "unless-stopped" restart policy so that Docker does expected things on a reboot.
I needed to temporarily enable logging due to a weird PDO binding issue and I wanted to see the actual query being executed. This question was the top search result and I wasn't satisfied with any of the answers. Assuming you already have volumes setup for the container, I got it working the following way:
Run the following queries on the database:
SET global general_log = on;
SET global general_log_file='/var/log/mysql/mysql.log';
SET global log_output = 'file';
Get the container ID using docker ps.
Run docker exec -it <id> /usr/bin/tail -f /var/log/mysql/mysql.log
Run the following query once you're done: SET global general_log = off;
If you're having problems setting the general_log_file variable, you probably need to /bin/bash into the container then manually create the log file with the correct permissions.
In addition to ponsfrilus answer
you can do the same from docker run command:
docker run --name some-mysql --restart always -v //d/MySqlDockerData:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:latest --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --log_error=/var/lib/mysql/mysql_error.log --general_log_file=/var/lib/mysql/mysql.log --general_log=1 --slow_query_log=1 --slow_query_log_file=/var/lib/mysql/mysql_slow.log --long_query_time=2 --log_queries_not_using_indexes=1

Docker - Can't access MySQL CLI in container

In my MySQL container that I have configured per the official image, it does not allow me to access the CLI from my host machine, nor any other of my containers, despite being linked:
Now, I know that I have not configured external access on my MySQL container, but how can I configure it if I can't even access the CLI to grant myself permissions? Could I create a Dockerfile and change some system settings before the initial installation of MySQL? If so, how would I go about doing that?
My fig.yml file, which is used to set up my MySQL container:
mysql:
image: mysql:latest
volumes:
- .:/mydir
working_dir: /mydir
ports:
- "3306:3306"
environment:
- MYSQL_DATABASE=mydb
- MYSQL_ROOT_PASSWORD=mypass
The answer is in your screenshot. (using password: NO). Add a -p to your mysql command line.
You can try to set the environment variable MYSQL_PWD with the password for root. It's the official environment variable for the client.
http://dev.mysql.com/doc/refman/5.6/en/password-security-user.html