I'm new in docker, so cant understand - if I want to build container of mysql/postgresql/clickhouse etc - how to create database and schema of database/table? Maybe in Dockerfile or i can do it from docker-compose.yml?
I mean, that I dont know when and where to use CREATE DATABASE; CREATE TABLE ...; queries if I use docker containers of popular databases
You can use both docker and docker-compose. For example with docker compose.
Create a file called docker-compose.yml like:
version: '3'
services:
db:
image: percona:5.7
container_name: whatever_you_want
environment:
- MYSQL_DATABASE=${DATABASE}
- MYSQL_ROOT_PASSWORD=${ROOT_PASSWORD}
- MYSQL_USER=${USER}
- MYSQL_PASSWORD=${PASSWORD}
volumes:
- ./data:/docker-entrypoint-initdb.d
ports:
- "3306:3306"
Additionally you need a file under ./data with whatever SQL commands you want to run and and .env file where you define the environmental variables I used in the docker-compose.yml file above like: ${DATABASE}
Your .env file:
# MySQL
DATABASE=db_name_here
ROOT_USER=root
ROOT_PASSWORD=root
USER=dev
PASSWORD=dev
Your file with SQL commands to execute ./data/init.sql (you can name the file whatever you want)
CREATE DATABASE 'whatever';
DROP DATABASE 'whatever';
-- you can do whatever you want here
This file will be executed each time you do:
docker-compose up -d db
At first you need to create docker a image for your db server, or use an already existing image.
Bellow is an example of mysql docker image.
version: "3"
services:
****************
mysql:
container_name: mysql
image: mysql:5.7
restart: on-failure
environment:
- MYSQL_DATABASE=YOUR_DB_NAME
- MYSQL_ROOT_PASSWORD=YOUR_ROOT_USER_PASSWORD
- MYSQL_USER=YOUR_USER
- MYSQL_PASSWORD=YOUR_USER_PASSWORD
ports:
- "33060:3306"
volumes:
- "./data/db/mysql:/var/lib/mysql"
Let's describe some sections:
volumes:
- "./data/db/mysql:/var/lib/mysql"
This is like "mounting" container's /var/lib/mysql to system's ./data/db/mysql. So your data will be on your system drive, because in debian the default path to MySQL data is /var/lib/mysql.
ports:
- "33060:3306"
This will map port 3306 from container to system's 33060 port, to avoid conflicts if you have installed MySQL server on system as well.
environment:
- MYSQL_DATABASE=YOUR_DB_NAME
- MYSQL_ROOT_PASSWORD=YOUR_ROOT_USER_PASSWORD
- MYSQL_USER=YOUR_USER
- MYSQL_PASSWORD=YOUR_USER_PASSWORD
This will create a database with the defined parameters: name, root password, ..., or if a database already exists it will try to access with the defined credentials. Functionality to check/create database is already defined in the image.
If you want to define your own functionality you can define your image (e.g. dockerfile: ./Dockerfile instead of image: mysql:5.7). Dockerfile can be something like this:
FROM mysql:5.7
ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_ROOT_PASSWORD
ENV MYSQL_DATABASE=${MYSQL_DATABASE}
ENV MYSQL_USER=${MYSQL_USER}
ENV MYSQL_PASSWORD=${MYSQL_PASSWORD}
ENV MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
# copy predefined config file
COPY configs/default.cnf /etc/mysql/conf.d/
# To be sure that MySQL will not ignore configs
RUN chmod og-w /etc/mysql/conf.d/default.cnf
# DO SOMETHING ELSE YOU WANT
EXPOSE 3306
CMD ["mysqld"]
So you can build and up your container with command docker-compose up -d --build
Here is an example I used to initialise SQL Server 2017 database using container.
https://www.handsonarchitect.com/2018/01/build-custom-sql-server-2017-linux.html
The trick is to use a shell script to run which will invoke the database initialisation script. You might have to wait for few seconds for the database engine service to start before executing the initialisation script.
Related
Hi I am building a service in which I need a Mysql/MariaDB database. I have been googling different solutions and I got the db started with a database created thanks to a guide a was following (never found the link again unfortunately).
Problem
The problem I am having is that the tables are not being created. I added the sql-scema file to /docker-entrypoint-initdb.d/ (you can check it down in the docker file) but it doesnt seem to be executing it (I have tried with both copy and ADD commands).
Current output
This is my current console output from the container:
[![image][1]][1]
The database is created but the SOW TABLES; command returns Empty Set.
Desired output
Since this db is going to be a service differents scripts connect to (currently python), I need to be able to create the db and the sql schema (tables, triggers, etc...) so my team can work with same configuration.
Some of the solutions I have tried (I cant find all the links i have visited only a few)
How to import a mysql dump file into a Docker mysql container
mysql:5.7 docker allow access from all hosts and create DB
Can't connect to mariadb outside of docker container
Mariadb tables are deleted when use volume in docker-compose
Project structure
The structure is pretty simple I am using the following docker-compose.yml
Docker-compose
I still have to try if the MARIADB_ enviroment variables are necessary here.
version: '3'
services:
db-mysql:
#image: mysql/mysql-server:latest
build: ./mysql-db
restart: always
container_name : db-music
ports:
- '3306:3306'
environment:
MYSQL_ROOT_PASSWORD: pwd
MYSQL_DATABASE : audio_service
MYSQL_USER : user
MYSQL_PASSWORD : password
environment:
MARIADB_ROOT_PASSWORD: pwd
MARIADB_DATABASE : audio_service
MARIADB_USER : user
MARIADB_PASSWORD : password
#https://stackoverflow.com/questions/29145370/how-can-i-initialize-a-mysql-database-with-schema-in-a-docker-container?rq=1
expose:
- '3306:3306'
volumes:
- type: bind
source : E:\python-code\Rockstar\volume\mysql
target : /var/lib/mysql
#- type: bind
#source : E:\python-code\Rockstar\mysql-db\sql_scripts\tables.sql
#target : /docker-entrypoint-initdb.d/init.sql
networks:
net:
ipam:
driver: default
config:
- subnet: 212.172.1.0/30
host:
name: host
external: true
Dockerfile
FROM mariadb:latest as builder
# That file does the DB initialization but also runs mysql daemon, by removing the last line it will only init
RUN ["sed", "-i", "s/exec \"$#\"/echo \"not running $#\"/", "/usr/local/bin/docker-entrypoint.sh"]
# needed for intialization
ENV MYSQL_ROOT_PASSWORD=root
ENV MYSQL_ROOT_PASSWORD = pwd
ENV MYSQL_DATABASE = audio_service
ENV MYSQL_USER = user
ENV MYSQL_PASSWORD = password
COPY sql_scripts/tables.sql /docker-entrypoint-initdb.d/
# Need to change the datadir to something else that /var/lib/mysql because the parent docker file defines it as a volume.
# https://docs.docker.com/engine/reference/builder/#volume :
# Changing the volume from within the Dockerfile: If any build steps change the data within the volume after
# it has been declared, those changes will be discarded.
RUN ["/usr/local/bin/docker-entrypoint.sh", "mysqld", "--datadir", "/initialized-db", "--aria-log-dir-path", "/initialized-db"]
FROM mariadb:latest
# needed for intialization
ENV MARIADB_ROOT_PASSWORD=root
ENV MARIADB_ROOT_PASSWORD = pwd
ENV MARIADB_DATABASE = audio_service
ENV MARIADB_USER = user
ENV MARIADB_PASSWORD = password
COPY --from=builder /initialized-db /var/lib/mysql
EXPOSE 3306
SQL schema file
create database audio_service;
use audio_service;
CREATE TABLE audio
(
audio_id BINARY(16),
title TEXT NOT NULL UNIQUE,
content MEDIUMBLOB NOT NULL,
PRIMARY KEY (audio_id)
) COMMENT='this table stores sons';
DELIMITER ;;
CREATE TRIGGER `audio_before_insert`
BEFORE INSERT ON `audio` FOR EACH ROW
BEGIN
IF new.audio_id IS NULL THEN
SET new.audio_id = UUID_TO_BIN(UUID(), TRUE);
END IF;
END;;
DELIMITER ;
There is no need to build your own image since the official mysql / mariadb images are already well suited. You only need to run them with the following as explained in their image documentations:
environment variables to initialize an new database with a respective user on the first run
a volume at /var/lib/mysql to persist the data
any initialization/sql scripts mounted into /docker-entrypoint-initdb.d
So storing your SQL* into a schema.sql file right next to the docker-compose.yml the following is enough to achieve what you want:
# docker-compose.yml
services:
db:
image: mariadb
environment:
MARIADB_ROOT_PASSWORD: pwd
MARIADB_DATABASE: audio_service
MARIADB_USER: user
MARIADB_PASSWORD: password
volumes:
# persist data files into `datadir` volume managed by docker
- datadir:/var/lib/mysql
# bind-mount any sql files that should be run while initializing
- ./schema.sql:/docker-entrypoint-initdb.d/schema.sql
volumes:
datadir:
*note that you can remove the CREATE DATABASE and USE statements from your schema.sql since these will be automatically done by the init script for you anyway
There are two reasons that your own setup isn't working as expected:
the line COPY --from=builder /initialized-db /var/lib/mysql won't work as expected for the same reason you described in your comment a bit above it: /var/lib/mysql is a volume and thus no new files a stored in it in the build steps after it was defined.
you are bind-mounting E:\python-code\Rockstar\volume\mysql to /var/lib/mysql in your docker-compose.yml.
But this will effectively override any contents of /var/lib/mysql of the image, i.e. although your own image built from your Dockerfile does include an initialized database this is overwritten by the contents of E:\python-code\Rockstar\volume\mysql when starting the service.
I have managed to create a MySQL and PHP container and my scripts execute and all my tables are there.
However I have a database that I call "myDb" and a user that is called "someuser" and when the database is created for some reason the name of the database is "somedatabase"
my docker-compose.yaml file:
services:
mysql:
image: mysql:latest
ports:
- 3307:3306
environment:
MYSQL_DATABASE: myDb
MYSQL_ROOT_PASSWORD: SomeRootPassword1!
MYSQL_USER: someuser
MYSQL_PASSWORD: Password1!
volumes:
- ./dbScript/winit_Script2.sql:/docker-entrypoint-initdb.d/winit_Script2.sql
- db_data:/var/lib/mysql
phpmyadmin:
image: phpmyadmin/phpmyadmin
container_name: dev_pma
links:
- mysql
environment:
PMA_HOST: mysql
PMA_PORT: 3307
PMA_ARBITRARY: 1
restart: always
ports:
- 8183:80
volumes:
db_data:
phpAdmin:
Mysqlworkbench:
What have I done wrong here?
A little edit after the comments:
It would seem that when having a volumes section you create volumes in docker
and when you create a volume on a specific port once then it gets reused every time you do docker-compose up. This was the case for me.
More details in accepted answer.
The mysql image does not initialize the database if the volume is not clean.
When you are stopping and starting the database from the same compose file, the volume is always the same, hence you want the data to be persisted even after an app restart.
To force the re-initialization of the data, you can delete that docker volume(only if you no longer need that database! this cannot be undone):
First, stop and delete the containers.
Then list and delete the volume that persists the database:
docker volume ls
DRIVER VOLUME NAME
local <your-deployment-name>_db_data
docker volume rm <your-deployment-name>_db_data
Then run again the docker-compose up command and you'll be able to find the myDb in phpMyAdmin instead of somedb
Edit:
Unless you change yourself the entrypoint and rebuild the image to force it initialize your DB according to the ENV you're passing, even if the volume is not clean, the only option that comes to my mind is to create the new DB manually. Here is the conditional that skips the re-initialization of the DB and here is the script that is invoked if the volume is clean.
I have a requirement where I need to wait for a few commands before I seed the data for the database:
I have some Migration scripts that create the schema in the database (this command runs from my app container). After this executes, I want to seed data to the database.
As I read, the docker-entrypoint-initdb scripts is executed when the container is initialized. If I mount my seed.sql script to it, the data is seeded before the Migrate scripts. (The Migrate scripts actually drop all tables and create them from scratch). The seeded data is therefore lost.
How can I achieve this? (I cannot change the Migrate scripts)
Here's my docker-compose.yml file
version: '3'
services:
app:
build: .
# mount the current directory (on the host) to /usr/src/app on the container, any changes in either would be reflected in both the host and the container
volumes:
- .:/usr/src/app
# expose application on localhost:36081
ports:
- "36081:36081"
# application restarts if stops for any reason - required for the container to restart when the application fails to start due to the database containers not being ready
restart: always
environment:
MIGRATE: Y
<some env variables here>
config-dev:
image: mysql/mysql-server:5.7
environment:
MYSQL_DATABASE: config_dev
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
volumes:
# to persist data
- config-dev-volume:/var/lib/mysql
restart: always
# to connect locally from SequelPro
ports:
- "1200:3306"
<other database containers>
My Dockerfile for app container has the following ENTRYPOINT
# start the application
ENTRYPOINT /usr/src/app/docker-entrypoint.sh
Here's the docker-entrypoint.sh file
#!/bin/bash
if [ "$MIGRATE" = "Y" ];
then
<command to start migration scripts>
echo "------------starting application--------------"
<command to start application>
else
echo "------------starting application--------------"
<command to start application>
fi
Edit: Is there a way I can run a script in config-db container from the docker-entrypoint.sh file in app container?
This can be solved in two steps:
You need to wait until your db container is started and is ready.
Wait until started can be handled by adding depends_on in docker-compose file:
version: '3'
services:
app:
build: .
# mount the current directory (on the host) to /usr/src/app on the container, any changes in either would be reflected in both the host and the container
depends_on:
- config-dev
- <other containers (if any)>
volumes:
- .:/usr/src/app
# expose application on localhost:36081
ports:
- "36081:36081"
# application restarts if stops for any reason - required for the container to restart when the application fails to start due to the database containers not being ready
restart: always
environment:
MIGRATE: Y
<some env variables here>
config-dev:
image: mysql/mysql-server:5.7
environment:
MYSQL_DATABASE: config_dev
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
volumes:
# to persist data
- config-dev-volume:/var/lib/mysql
restart: always
# to connect locally from SequelPro
ports:
- "1200:3306"
<other database containers>
Wait until db is ready is another case because sometimes it takes time for the db process to start listening on the tcp port.
Unfortunately, Docker does not provide a way to hook onto container state. There are many tools and scripts to have a workaround this.
You can go through this to implement the workaround.
https://docs.docker.com/compose/startup-order/
TL;DR
Download https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh inside the container and delete the ENTRYPOINT field (Not required for your use case) and use CMD field instead:
CMD ["./wait-for-it.sh", "<db_service_name_as_per_compose_file>:<port>", "--", "/usr/src/app/docker-entrypoint.sh"]
Now, That this is complete. Next part is to execute your seed.sql script.
That is easy and can be executed by adding following line into your /usr/src/app/docker-entrypoint.sh script.
sqlcmd -S -U -P -i inputquery_file_name -o outputfile_name
Place above command after migrate script in /usr/src/app/docker-entrypoint.sh
I am trying to set up a mysql docker container and execute init sql script. Unfortunately the sql script is not executed. What am I doing wrong?
version: '3.3'
services:
api:
container_name: 'api'
build: './api'
ports:
- target: 8080
published: 8888
protocol: tcp
mode: host
volumes:
- './api:/go/src/app'
depends_on:
- 'mysql'
mysql:
image: 'mysql:latest'
container_name: 'mysql'
volumes:
- ./db_data:/var/lib/mysql:rw
- ./database/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
restart: always
environment:
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: test
MYSQL_DATABASE: test
ports:
- '3306:3306'
volumes:
db_data:
I execute file with docker-compose up -d --build
The docker-entrypoint-initdb.d folder will only be run once while the container is created (instantiated) so you actually have to do a docker-compose down -v to re-activate this for the next run.
If you want to be able to add sql files at any moment you can look here at a specialized MySql docker image... http://ivo2u.nl/o4
Update for M1 arch:
Here an almost drop-in replacement in MariaDB: http://ivo2u.nl/V1
Many containerized applications, especially stateful ones, have a way of running init scripts (like the sql scripts here) and they are supposed to run only once.
And since they are stateful, the volumes are a source of truth for the containers on whether to run the init scripts or not on container restart.
Like in your case, deleting the folder used for bind mount or using a new named volume should re-run any init scripts present.
These scripts run when you create the container, not every time you start it.
You can docker-compose up --force-recreate mysql to force those scripts to re-run.
Additionally, if you have a volume like this ./db_data:/var/lib/mysql:rw, then you also need to remove ./db_data before recreating the container.
I'm not a docker expert, but this worked for me.
I am having trouble to load in my SQL File to create the schema and the associated tables, I have also tried to use the 'volumes' option in the compose file to copy the sql file into the 'docker-entrypoint-initdb.d' directory but that fails to so anything.
Table + Schema: https://pastebin.com/RTBBGZhn
MySQL Dump: https://pastebin.com/6ApQwt1F
Docker Compose File
version: '2'
services:
melissabot:
image: melissabot
build: .
ports:
- 7000:7000
- 7070:7070
depends_on:
- mysqlMelissa
links:
- mysqlMelissa:db
mysqlMelissa:
image: mysql
build: ${PWD}/Docker/DB/
environment:
- MYSQL_ROOT_PASSWORD=root
# - MYSQL_DATABASE=MelissaBot
ports:
- "3306:3306"
phpmyadmin:
image: phpmyadmin/phpmyadmin
container_name: phpmyadmin
restart: always
ports:
- 8080:80
volumes:
- /sessions
links:
- mysqlMelissa:db
Dockerfile
FROM mysql:5.6
ENV MYSQL_ROOT_PASSWORD=root
# ENV MYSQL_DATABASE=MelissaBot
COPY setup.sh /mysql/setup.sh
COPY dump.sql /mysql/Melissa.sql
RUN chmod +x /mysql/setup.sh
RUN /mysql/setup.sh
setup.sh
#!/bin/bash
set -e
service mysql start
mysql -u root MelissaBot < /mysql/Melissa.sql
service mysql stop
You are doing it wrongly. There is no init system inside docker. So you should not be using server start/stop inside a container. Look at the official image documentation always
https://hub.docker.com/_/mysql
Initializing a fresh instance
When a container is started for the first time, a new database with the specified name will be created and initialized with the provided configuration variables. Furthermore, it will execute files with extensions .sh, .sql and .sql.gz that are found in /docker-entrypoint-initdb.d. Files will be executed in alphabetical order. You can easily populate your mysql services by mounting a SQL dump into that directory and provide custom images with contributed data. SQL files will be imported by default to the database specified by the MYSQL_DATABASE variable.
So you can either copy your default dump files to /docker-entrypoint-initdb.d or mount them using a volume mount when running the container. So you chuck your setup.sh and change your Dockerfile to something like below
FROM mysql:5.6
ENV MYSQL_ROOT_PASSWORD=root
ENV MYSQL_DATABASE=MelissaBot
COPY dump.sql /docker-entrypoint-initdb.d/
But I would rather prefer the approach of volume mounting it inside my compose file
mysqlMelissa:
image: mysql:5.6
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=MelissaBot
ports:
- "3306:3306"
volumes:
- dump.sql:/docker-entrypoint-initdb.d/dump.sql