Docker compose wait for database service initialisation - mysql

I have a spring boot project which I'd like to containerize using docker.
I have a couple of spring boot applications which connect to same MySql server.
My spring applications requires the database to be completely setup (i.e. all the tables to be created and some data to be inserted in some of the tables) in order to start.
I am using Docker version 18.09.0 and docker-compose version 1.23.1 and ubuntu 16.04 LTS
I have two files create.sql and insert.sql, which I use to initialise the database to be used by the application.
I create the images using the command docker-compose.yml and it runs successfully and creates the images.
I have the following questions.
I assume when using docker-compose, a container starts as soon as all its dependent containers have started. Is there a way to wait for the mysql server to be up and ready to accept connections, before my API container gets started?
If I chose to create containers separately for the applications and mysql, and not use docker-compose, how do I make sure that my applications connect to the mysql container?
Is there any other tool which might help me achieve this?
Note:
I have tried to use docker inspect <container_id> to find the the IpAddress for the mysql container and use it to connect, but it doesn't work as well.
The following are the files I am using to create images.
docker-compose.yml file.
version: '3'
services:
demo-mysql:
image: demo-mysql
build: ./demo-mysql
volumes:
- /mnt/data/mysql-data:/var/lib/mysql
ports:
- 3306:3306
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=demo
- MYSQL_PASSWORD=root
demo-api:
image: demo-api-1.0
build: ./api
depends_on:
- demo-mysql
ports:
- 8080:8080
environment:
- DATABASE_HOST=demo-mysql
- DATABASE_USER=root
- DATABASE_PASSWORD=root
- DATABASE_NAME=demo
- DATABASE_PORT=3306
demo1-app:
image: demo1-app-1.0
build: ./demo1
depends_on:
- demo-mysql
ports:
- 8090:8090
environment:
- DATABASE_HOST=demo-mysql
- DATABASE_USER=root
- DATABASE_PASSWORD=root
- DATABASE_NAME=demo
- DATABASE_PORT=3306
The following is the Dockerfile for the spring boot project
FROM java:8
VOLUME /tmp
ARG DATA_PATH=/src/main/resources
ARG APP_PORT=8080
EXPOSE ${APP_PORT}
ADD /build/libs/demo-api.jar demo-api.jar
ENTRYPOINT ["java","-jar","demo-api.jar"]
The following is the Dockerfile I used to create my mysql image
FROM mysql:5.7
ENV MYSQL_DATABASE=demo \
MYSQL_USER=root \
MYSQL_ROOT_PASSWORD=root
ADD ./1.0/create.sql /docker-entrypoint-initdb.d
ADD ./1.0/insert.sql /docker-entrypoint-initdb.d
EXPOSE 3306

Use the healthcheck feature of docker-compose (https://docs.docker.com/compose/compose-file/#healthcheck).
Something like this:
services:
demo-mysql:
image: demo-mysql
build: ./demo-mysql
volumes:
- /mnt/data/mysql-data:/var/lib/mysql
ports:
- 3306:3306
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=demo
- MYSQL_PASSWORD=root
healthcheck:
test: ["CMD-SHELL", 'mysqladmin ping']
interval: 10s
timeout: 2s
retries: 10
The depending containers will not start until the demo-mysql container is healthy

After trying several approaches, IMO the simplest and most elegant option is using the jwilder/dockerize utility image with its -wait flag. Here is a simple example where I need a MySQL database to be ready before starting my app:
version: "3.8"
services:
# Start MySQL.
db:
image: mysql
# Wait for MySQL to be joinable.
check-db-started:
image: jwilder/dockerize:0.6.1
depends_on:
- db
command: 'dockerize -wait=tcp://db:3306'
# Only start myapp once MySQL is joinable.
myapp:
image: myapp:latest
depends_on:
- check-db-started

Related

Can't connect to MySQL container from other container

I'm trying to connect my FASTAPI app container to a MySQL database container using the docker-compose file. In the Docker documentation it says that docker creates a default network for both containers. However, I would like to use a pre-existing network that I've created(app-net).
This is my docker-compose file:
version: '3.4'
services:
mysql:
image: mysql:latest
command: mysqld --default-authentication-plugin=mysql_native_password
container_name: mysql
restart: always
ports:
- 3307:3306
environment:
MYSQL_ROOT_PASSWORD: password
volumes:
- mysql-data:/var/lib/mysql
app:
build: .
image: app:1.0
container_name: app
ports:
- 8000:8000
environment:
PASSWORD: password
USERNAME: root
networks:
default:
external: true
name: app-net
volumes:
mysql-data:
driver: local
this is the output I get when i run docker inspect mysql -f "{{json .NetworkSettings.Networks }}":
{"app-net":{"IPAMConfig":null,"Links":null,"Aliases":["mysql","5e998f9fb646"],"NetworkID":"7f60c83e4c88d25e674461521446ec9fa98baca8639c782c79671c4fcb77ba88","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"","DriverOpts":null}}
However, when I run each container individually using CMD with the --network app-net the output is different:
{"app-net":{"IPAMConfig":null,"Links":null,"Aliases":["46157e588c87"],"NetworkID":"7f60c83e4c88d25e674461521446ec9fa98baca8639c782c79671c4fcb77ba88","EndpointID":"6a6922a9a6ea8f9d113447decbbb927cb93ddffd3b9563ee882fa2e44970cde5","Gateway":"172.20.0.1","IPAddress":"172.20.0.2","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:ac:14:00:02","DriverOpts":null}}
In my app code in order to connect the mysql server, I specified the container name as the hostname since they are supposed to share the same network. But, as I mentioned it seems both containers can't talk to each other.
I'm pretty sure that is the reason I can't connect the database through my app and get that error when I run:
docker-compose -f docker-compose.yml up
I get this error:
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2005, "Unknown MySQL server host 'mysql' (0)")
What am I missing?
If the error you're getting is specifically "unknown host" or something similar, this can happen if your application container starts before the database container exists. You can work around this situation by telling Compose about the dependency:
version: '3.8'
services:
mysql: { ... }
app:
depends_on:
- mysql
This has two key effects. If you try to start only the application container docker-compose up app, it will also start the mysql container, following the depends_on: chain. When Compose does start containers, it at least creates the mysql container before creating the app container.
Note that this doesn't guarantee the database will actually be running by the time your application starts up. If you do encounter this, you will get a different error like "connection refused". You can work around this by embedding a script in your image that waits for the database to be available, or by using a version 2 Compose file with health-check support; see Docker Compose wait for container X before starting Y for more details on this specific problem.
You could add the key networks to both containers, this way:
version: '3.4'
services:
mysql:
image: mysql:latest
command: mysqld --default-authentication-plugin=mysql_native_password
container_name: mysql
restart: always
ports:
- 3307:3306
environment:
MYSQL_ROOT_PASSWORD: password
volumes:
- mysql-data:/var/lib/mysql
networks:
- app-net
app:
build: .
image: app:1.0
container_name: app
ports:
- 8000:8000
environment:
PASSWORD: password
USERNAME: root
networks:
- app-net
networks:
default:
external: true
name: app-net
volumes:
mysql-data:
driver: local

SQLSTATE[HY000] [2002] Connection refused (SQL: select * from information_schema.tables where table_schema = ms_api_shop

I am using Lumen with Docker to create simple API for authentication. After installing LumenPassport, I cannot migrate the database. I can easily connect to the MySQL db with Dbeaver.
I have already created one Lumen Docker project for the same purpose, it is the second. The first one worked without a problem. Moreover, I have checked the MySQL databases, ms_api_shop was there
Errors:
Here is my docker-compose
services:
nginx:
build:
context: .
dockerfile: docker/Nginx.Dockerfile
image: nginx
ports:
- 8092:80
depends_on:
- fpm
volumes:
- ./:/var/www/lumen-docker
links:
- mysql
fpm:
build:
context: .
dockerfile: docker/fpm.Dockerfile
volumes:
- ./:/var/www/lumen-docker
depends_on:
- mysql
links:
- mysql
mysql:
image: mysql:5.7
ports:
- 33006:3306
environment:
- MYSQL_ROOT_PASSWORD=
- MYSQL_DATABASE=ms_api_shop
- MYSQL_ROOT_USER=
volumes:
- mysql-data:/var/lib/mysql
volumes:
mysql-data:
And env:
DB_HOST=mysql
DB_PORT=33006
DB_DATABASE=ms_api_shop
DB_USERNAME=
DB_PASSWORD=
In your docker-file, you are binding the 33006 container port to the 3306 of the host port. In case you want to access the MySQL, you should use 3306 not 33006 as you did in your .env
I have a Laravel app running in Docker and a part of my docker config looks like:
But I personally feel that they should be within a docker network, look at the last two lines of the code in my docker-compose.yml below:
mysql:
image: mysql:5.7.29
container_name: mysql
restart: unless-stopped
tty: true
ports:
- "3306:3306"
environment:
MYSQL_DATABASE: homestead
MYSQL_USER: homestead
MYSQL_PASSWORD: secret
MYSQL_ROOT_PASSWORD: secret
SERVICE_TAGS: dev
SERVICE_NAME: mysql
volumes:
- ./mysql:/var/lib/mysql
networks:
- laravel
According to my knowledge, docker containers can communicate with each other when they are on the same network. You seem to have not connected these two docker in docker-compose yet. To get network information from a container, you can use the below command.
$ docker inspect --format='{{json .NetworkSettings.Networks}}' <docker_container_name>
In case you need to connect these two containers, follow these steps:
First, you create a network
$ docker network create my-net
Second, you connect container to the network.
$ docker network connect my-net <docker_container_name>
Don't forget to connect these two containers to the network.

Docker-compose up for a spring service and a mysql service works, but init.sql does is not loaded and Postman return empty data

I am working for the first time with Docker. In my job, they asked me to deploy a complete application using docker-compose and I am learning right now.
I have a docker-compose.yml that starts two services: one for spring and another one for MySQL.
The one for Spring uses a Dockerfile to build the image.
The one for MySQL uses the official MySQL image.
I don´t know how to load the init.sql to initialize and load data into the database of the MySQL container.
I´ve tried to use another Dockerfile in order to copy init.sql into the MySQL container but,
how do I tell the compose to use it when creating the image or running the container?
This is for a Windows 10 OS Desktop. Docker version 18.09.2. Docker-compose version 1.23.2
Docker Desktop
Docker-compose.yml
version: '3.7'
services:
mysql-docker-container:
image: mysql
container_name: mysql-docker-container
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=database
- MYSQL_USER=user
- MYSQL_PASSWORD=user
ports:
- 2012:3306
tty: true
spring-jpa-app:
build:
context: .
dockerfile: Dockerfile
container_name: spring-jpa-app
links:
- mysql-docker-container:mysql-docker-container
depends_on:
- mysql-docker-container
ports:
- 8087:8080
tty: true
restart: always
Dockerfile
FROM java:8
VOLUME /tmp
EXPOSE 8080
ADD app-1.0.0.jar app-1.0.0.jar
ENTRYPOINT ["java","-jar","app-1.0.0.jar"]
application.properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://mysql-docker-container:2012/database?autoReconnect=true&useSSL=false
spring.datasource.username=user
spring.datasource.password=user
spring.jpa.database=database
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
logging.file=log/log.log
logging.level.org.springframework=info
logging.level.orghibernate=info
Docker Desktop running
Opened Git Bash as administrator
docker-machine ls
NAME ACTIVE DRIVER STATE URL DOCKER
app * hyperv Running tcp://172.18.67.35:2376 v19.03.0
In the app folder, where is located /src
mvn clean install -DskipTests
BUILD SUCCESS
In the app folder where is located application.properties, app.jar, docker-compose.yml, Dockerfile and init.sql
docker-compose up --build
Postman
URL = http://172.18.67.35:8087
POST {{url}}/login
{
"Authorization": "gnioengrlnkwejnR",
"username": "Administrator",
"role": "ADMIN",
"id": "1"
}
GET {{url}}/listAllCenters
[]
Returns empty because init.sql is not being loaded
PD: Also, if I open a new git Bash as Administrator and do
docker ps -a
I do not get the containers running so I cannot know container's ID to enter the MySQL container.
you need to add a volume to copy the sql.init to your db continer:
version: '3.7'
services:
mysql-docker-container:
image: mysql
container_name: mysql-docker-container
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=database
- MYSQL_USER=user
- MYSQL_PASSWORD=user
ports:
- 2012:3306
volumes:
- /bath/to/file/on/host/init.sql:/docker-entrypoint-initdb.d/init.sql
tty: true
network_mode: "host"
spring-jpa-app:
build:
context: .
dockerfile: Dockerfile
container_name: spring-jpa-app
links:
- mysql-docker-container:mysql-docker-container
depends_on:
- mysql-docker-container
ports:
- 8087:8080
tty: true
restart: always
network_mode: "host"
see this and search for "Initializing a fresh instance"
on the other hand you need to use network to setup the connection between the containers - see the compose file above-
and you need to change you configs to
spring.datasource.url=jdbc:mysql://localhost:2012/database?autoReconnect=true&useSSL=false

How to connect a Docker container of Grafana to a Docker conatiner of MySql?

I wish to set up 2 docker containers: grafana and mysql and to allow queries from the grafana to the mysql db.
I have an AWS machine, in which I built the following folders structure:
- docker
- docker-compose.yml
- grafana:
- config.ini
- dashboards:
- grafana_dashboard.json
- provisioning:
- dashboards:
- all.yml
- datasources:
- all.yml
- Dockerfile
- mysql:
- dbcreation.sql
- Dockerfile
- dashboards:
- import.sh
the content of the docker-compose.yml is:
version: '2'
services:
db-service:
build: './mysql'
container_name: mysql
restart: always
ports:
- "3306:3306"
networks:
net:
ipv4_address: 172.16.1.3
grafana-service:
build: './grafana'
container_name: grafana
restart: always
ports:
- "3000:3000"
environment:
GF_SECURITY_ADMIN_PASSWORD: "XXX1"
GF_AUTH_PROXY_ENABLED: "true"
GF_SECURITY_DATA_SOURCE_PROXY_WHITELIST: 172.16.1.3:3306
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_LOG_LEVEL: "debug"
depends_on:
- db-service
networks:
net:
ipv4_address: 172.16.1.4
networks:
net:
external: true
volumes:
grafanadata:
driver: local
mysqldata:
The dockerfile for the grafana:
FROM grafana/grafana:5.2.2
ADD ./provisioning /etc/grafana/provisioning
ADD ./dashboards /var/lib/grafana/dashboards
ENV DS_DB "grafana"
The content of the mysql/Dockerfile is:
FROM mysql:8.0.12
ENV MYSQL_ROOT_PASSWORD="XXX2"
ENV MYSQL_DATABASE="grafana"
ADD ./dbcreation.sql /docker-entrypoint-initdb.d/dbcreation.sql
EXPOSE 3306
The grafana_dashboard.json file has the exported json from the Grafana I had set up locally on my own computer.
The dbcreation.sql file has the exported data from the local DB I had set up locally on my own computer.
I'm running the following commands:
docker network create --gateway 172.16.1.1 --subnet 172.16.1.0/24 net
docker-compose up --build
I'm getting an error: "The authentication plugin is not supported"
when turning the log level of Grafana to debug I'm seeing this:
t=2018-08-19T10:55:20+0000 lvl=dbug msg=getEngine logger=tsdb.mysql connection="root:XXX2#tcp(172.16.1.3:3306)/grafana?collation=utf8mb4_unicode_ci&parseTime=true&loc=UTC&allowNativePasswords=true"
t=2018-08-19T10:55:47+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/tsdb/query status=500 remote_addr=XX.XXX.XXX.XXX time_ms=2 size=195 referer=http://XX.XXX.XXX.XXX:3000/datasources/edit/1
I have used the following sources already to set this up:
https://ops.tips/blog/initialize-grafana-with-preconfigured-dashboards/
https://storage.pardot.com/138181/61672/mysql_on_docker_how_to_containerize_your_database.pdf
Any help would be appreciated!
Thanks
As stated here:
go-mysql: authentication plugin not supported while connecting from go app container to mysql container
The problem was compatibility between the Grafana versions and the MySQL version.
Once moving to docker image mysql:5.7 (had to migrate the data too) - the issue was resolved (need to also change the COLLATION and CHAR_SET in the DB to downgrade from version 8.0.12 to 5.7).

How to create Mysql test database in dockerized Rails application?

I managed to dockerize my existing Rails application running on a Mysql database. But I wonder if it is possible to setup docker-compose to create the test database in the same container?
Here is my docker-compose.yml and it wirks fine with the mysql for developing
version: '2'
volumes:
db-data:
services:
db:
image: mysql:5.5
restart: always
ports:
- "3307:3306"
environment:
MYSQL_ROOT_PASSWORD: verysecret
MYSQL_USER: appdb
MYSQL_PASSWORD: secret
MYSQL_DATABASE: appdb
volumes:
- db-data:/var/lib/mysql
web:
build: .
command: bundle exec rails s -p 3000
volumes:
- .:/app
ports:
- "3000:3000"
links:
- db
depends_on:
- db
Can I add a darabase more in environment part somehow?
You can create only one DB per Mysql container with docker compose. In your case, I think you should create a second DB container for the second database (isolated from the "real" DB, which is a good practice).
If you really want to have the 2 databases in the same container, you will have to create a Dockerfile based on the Mysql image, and add your command lines (RUN) to create the second DB.
HTH
Yes, you can, but it won't necessarily be as a single container, but a manager for coupling containers. If this is okay, I've added some steps that will help you configure your project. If not, I've added how to run a single image from a docker-compose file
You will want to start off by creating a docker-compose.yml file in the source directory for your project.
Then you'll want to add something like this inside your yml file.(this was taken from Docker's quick-start documentation. Modified to show mysql instead of postgres)
version: '3'
services:
db:
image: mysql
web:
build: .
command: bundle exec rails s -p 3306 -b '0.0.0.0'
volumes:
- .:/myapp
ports:
- "3000:3000"
depends_on:
- db
Detail on how they create one can be found here:
https://docs.docker.com/compose/rails/#define-the-project.
Things to note:
web is the details about the rails container. So you will want to add an image property if you have already created your rails image.
Also, build: . is expecting your Dockerfile to be in the same location as your project. So if you create this docker-compose.yml somewhere else, you'll have to provide the path.
depends_on allows your app to build the DB before running rails
Once you finished creating the docker-compose.yml file run:
docker-compose build
followed by:
docker-compose up
If separating the containers, isn't what you are looking for: then you might want to look into creating a single image running both applications. Then use something like this to run from docker-compose.
version: '3'
services:
app:
image: {your-app-image}
build: .
volumes:
- .:/myapp
ports:
- "3000:3000"
- "3306:3306"
Note: somethings might vary on how you create your image from the Dockerfile.