Docker - how to alter my Docker Compose file to automate a bash script for the MySQL container? - mysql

I have the following setup Docker Composer setup and want to run a shell script to automate tasks like importing the DB into the MySQL database.
# Adopt version 2 syntax:
version: '2'
volumes:
database_data:
driver: local
services:
###########################
# Setup the Nginx container
###########################
nginx:
image: nginx:latest
ports:
- 8080:80
volumes:
- ./docker/nginx/default.conf:/etc/nginx/conf.d/default.conf
volumes_from:
- php
###########################
# Setup the PHP container
###########################
php:
build: ./docker/php/
expose:
- 9000
volumes:
- .:/var/www
###########################
# Setup the Database (MySQL) container
###########################
mysql:
image: mysql:latest
expose:
- 3306
volumes:
- database_data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
MYSQL_DATABASE: project
MYSQL_USER: project
MYSQL_PASSWORD: project

Best solution is to create a custom Dockerfile ,which extends mysql and add a custom shell script, which does what you want. For example:
start.sh
#!/bin/sh
mysqld
mysql -u project -ppropject project < /path/to/backup.sql
Don't forget to add your backup.sql either to your Dockerfile or docker-compose.yml
Now, Dockerfile:
FROM mysql:latest
COPY start.sh /tmp/start.sh
COPY backup.sql /path/to/backup.sql
CMD ["/tmp/start.sh"]
If you change your backup.sql frequently, it makes no sense to add it to Dockerfile. Instead, put it under volumes in docker-compose.yml:
mysql:
build: .
expose:
- 3306
volumes:
- ./backup.sql:/path/to/backup.sql
- database_data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
MYSQL_DATABASE: project
MYSQL_USER: project
MYSQL_PASSWORD: project

You can keep using a the original image: load your setup script into the container as a Config (using the long-form definition so you can set the execute permission), and then override the Entrypoint to run your script (which should probably run the original entrypoint script once it finishes). So something like:
mysql:
image: mysql:latest
expose:
- 3306
volumes:
- database_data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
MYSQL_DATABASE: project
MYSQL_USER: project
MYSQL_PASSWORD: project
configs:
- source: ./OverrideScript.sh
target: /OverrideScript.sh
#0777 will work too, but you can't write to it either way
mode: 0555
entrypoint: /OverrideScript.sh
The other answers are right that the "Proper" way would be to make your own image. But TBH if your override script is relatively small and lightweight, the workaround isn't so bad, and it gets you out of having to rebuild your custom image every time MySQL releases a new image.

Related

Docker importing a sql script on mysql container

How can this docker script be modified to allow a sql file to be imported into the mysql container? I need to modify the database on the mysql container.
version: '3'
services:
devbox:
build:
context: ./
dockerfile: DevBox.DockerFile
ports:
- "80:80"
- "443:443"
volumes:
- .:/var/gen4
- ./offers:/var/www/vhosts/offers
devmysql:
image: mysql:5.7
platform: linux/x86_64
environment:
MYSQL_ROOT_PASSWORD: mypwd
MYSQL_DATABASE: offers
ports:
- "3306:3306"
restart: always
The official MySQL images support creating a volume called /docker-entrypoint-initdb.d/
So in your devmysql section of your compose file do something like this
volumes:
- ./data:/docker-entrypoint-initdb.d/:ro
In this case, you'd want to have a data/ folder in the root of your project (wherever compose is being run from) and in that data/ folder you can put a SQL file with whatever commands you want. They'll be run.
If you're not running the official images, you might be able to create your own image that manually does something similar.

Spring Boot + MySQL + Docker Compose - Cannot make Spring Boot connect to MySQL

I've been trying to set up a connection between a backend (runs on Spring Boot) container and a pre-built MySQL container. However, I cannot get it to connect. My docker compose file is:
version: '3.7'
services:
test-mysql:
image: mysql
restart: always
volumes:
- db_data:/var/lib/mysql
environment:
MYSQL_DATABASE: testdb
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
backend:
depends_on:
- test-mysql
build:
context: backend
dockerfile: Dockerfile
ports:
- "8080:8080"
restart: always
volumes:
db_data: {}
my application.properties:
spring.jpa.hibernate.ddl-auto=update
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5InnoDBDialect
spring.datasource.url=jdbc:mysql://test-mysql:3306/testdb?autoReconnect=true&failOverReadOnly=false&maxReconnects=10
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.username=test
spring.datasource.password=test
When I use docker-compose up, Spring Boot is not able to recognize the container name test-mysql. It throws: java.net.UnknownHostException
When I change it to an IP, it says connection refused. I have been looking everywhere and couldn't come with a fix. I hope anyone can help me out. Thank you!
You have to mention the backend mysql properties in the composer file like below,
backend:
depends_on:
- test-mysql
build:
context: backend
dockerfile: Dockerfile
ports:
- "8080:8080"
restart: always
environment:
SPRING_DATASOURCE_URL: jdbc:mysql://test-
mysql:3306/testdbautoReconnect=true&failOverReadOnly=false&maxReconnects=10
SPRING_DATASOURCE_USERNAME: test
SPRING_DATASOURCE_PASSWORD: test
links:
- test-mysql:test-mysql
If this wouldn't work try to create a common docker network and add it to your composer file like below,
backend:
depends_on:
- test-mysql
build:
context: backend
dockerfile: Dockerfile
ports:
- "8080:8080"
restart: always
environment:
SPRING_DATASOURCE_URL: jdbc:mysql://test-
mysql:3306/testdbautoReconnect=true&failOverReadOnly=false&maxReconnects=10
SPRING_DATASOURCE_USERNAME: test
SPRING_DATASOURCE_PASSWORD: test
networks:
-common-network
test-mysql:
image: mysql
restart: always
volumes:
- db_data:/var/lib/mysql
environment:
MYSQL_DATABASE: testdb
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
networks:
-common-network
#Docker Networks
networks:
common-network:
driver: bridge
#Volumes
volumes:
dbdata:
driver: local
You can define a common network on which both the application server and the database can connect. Please check the file (docker-compose.yml) below where I have defined a common network: backend
# Docker Compose file Reference (https://docs.docker.com/compose/compose-file/)
version: '3.7'
# Define services
services:
# App backend service
app-server:
# Configuration for building the docker image for the backend service
build:
context: . # Use an image built from the specified dockerfile in the `springboot-app-server` directory.
dockerfile: Dockerfile
ports:
- "8080:8080" # Forward the exposed port 4000 on the container to port 4000 on the host machine
restart: always
depends_on:
- db # This service depends on mysql. Start that first.
environment: # Pass environment variables to the service
SPRING_DATASOURCE_URL: jdbc:mysql://test-mysql:3306/testdb?useSSL=false&serverTimezone=UTC&useLegacyDatetimeCode=false
SPRING_DATASOURCE_USERNAME: test
SPRING_DATASOURCE_PASSWORD: test
networks: # Networks to join (Services on the same network can communicate with each other using their name)
- backend
# Database Service (Mysql)
db:
image: mysql:5.7
ports:
- "3306:3306"
restart: always
environment:
MYSQL_DATABASE: testdb
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
volumes:
- db-data:/var/lib/mysql
networks:
- backend
# Volumes
volumes:
db-data:
# Networks to be created to facilitate communication between containers
networks:
backend:
I have written a blog and a simple working Spring Boot MySQL application on GitHub which tells about using Docker Compose. Please check: http://softwaredevelopercentral.blogspot.com/2020/10/spring-boot-mysql-docker-compose-example.html
If you would like to use this test-mysql in your spring config
spring.datasource.url=jdbc:mysql://test-mysql:3306/testdb?autoReconnect=true&failOverReadOnly=false&maxReconnects=10
Then add the hostname attribute at service test-mysql
version: '3.7'
services:
test-mysql:
image: mysql
hostname: test-mysql
...
I hope this has been already solved but in case it hasn't yet, the problem lies in mysql docker container lagging behind the start up.
Another problem is that you might need to build the jar file and then copy it the container. That's a big problem because when you build the jar file, the database with db as hostname is unavailable. So when you are building the jar file, skip the test.
This is bash script i created but you can run command one by one:
#!/bin/bash
cd storage-service
rm -rf target/
mvn clean compile package -Dmaven.test.skip=true
cd ..
docker-compose up
In case you want to initialize db in the container. That file is in the folder env where i have a file database.env
-- create the databases
CREATE DATABASE IF NOT EXISTS model_storage;
-- create the users for each database
CREATE USER 'arsene'#'localhost' IDENTIFIED BY 'arsene';
GRANT CREATE, ALTER, INDEX, LOCK TABLES, REFERENCES, UPDATE, DELETE, DROP, SELECT, INSERT ON `model_storage`.* TO 'arsene'#'localhost';
FLUSH PRIVILEGES;
The backend service Dockerfile looks like this:
FROM adoptopenjdk/openjdk11
COPY target/*.jar storage.jar
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /storage.jar" ]
EXPOSE 8089
The database env file looks like this:
MYSQL_ROOT_PASSWORD=arsene
MYSQL_DATABASE=model_storage
MYSQL_USER=arsene
MYSQL_PASSWORD=arsene
DATABASE_HOST=model_storage
DATABASE_USER=arsene
DATABASE_PASSWORD=arsene
DATABASE_NAME=model_storage
DATABASE_PORT=3306
In case you intend to pass JAVA_OPTS env in the image. These can be used later as seen in docker-compose.yml below
Your backend (the service that depends on the mysql db) needs to restart until the docker-compose is able to resolve the the container name of mysql, in my case its name is db. And don't forget to include datasource connection properties in docker-compose backend service image as i did below. I am not an expert in spring boot and neither in docker but for now it works!
Below is the way mine is structured:
I am using docker version: "3.8"
Storage service
storage-service:
container_name: storage-service
restart: always
build:
context: storage-service
image: "service_storage_image"
depends_on:
- db
ports:
- "8089:8089"
links:
- db
env_file:
- env/database.env
environment:
WAIT_HOSTS: db:3306
SPRING_DATASOURCE_URL: jdbc:mysql://db:3306/model_storage?allowPublicKeyRetrieval=true&useSSL=false
SPRING_DATASOURCE_USERNAME: root
SPRING_DATASOURCE_PASSWORD: arsene
healthcheck:
test: "/usr/bin/mysql --user=arsene --password=arsene--execute \"SHOW DATABASES;\""
interval: 2s
timeout: 20s
retries: 10
environment:
- JAVA_OPTS=
-DEUREKA_SERVER=http://eureka-registry-server:7070/eureka
-DZIPKIN_SERVER=http://zipkin:9411/
networks:
- private-network-mms
My db in docker-compose is structured this way:
Mysql database
db:
hostname: db
container_name: db
image: "mysql:latest"
env_file:
- env/database.env
volumes:
- type: bind
source: ./env/setup.sql
target: /docker-entrypoint-initdb.d/setup.sql
- db_volume:/var/lib/mysql
ports:
- 3307:3306
networks:
- private-network-mms

Why edit contents of docker-compose.yml are not reflected?

I want to use MySQL by using Docker.
I wrote the following DockerFile and docker-compose.yml.
Dockerfile
FROM mysql:8.0
RUN mkdir /var/log/mysql
RUN touch /var/log/mysql/mysqld.log
docker-compose.yml
version: '3'
services:
dbserver:
build: ./docker/mysql
image: test-db:0.0.1
restart: always
environment:
MYSQL_DATABASE: prototype
MYSQL_USER: user
MYSQL_PASSWORD: password
MYSQL_ROOT_PASSWORD: rootpassword
ports:
- "3306:3306"
volumes:
- ./docker/mysql/initdb.d:/docker-entrypoint-initdb.d
- ./docker/mysql/conf.d:/etc/mysql/conf.d
- ./log/mysql:/var/log/mysql
- ./docker/mysql/data:/var/lib/mysql
volumes:
mysql-bd:
driver: local
I succeeded build and could confirm the database.
Then I wanted to change the database name, so I edited a part of the yml file following.
Before
MYSQL_DATABASE: prototype
After
MYSQL_DATABASE: test_db
Then, I confirmed the database but its name was not changed.
I removed the MySQL container and tried again, but the result was not changed.
Why edit contents of docker-compose.yml are not reflected?
You are using a host volume for your database, meaning that the databases are persisted between containers restarts.
...
volumes:
./docker/mysql/data:/var/lib/mysql
...
Delete the local directory ./docker/mysql/data and restart your services. The database change will be reflected.

logging mysql queries from docker container

i am new to docker and I am still figuring out how it works. I have setup through docker-compose a wordpress site and the yaml file i am using is the following.
version: '3.3'
services:
db:
image: mysql:5.7
volumes:
- ./db_data/db/backup.sql:/docker-entrypoint-initdb.d/backup.sql #it loads an already existing database
#- ./db_data/all.log:/var/log/mysql/all.log # i want here to have the logs of mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: somewordpress
MYSQL_DATABASE: wordpress
MYSQL_USER: wordpress
MYSQL_PASSWORD: wordpress
wordpress:
depends_on:
- db
image: wordpress:latest
ports:
- "8000:80"
restart: always
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_USER: wordpress
WORDPRESS_DB_PASSWORD: wordpress
volumes:
- ./db_data/wp-content:/var/www/html/wp-content # loads the files from an existing installation of wordpress
volumes:
db_data:
as you can see by this line
- ./db_data/all.log:/var/log/mysql/all.log
i am trying to somehow make it so i can log all the queries made. I have managed to do that by doing the following:
1) docker-compose up, while having commented out the line with all.log from yaml file.
2) logging to the container of the mysql with bash and running the following
mysql -u root -psomewordpress -e "SET global log_output = 'FILE'; SET global general_log_file='/var/log/mysql/all.log'; SET global general_log = 1;"
3) Going to the /var/lib/docker/volumes and getting the file created.
What i am trying to achieve is to make somehow this command part of the yaml file so I can also have the all.log file mounted at the location where the yaml file is.
What I did to solve my problem was using the following:
version: '3.3'
services:
db:
image: mysql:5.7
volumes:
- ./db_data/db/backup.sql:/docker-entrypoint-initdb.d/backup.sql #it loads an already existing database
-./logs/mysql:/var/log/mysql # to get the folder of mysql in my logs/mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: somewordpress
MYSQL_DATABASE: wordpress
MYSQL_USER: wordpress
MYSQL_PASSWORD: wordpress
wordpress:
depends_on:
- db
image: wordpress:latest
ports:
- "8000:80"
restart: always
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_USER: wordpress
WORDPRESS_DB_PASSWORD: wordpress
volumes:
- ./db_data/wp-content:/var/www/html/wp-content
This code would sync what is inside the docker container of mysql with my folder logs/mysql. So I put a little shell script in there.
#!/bin/bash
if [ ! -f /var/log/mysql/all.log ]; then
touch /var/log/mysql/all.log
fi
chmod 777 /var/log/mysql/all.log
mysql -u root -psomewordpress -e "SET global log_output = 'FILE'; SET global general_log_file='/var/log/mysql/all.log'; SET global general_log = 1;"
It checks if the all.log file is there, if not it creates it and then starts mysql logging. Thats it, after that all queries are logged directly to all.log in which i have direct access.
One more thing, to execute that shell script you might want to use
docker exec 5.7-mysql "./var/log/mysql/initlogs.sh"
where initlogs.sh is the name of the shell script.
I really hope it would help someone!

Docker Compose mysql import .sql

I'm having trouble importing an .sql dump file with docker-compose. I've followed the docs, which apparently will load the .sql file from docker-entrypoint-initdb.d. However, when I run docker-compose up, the sql file is not copied over to the container.
I've tried stopping the containers with -vf flag, but that didn't work either. Am I doing something wrong in my .yml script?
I have dump.sql in the directory database/db-dump/ in the root where my compose file is.
frontend:
image: myimage
ports:
- "80:80"
links:
- mysql
mysql:
image: mysql
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: rootpass
MYSQL_USER: dbuser
MYSQL_PASSWORD: userpass
MYSQL_DATABASE: myimage_db
volumes:
- ./database/db-dump:/docker-entrypoint-initdb.d
This worked for me,
version: '3.1'
services:
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
volumes:
- ./mysql-dump:/docker-entrypoint-initdb.d
environment:
MYSQL_ROOT_PASSWORD: example
MYSQL_DATABASE: ecommerce
adminer:
image: adminer
restart: always
ports:
- 8080:8080
mysql-dump must be a directory. All the .sql's in the directory will be imported.
After many attempts with the volumes setting i found a workaround
I created another image based on mysql with the following in the Dockerfile
FROM mysql:5.6
ADD dump.sql /docker-entrypoint-initdb.d
Then removed the volumes from compose and ran the new image
frontend:
image: myimage
ports:
- "80:80"
links:
- mysql
mysql:
image: mymysql
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: rootpass
MYSQL_USER: dbuser
MYSQL_PASSWORD: userpass
MYSQL_DATABASE: myimage_db
This way the dump is always copied over and run on startup
This appears on the documentation page of Docker MySQL image: https://hub.docker.com/_/mysql/
Initializing a fresh instance
When a container is started for the first time, a new database with
the specified name will be created and initialized with the provided
configuration variables. Furthermore, it will execute files with
extensions .sh, .sql and .sql.gz that are found in
/docker-entrypoint-initdb.d. Files will be executed in alphabetical
order. You can easily populate your mysql services by mounting a SQL
dump into that
directory
and provide custom
images with contributed
data. SQL files will be imported by default to the database specified
by the MYSQL_DATABASE variable.
Mysql database dump schema.sql is residing in the /mysql-dump/schema.sql directory and it creates tables during the initialization process.
docker-compose.yml:
mysql:
image: mysql:5.7
command: mysqld --user=root
volumes:
- ./mysql-dump:/docker-entrypoint-initdb.d
environment:
MYSQL_DATABASE: ${MYSQL_DATABASE}
MYSQL_USER: ${MYSQL_USER}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
I was having a similar issue with mysql where I would mount a local directory at /configs/mysql/data containing a mydatabasedump.sql file via docker-compose to the docker-entrypoint-initdb.d volume,
the file would get loaded on to the container but not execute or populate the database when the container initialized. My intial docker-compose.yml looke like this:
#docker-compose.yml
version: '3'
services:
db:
build: ./build/mysql/ #this is pointing to my Dockerfile
container_name: MYSQL_Database
restart: always
environment:
MYSQL_PORT: 3306
MYSQL_ROOT_PASSWORD: admin
MYSQL_DATABASE: my_app_database
MYSQL_USER: admin
MYSQL_PASSWORD: admin
volumes:
- ./configs/mysql/data:/docker-entrypoint-initdb.d:
I found two working solutions for this problem:
The first came after I logged in the running container and confirmed that mydatabasedump.sq file was present and executable in the container's docker-entrypoint-initdb.d directory; I created and added
a bash script to my local /configs/mysql/data directory called dump.sh that excuted after the container was initialized. It contains a single mysql command that copies my_database_dump.sql to my_app_database.
The bash script looks like this
#!/bin/bash
#dump.sh
mysql -uadmin -padmin my_app_database < my_database_dump.sql
#end of dump.sh
I executed this script via my Dockerfile in the ENTRYPOINT directive like this:
#Dockerfile
FROM mysql:5.5
ENTRYPOINT [ "dump.sh" ]
EXPOSE 80
#end of Dockerfile
After realizing the initial issue was due to the volumes being mouted after the cotainer is built and therefore not intilizing the database with the dump file (or executing any scripts in that directory) at boot time, the second solution was simply to
move the volumes directive in my compose-file above the built directive. This worked and allowed me to remove the dump.sh scrip and the DOCKERENTRY directive in my Dockerfile.
The modified docker-compose.yml looks like this
#docker-compose.yml
version: '3'
services:
db:
volumes:
- ./configs/mysql/data:/docker-entrypoint-initdb.d
build: ./build/mysql/ #this is pointing to my Dockerfile
container_name: MYSQL_Database
restart: always
environment:
MYSQL_PORT: 3306
MYSQL_ROOT_PASSWORD: admin
MYSQL_DATABASE: my_app_database
MYSQL_USER: admin
MYSQL_PASSWORD: admin
I also have this problem. I mount a local directory at ./mysql-dump containing a init.sql file via docker-compose to the docker-entrypoint-initdb.d volume, the file would get loaded on to the container but not execute or populate the database when the container initialized.
My intial docker-compose.yml looke like this:
mysqld:
image: mysql
container_name: mysqld
volumes:
- ./mysql/data:/var/lib/mysql
- ./mysql/my.cnf:/etc/my.cnf
- ./init:/docker-entrypoint-initdb.d
env_file: .env
restart: always
environment:
- MYSQL_ROOT_PASSWORD=123456
- MYSQL_DATABASE=fendou
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
--default-authentication-plugin=mysql_native_password
but it doesn't work for me.
I found another working solutions for this problem:
add --init-file /data/application/init.sql to mysql command.change above configuration like
mysqld:
image: mysql
container_name: mysqld
volumes:
- ./mysql/data:/var/lib/mysql
- ./mysql/my.cnf:/etc/my.cnf
# - ./init:/docker-entrypoint-initdb.d
env_file: .env
restart: always
environment:
- MYSQL_ROOT_PASSWORD=123456
- MYSQL_DATABASE=fendou
command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
--default-authentication-plugin=mysql_native_password
--init-file /docker-entrypoint-initdb.d/init.sql #attention here
hope it help for you
I wanted to keep the original setup of the container, so I tried a restore on the already running container. This seemed to work:
cat dump.sql | docker-compose exec -T db mysql -h localhost -u root -psomewordpress -v
But it was very slow and the verbose output seemed to be buffered, so I tried:
docker-compose cp dump.sql db:/tmp/
docker-compose exec db sh -c "mysql -h localhost -u root -psomewordpress -v < /tmp/dump.sql"
Which at least provided faster feedback.
Might be useful for someone? Looks like it was mainly slow because I used --skip-extended-insert on the dump, without the extended inserts it went faster 🙂