Adding Flyway to a MySQL Docker Container - mysql

I'm building an derivative to this Docker container for mysql (using it as a starting point): https://github.com/docker-library/mysql
I've amended the Dockerfile to add in Flyway. Everything is set up to edit the config file to connect to the local DB instance, etc. The intent is to call this command from inside the https://github.com/docker-library/mysql/blob/master/5.7/docker-entrypoint.sh file (which runs as the ENTRYPOINT) around line 186:
flyway migrate
I get a connection refused when this is run from inside the shell script:
Flyway 4.1.2 by Boxfuse
ERROR:
Unable to obtain Jdbc connection from DataSource
(jdbc:mysql://localhost:3306/db-name) for user 'root': Could not connect to address=(host=localhost)(port=3306)(type=master) : Connection refused
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 08
Error Code : -1
Message : Could not connect to address=(host=localhost)(port=3306)(type=master) : Connection refused
But, if I remove the command from the shell script, rebuild and log in to the container, and run the same command manually, it works with no problems.
I suspect that there may be some differences with how the script connects to the DB to do its thing (it has a built in SQL "runner"), but I can't seem to hunt it down. The container restarts the server during the process, which is what may be the difference here.
Since this container is intended for development, one alternative (a work-around, really) is to use the built in SQL "runner" for this container, using the filename format that Flyway expects, then use Flyway to manage the production DB's versions.
Thanks in advance for any help.

I mean it's the good way to start from the ready image (for start).
You may start from image docker "mysql"
FROM mysql
If you start the finished image - when creating new version your docker then
will only update the difference.
Next, step you may install java and net-tools
RUN apt-get -y install apt-utils openjdk-8-jdk net-tools
Config mysql
ENV MYSQL_DATABASE=mydb
ENV MYSQL_ROOT_PASSWORD=root
Add flyway
ADD flyway /opt/flyway
Add migrations
ADD sql /opt/flyway/sql
Add config flyway
ADD config /opt/flyway/conf
Add script to start
ADD start /root/start.sh
Check start mysql
RUN netstat -ntlp
Check java version
RUN java -version
Example file: /opt/flyway/conf/flyway.conf
flyway.driver=com.mysql.jdbc.Driver
flyway.url=jdbc:mysql://localhost:3306/mydb
flyway.user=root
flyway.password=root
Example file: start.sh
#!/bin/bash
cd /opt/flyway
flyway migrate
# may change to start.sh to start product migration or development.
Flyway documentation
I mean that you in next step may use flyway as service:
For example:
docker run -it -p 3307:3306 my_docker_flyway /root/start << migration_prod.sh
docker run -it -p 3308:3306 my_docker_flayway /root/start << migration_dev.sh
etc ...

services:
# Standard Mysql Box, we have to add tricky things else logging by workbench is hard
supermonk-mysql:
image: mysql
command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
environment:
- MYSQL_ROOT_PASSWORD=P#ssw0rd
- MYSQL_ROOT_HOST=%
- MYSQL_DATABASE=test
ports:
- "3306:3306"
healthcheck:
test: ["CMD-SHELL", "nc -z 127.0.0.1 3306 || exit 1"]
interval: 1m30s
timeout: 60s
retries: 6
# Flyway is best for mysql schema migration history.
supermonk-flyway:
container_name: supermonk-flyway
image: boxfuse/flyway
command: -url=jdbc:mysql://supermonk-mysql:3306/test?verifyServerCertificate=false&useSSL=true -schemas=test -user=root -password=P#ssw0rd migrate
volumes:
- "./sql:/flyway/sql"
depends_on:
- supermonk-mysql
mkdir ./sql
vi ./sql/V1.1__Init.sql # and paste below
CREATE TABLE IF NOT EXISTS test.USER (
id VARCHAR(64),
fname VARCHAR(256),
lname VARCHAR(256),
CONSTRAINT pk PRIMARY KEY (id));
save and close
docker-compose up -d
wait for 2 minutes
docker-compose run supermonk-flyway
Ref :
https://github.com/supermonk/webapp/tree/branch-1/docker/docker-database
Thanks to docker community and mysql community
docker-compose logs -f

Related

CircleCI job creates docker MySQL 8 but nothing can connect

(See UPDATE at end of post for potentially helpful debug info.)
I have a CircleCI job that deploys MySQL 8 via - setup_remote_docker+docker-compose and then attempts to start a Java app to communicate with MySQL 8. Unfortunately, even though docker ps shows the container is up and running, any attempt to communicate with MySQL--either through the Java app or docker exec--fails, saying the container is not running (and Java throws a "Communications Link Failure" exception). It's a bit confusing because the container appears to be up, and the exact same commands work on my local machine.
Here's my CircleCI config.yml:
Build and Test:
<<: *configure_machine
steps:
- *load_repo
- ... other unrelated stuff ...
- *load_gradle_wrapper
- run:
name: Install Docker Compose
environment:
COMPOSE_VERSION: '1.29.2'
command: |
curl -L "https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o ~/docker-compose
chmod +x ~/docker-compose
sudo mv ~/docker-compose /usr/local/bin/docker-compose
- setup_remote_docker
- run:
name: Start MySQL docker
command: docker-compose up -d
- run:
name: Check Docker MySQL
command: docker ps
- run:
name: Query MySQL #test that fails
command: docker exec -it mysql8_test_mysql mysql mysql -h 127.0.0.1 --port 3306 -u root -prootpass -e "show databases;"
And here's my docker-compose.yml that is run in one of the steps:
version: "3.1"
services:
# MySQL Dev Image
mysql-migrate:
container_name: mysql8_test_mysql
image: mysql:8.0
command:
mysqld --default-authentication-plugin=mysql_native_password
--character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
--log-bin-trust-function-creators=true
environment:
MYSQL_DATABASE: test_db
MYSQL_ROOT_PASSWORD: rootpass
ports:
- "3306:3306"
volumes:
- "./docker/mysql/data:/var/lib/mysql"
- "./docker/mysql/my.cnf:/etc/mysql/conf.d/my.cnf"
- "./mysql_schema_v1.sql:/docker-entrypoint-initdb.d/mysql_schema_v1.sql"
It's a fairly simple setup and the output from CircleCI is positive until it reaches the docker exec, which I added to test the connection. Here is what the output from CircleCI says per step:
Start MySQL Docker:
#!/bin/bash -eo pipefail
docker-compose up -d
Creating network "project_default" with the default driver
Pulling mysql-migrate (mysql:8.0)...
8.0: Pulling from library/mysql
5158dd02: Pulling fs layer
f6778b18: Pulling fs layer
a6c74a04: Pulling fs layer
4028a805: Pulling fs layer
7163f0f6: Pulling fs layer
cb7f57e0: Pulling fs layer
7a431703: Pulling fs layer
5fe86aaf: Pulling fs layer
add93486: Pulling fs layer
960383f3: Pulling fs layer
80965951: Pulling fs layer
Digest: sha256:b17a66b49277a68066559416cf44a185cfee538d0e16b5624781019bc716c122 121B/121BkBBB
Status: Downloaded newer image for mysql:8.0
Creating mysql8_******_mysql ...
Creating mysql8_******_mysql ... done
So we know MySQL 8 was pulled fine (and therefore the previous step worked). Next step is to ask Docker what's running.
Check Docker MySQL:
#!/bin/bash -eo pipefail
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cb6b7941ad65 mysql:8.0 "docker-entrypoint.s…" 1 second ago Up Less than a second 0.0.0.0:3306->3306/tcp, 33060/tcp mysql8_test_mysql
CircleCI received exit code 0
Looks good so far. But now let's actually try to run a command against it via docker exec.
Query MySQL:
#!/bin/bash -eo pipefail
docker exec -it mysql8_test_mysql mysql mysql -h 127.0.0.1 --port 3306 -u root -prootpass -e "show databases;"
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:3306' (111)
Exited with code exit status 1
CircleCI received exit code 1
So now we can't connect to MySQL even though docker ps showed it up and running. I even tried adding an absurd step to wait in case MySQL needed more time:
- run:
name: Start MySQL docker
command: docker-compose up -d
- run:
name: Check Docker MySQL
command: docker ps
- run:
name: Wait Until Ready
command: sleep 120
- run:
name: Query MySQL
command: docker exec -it mysql8_test_mysql mysql mysql -h 127.0.0.1 --port 3306 -u root -prootpass -e "show databases;"
Of course adding a 2 minute wait for MySQL to spin up didn't help. Any ideas as to why this is so difficult in CircleCI?
Thanks in advance.
UPDATE 1: I can successfully start MySQL if I SSH into the job's server and run the same command myself:
docker-compose up
Then in another terminal run this:
docker exec -it mysql8_test_mysql mysql mysql -h localhost --port 3306 -u root -prootpass -e "show databases;"
+--------------------+
| Database |
+--------------------+
| information_schema |
| test_db |
| mysql |
| performance_schema |
| sys |
+--------------------+
So it is possible to start MySQL. It's just not working right when through job steps.
UPDATE 2: I moved the two minute wait between docker-compose up -d and docker ps and now it shows nothing is running. So the container must be starting then crashing and that's the reason for why it's not available moments later.
The cause of the problem was the volumes entry in my docker-compose.yml with this line:
- "./mysql_schema_v1.sql:/docker-entrypoint-initdb.d/mysql_schema_v1.sql"
The container appeared to be up when I checked immediately after docker-compose up -d but in actuality it would crash seconds later because CircleCI appears to have an issue with Docker volume, potentially related to this: https://discuss.circleci.com/t/docker-compose-doesnt-mount-volumes-with-host-files-with-circle-ci/19099.
To make it work I removed that volume entry and added run commands to copy and import the schema like so:
- run:
name: Start MySQL docker
command: docker-compose up -d
# Manually copy schema file instead of using docker-compose volumes (has issues with CircleCI)
- run:
name: Copy Schema
command: docker cp mysql_schema_v1.sql mysql8_mobile_mysql:docker-entrypoint-initdb.d/mysql_schema_v1.sql
- run:
name: Import Schema
command: docker exec mysql8_mobile_mysql /bin/sh -c 'mysql -u root -prootpass < docker-entrypoint-initdb.d/mysql_schema_v1.sql'
With this new setup I've been able to create the tables and connect to MySQL. However, there appears to be an issue running tests against MySQL causing hangups but that might be unrelated. I will follow up with more information, but at least I hope this can help someone else.

How to execute query using script in MySQL Docker image?

I am trying to give user a web interface in which , user can write a query and then i will be executing that query on my server.
I am using the following MySQL docker image with the latest tag i.e. mysql:latest
https://hub.docker.com/_/mysql/
So i am runnig the docker image using this command
docker run -it --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -v /root/ServerCode/:/usercode mysql /bin/bash
My root/ServerCode directory contains a script which i want to use for running mysql server and user's query.
My script is
#!/bin/bash
set -e
/etc/init.d/mysqld start
It gives me error
bash: /etc/init.d/mysqld: No such file or directory
I have also tried using this
service mysqld start
It is also giving error
mysqld: unrecognized service
Edit:
#!/bin/bash
set -e
exec 1> $"/usercode/logfile.txt"
exec 2> $"/usercode/errors"
# These output and error files are in mounted folder which i check after running script
/etc/init.d/mysqld start // run sql server here
#here i want to run that query and then get out of conatiner `
The entyrypoint scipt only does the initdb if mysqld is the argument; in your case it sees bash and so skips the initdb and just runs bash with its arguments.
If you are just trying to run some setup scripts once mysql is running have you looked at /docker-entrypoint-initdb.d/?
Create a docker-compose.yml
version: '2'
services:
db:
image: mysql
container_name: mysql
restart: always
volumes:
- /var/db/startuphry/mysql:/var/lib/mysql
- ./conf/my.cnf:/etc/mysql/conf.d/settings.cnf
- ./conf/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
environment:
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
- MYSQL_DATABASE=${MYSQL_DATABASE}
- MYSQL_USER=${MYSQL_USER}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
ports:
- "${MYSQL_PORT}:3306"
Create a conf folder add the my.cnf file to it
[mysqld]
local-infile=0
Create folder docker-entrypoint-initdb.d inside conf folder and all sql files inside this folder
Tree looks likes this
|____docker-compose.yml
|____conf
|___my.cnf
|___docker-entrypoint-initdb.d
|___one.sql
|___two.sql
You can put any .sh files or .sql files in there and they will be run/imported before the mysql service is available outside the container.
Try running the "/etc/init.d/mysqld start" inside the mysql docker container.
/root/server is a host machine path . Mysql has been installed in container not in the host machine. Please run the "/etc/init.d/mysqld start " not in the host machine.

Docker MySQL container unable to wait for MySQL server ready

I am attempting to create a docker container using the mysql:5 docker image. Once the MySQL server is up and running I want to create some databases, users and tables.
My Dockerfile looks like this;
FROM mysql:5
# Add db.data with correct permissions
RUN mkdir /server_data
WORKDIR /server_data
ADD --chown="root:root" ./db.data .
# Copy setup directory
COPY ./setup setup
COPY ./config /etc/mysql/conf.d
CMD ["./setup/setup.sh", "mysql", "-u", "root", "<", "./setup/schema.sql"]
My ./setup/setup.sh script looks like this;
#!/bin/bash
# wait-for-mysql.sh
set -e
shift
cmd="$#"
until mysql -uroot -c '\q'; do
>&2 echo "mysql is unavailable - sleeping"
sleep 1
done
>&2 echo "mysql is up - executing command"
exec $cmd
My docker-compose.yml looks like this;
version: "3"
services:
db:
build: ./db
volumes:
- data-db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=password
restart: always
container_name: db
volumes:
data-db:
When I run 'docker-compose up --build' I get the following output;
Building db
Step 1/7 : FROM mysql:5
---> 0d16d0a97dd1
Step 2/7 : RUN mkdir /server_data
---> Using cache
---> 087b5ded3a53
Step 3/7 : WORKDIR /server_data
---> Using cache
---> 5a32ea1b0a49
Step 4/7 : ADD --chown="root:root" ./db.data .
---> Using cache
---> 5d453c52a9f1
Step 5/7 : COPY ./setup setup
---> 9c5359818748
Step 6/7 : COPY ./config /etc/mysql/conf.d
---> b663a380813f
Step 7/7 : CMD ["./setup/setup.sh", "mysql", "-u", "root", "<", "./setup/schema.sql"]
---> Running in 4535b2620141
Removing intermediate container 4535b2620141
---> 2d2fb7e308ad
Successfully built 2d2fb7e308ad
Successfully tagged wasdbsandbox_db:latest
Recreating db ... done
Attaching to db
db | ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
db | mysql is unavailable - sleeping
db | ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
db | mysql is unavailable - sleeping
This goes on interminably until I press Ctrl + c.
If I comment out the CMD line in my Dockerfile the output from running 'docker-compose up --build' is the output of the ENTRYPOINT command that is defined in the official mysql Dockerfile.
Why is mysql never starting when I use my own CMD command?
This is supported already by the official mysql image. No need to make your own custom solution.
Look at the Docker Hub README under "Initializing a fresh instance".
You can see in the official image under the 5.7 Dockerfile (for example) that it copies in a ENTRYPOINT script. That script doesn't run at build time, but at run-time right before the CMD starts the daemon.
In that existing ENTRYPOINT script you'll see that it will process any files you put in /docker-entrypoint-initdb.d/
So in short, when you start a new container from that existing official image it will:
start the mysqld in local-only mode
creates default user, db, pw, etc.
runs any scripts you put in /docker-entrypoint-initdb.d/
stops the mysqld and hands off to the Dockerfile CMD
CMD will run the mysqld to listen on the network

Import into dockerized mariadb on initial build with script

I'm using MariaDB, but I think this could probably apply to MySQL as well.
I have a project that works off of MariaDB, and there is some initial setup for the database that needs to be done to create tables, insert initial data, etc. Based on other answers, I could normally do ADD dump.sql /docker-entrypoint-initdb.d, but I don't have a dump.sql -- instead what I have is a python script that connects to MariaDB directly and creates the tables and data.
I have a docker-compose.yml
version: '3'
services:
db:
build: ./db
ports:
- "3306:3306"
container_name: db
environment:
- MYSQL_ROOT_PASSWORD=root
web:
build: ./web
command: node /app/src/index.js
ports:
- "3000:3000"
links:
- db
"Web" is not so important right now since I just want to get db working.
The Dockerfile I've attempted for DB is:
# db/Dockerfile
FROM mariadb:10.3.2
RUN apt-get update && apt-get install -y python python-pip python-dev libmariadbclient-dev
RUN pip install requests mysql-python
ADD pricing_import.py /scripts/
RUN ["/bin/sh", "-c", "python /scripts/pricing_import.py"]
However this doesn't work for various reasons. I've gotten up to the point where pip install mysql-python doesn't compile:
_mysql.c:2005:41: error: 'MYSQL' has no member named 'reconnect'
if ( reconnect != -1 ) self->connection.reconnect = reconnect;
I think this has to do with the installation of mysql-python.
However before I go down the hole too far, I want to make sure my approach even makes sense since I don't even think the database will be started once I get to the ./pricing_import.py script and since it tries to connect to the database and runs queries, it probably won't work.
Since I can't get the python installation to work on the mariadb container anyway, I was also thinking about creating another docker-compose entry that depends on db and runs the python script on build to do the initial import during docker-compose build.
Are either of these approaches correct, or is there a better way to handle running an initialization script against MariaDB?
We use docker-compose healthcheck with combination of makefile and bash to handle running an initialization script. So your docker-compose.yml would look something like that:
version: '3'
services:
db:
build: ./db
ports:
- "3306:3306"
container_name: db
environment:
- MYSQL_ROOT_PASSWORD=root
healthcheck:
test: mysqlshow --defaults-extra-file=./database/my.cnf
interval: 5s
timeout: 60s
Where ./database/my.cnf is the config with credentials:
[client]
host = db
user = root
password = password
Then you can use this health-check.bash script to check the health:
#!/usr/bin/env bash
DATABASE_DOCKER_CONTAINER=$1
# Check on health database server before continuing
function get_service_health {
# https://gist.github.com/mixja/1ed1314525ba4a04807303dad229f2e1
docker inspect -f '{{if .State.Running}}{{ .State.Health.Status }}{{end}}' $DATABASE_DOCKER_CONTAINER
}
until [[ $(get_service_health) != starting ]];
do echo "database: ... Waiting on database Docker Instance to Start";
sleep 5;
done;
# Instance has finished starting, will be unhealthy until database finishes startup
MYSQL_HEALTH_CHECK_ATTEMPTS=12
until [[ $(get_service_health) == healthy ]]; do
echo "database: ... Waiting on database service"
sleep 5
if [[ $MYSQL_HEALTH_CHECK_ATTEMPTS == 0 ]];
then echo $DATABASE_DOCKER_CONTAINER ' failed health check (not running or unhealthy) - ' $(get_service_mysql_health)
exit 1
fi;
MYSQL_HEALTH_CHECK_ATTEMPTS=$((MYSQL_HEALTH_CHECK_ATTEMPTS-1))
done;
echo "Database is healthy"
Finally, you can use makefile to connect all things together. Something like that:
docker-up:
docker-compose up -d
db-health-check:
db/health-check.bash db
load-database:
docker run --rm --interactive --tty --network your_docker_network_name -v `pwd`:/application -w /application your_docker_db_image_name python /application/pricing_import.py
start: docker-up db-health-check load-database
Then start your app with make start.

Can't connect nodejs and mysql in same docker

I'm new in docker and i'm trying to make my nodejs express run inside it.
I'm trying to install the dependencies using shellscript and its working but in the end I can't connect to mysql.
My docker file install mysql, create an user and a database, and install nodejs too.
Then it runs npm install and try to start my app but knex says it can't connect to the mysql with the message:
Knex:Error Pool2 - Error: connect ECONNREFUSED /var/run/mysqld/mysqld.sock
Here's a gist with the code i'm using. (nodejs part is incomplete, just with the important part):
https://gist.github.com/jradesenv/527f6e59ab2e7985c38fbed3a2084c83
I hope anyone will have a good ideia on how to resolve or debbug this.
The best practice is to keep the components of a micro-service separate in their own container.
See for instance "Learn Docker by building a Microservice" from Dave Kerr.
He does declare two services:
version: '2'
services:
users-service:
build: ./users-service
ports:
- "8123:8123"
depends_on:
- db
environment:
- DATABASE_HOST=db
db:
build: ./test-database
With a dedicated Dockerfile for the database:
FROM mysql:5
ENV MYSQL_ROOT_PASSWORD 123
ENV MYSQL_DATABASE users
ENV MYSQL_USER users_service
ENV MYSQL_PASSWORD 123
ADD setup.sql /docker-entrypoint-initdb.d
Docker containers are designed to run a single command. The mysql installer expects the service it registered to automatically be started on the OS bootup, but that's not the case inside of a container.
The proper solution is to split these into two separate containers, one db container, and another nodejs/app container. Link and run the two together with a docker-compose configuration that automatically sets up the host names.
The less ideal option is supervisord which you can use to run and manage multiple processes inside of the container. You install it just like any other app, configure your db and node app as two services for supervisord to manage, and then launch supervisord as your container's run command.
Use docker-compose and create a dockerfile for your nodejs and one for mysql. Each container is responsible for doing their thing. In your compose, link them. Then point your nodejs db connection to the mysql container.