Related
I am deploying a few different docker containers, mysql being the first one. I want to run scripts as soon as database is up and proceed to building other containers. The script has been failing because it was trying to run when the entrypoint script, which sets up mysql (from this official mysql container), was still running.
sudo docker run --name mysql -e MYSQL_ROOT_PASSWORD=MY_ROOT_PASS -p 3306:3306 -d mysql
[..] wait for mysql to be ready [..]
mysql -h 127.0.0.1 -P 3306 -u root --password=MY_ROOT_PASS < MY_SQL_SCRIPT.sql
Is there a way to wait for a signal of an entrypoiny mysql setup script finishing inside the docker container? Bash sleep seems like a suboptimal solution.
EDIT: Went for a bash script like this. Not the most elegant and kinda brute force but works like a charm. Maybe someone will find that useful.
OUTPUT="Can't connect"
while [[ $OUTPUT == *"Can't connect"* ]]
do
OUTPUT=$(mysql -h $APP_IP -P :$APP_PORT -u yyy --password=xxx < ./my_script.sql 2>&1)
done
You can install mysql-client package and use mysqladmin to ping target server. Useful when working with multiple docker container. Combine with sleep and create a simple wait-loop:
while ! mysqladmin ping -h"$DB_HOST" --silent; do
sleep 1
done
This little bash loop waits for mysql to be open, shouldn't require any extra packages to be installed:
until nc -z -v -w30 $CFG_MYSQL_HOST 3306
do
echo "Waiting for database connection..."
# wait for 5 seconds before check again
sleep 5
done
This was more or less mentioned in comments to other answers, but I think it deserves its own entry.
First of all you can run your container in the following manner:
docker run --name mysql --health-cmd='mysqladmin ping --silent' -d mysql
There is also an equivalent in the Dockerfile.
With that command your docker ps and docker inspect will show you health status of your container. For mysql in particular this method has the advantage of mysqladmin being available inside the container, so you do not need to install it on the docker host.
Then you can simply loop in a bash script to wait on the status to become healthy. The following bash script is created by Dennis.
function getContainerHealth {
docker inspect --format "{{.State.Health.Status}}" $1
}
function waitContainer {
while STATUS=$(getContainerHealth $1); [ $STATUS != "healthy" ]; do
if [ $STATUS == "unhealthy" ]; then
echo "Failed!"
exit -1
fi
printf .
lf=$'\n'
sleep 1
done
printf "$lf"
}
Now you can do this in your script:
waitContainer mysql
and your script will wait until the container is up and running. The script will exit if the container becomes unhealthy, which is possible, if for example docker host is out of memory, so that the mysql cannot allocate enough of it for itself.
I've found that using the mysqladmin ping approach isn't always reliable, especially if you're bringing up a new DB. In that case, even if you're able to ping the server, you might be unable to connect if the user/privilege tables are still being initialized. Instead I do something like the following:
while ! docker exec db-container mysql --user=foo --password=bar -e "SELECT 1" >/dev/null 2>&1; do
sleep 1
done
So far I haven't encountered any problems with this method. I see that something similar was suggested by VinGarcia in a comment to one of the mysqladmin ping answers.
Some times the problem with the port is that the port could be open, but the database is not ready yet.
Other solutions require that you have installed the mysql o a mysql client in your host machine, but really you already have it inside the Docker container, so I prefer to use something like this:
Option 1:
while ! docker exec mysql mysqladmin --user=root --password=pass --host "127.0.0.1" ping --silent &> /dev/null ; do
echo "Waiting for database connection..."
sleep 2
done
Option 2 (from #VinGarcia):
while ! docker exec container_name mysql --user=root --password=pass -e "status" &> /dev/null ; do
echo "Waiting for database connection..."
sleep 2
done
One liner using curl, found on all linux distributions:
while ! curl -o - db-host:3306; do sleep 1; done
The following health-check works for all my mysql containers:
db:
image: mysql:5.7.16
healthcheck:
test: ["CMD-SHELL", 'mysql --database=$$MYSQL_DATABASE --password=$$MYSQL_ROOT_PASSWORD --execute="SELECT count(table_name) > 0 FROM information_schema.tables;" --skip-column-names -B']
interval: 30s
timeout: 10s
retries: 4
extends:
file: docker-compose-common-config.yml
service: common_service
So I am not sure if any one has posted this. It doesn't look like any one has, so... there is a command in mysqladmin that features a wait, it handles testing of the connection, then retries internally and returns a success upon completion.
sudo docker run --name mysql -e MYSQL_ROOT_PASSWORD=MY_ROOT_PASS -p 3306:3306 -d mysql
mysqladmin ping -h 127.0.0.1 -u root --password=MY_ROOT_PASS --wait=30 && mysql -h 127.0.0.1 -P 3306 -u root --password=MY_ROOT_PASS < MY_SQL_SCRIPT.sql
The important piece is mysqladmin ping -h 127.0.0.1 -u root --password=MY_ROOT_PASS --wait=30 -v with the --wait being the flag to wait until the connection is successful and the number being the amount of attempts to retry.
Ideally you would run that command from inside the docker container, but I didn't want to modify the original posters command too much.
When used in my make file for initialization
db.initialize: db.wait db.initialize
db.wait:
docker-compose exec -T db mysqladmin ping -u $(DATABASE_USERNAME) -p$(DATABASE_PASSWORD) --wait=30 --silent
db.initialize:
docker-compose exec -T db mysql -u $(DATABASE_USERNAME) -p$(DATABASE_PASSWORD) $(DATABASE_NAME) < dev/sql/base_instance.sql
I had the same issue when my Django container tried to connect the mysql container just after it started. I solved it using the vishnubob's wait-for.it.sh script. Its a shell script which waits for an IP and a host to be ready before continuing. Here is the example I use for my applicaction.
./wait-for-it.sh \
-h $(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $MYSQL_CONTAINER_NAME) \
-p 3306 \
-t 90
In that script I'm asking to the mysql container to wait maximum 90 seconds (it will run normally when ready) in the port 3306 (default mysql port) and the host asigned by docker for my MYSQL_CONTAINER_NAME. The script have more variables but for mw worked with these three.
If the docker container waiting for a mysql container is based on a python image (for instance for a Django application), you can use the code below.
Advantages are:
It's not based on wait-for-it.sh, which does wait for the IP and port of mysql to be ready, but this doesn't automatically mean also that the mysql initialization has finished.
It's not a shell script based on a mysql or mysqladmin executable that must be present in your container: since your container is based on a python image, this would require installing mysql on top of that image. With the below solution, you use the technology that is already present in the container: pure python.
Code:
import time
import pymysql
def database_not_ready_yet(error, checking_interval_seconds):
print('Database initialization has not yet finished. Retrying over {0} second(s). The encountered error was: {1}.'
.format(checking_interval_seconds,
repr(error)))
time.sleep(checking_interval_seconds)
def wait_for_database(host, port, db, user, password, checking_interval_seconds):
"""
Wait until the database is ready to handle connections.
This is necessary to ensure that the application docker container
only starts working after the MySQL database container has finished initializing.
More info: https://docs.docker.com/compose/startup-order/ and https://docs.docker.com/compose/compose-file/#depends_on .
"""
print('Waiting until the database is ready to handle connections....')
database_ready = False
while not database_ready:
db_connection = None
try:
db_connection = pymysql.connect(host=host,
port=port,
db=db,
user=user,
password=password,
charset='utf8mb4',
connect_timeout=5)
print('Database connection made.')
db_connection.ping()
print('Database ping successful.')
database_ready = True
print('The database is ready for handling incoming connections.')
except pymysql.err.OperationalError as err:
database_not_ready_yet(err, checking_interval_seconds)
except pymysql.err.MySQLError as err:
database_not_ready_yet(err, checking_interval_seconds)
except Exception as err:
database_not_ready_yet(err, checking_interval_seconds)
finally:
if db_connection is not None and db_connection.open:
db_connection.close()
Usage:
Add this code into a python file (wait-for-mysql-db.py for instance) inside your application's source code.
Write another python script (startup.py for instance) that first executes the above code, and afterwards starts up your application.
Make sure your application container's Dockerfile packs these two python scripts together with the application's source code into a Docker image.
In your docker-compose file, configure your application container with: command: ["python3", "startup.py"].
Note that this solution is made for a MySQL database. You'll need to adapt it slightly for another database.
I developed a new solution for this issue based on a new approach. All approaches I found rely on a script that tries over and over to connect to the database, or try to establish a TCP connection with the container. The full details can be found on the waitdb repository, but, my solution is to rely on the retrieved log from the container. The script waits until the log fires the message ready for connections. The script can identify if the container is starting for the first time. In this case the script waits until the initial database script is executed and the database is restarted, waiting again for a new ready for connections message. I tested this solution on MySQL 5.7 and MySQL 8.0.
The script itself (wait_db.sh):
#!/bin/bash
STRING_CONNECT="mysqld: ready for connections"
findString() {
($1 logs -f $4 $5 $6 $7 $8 $9 2>&1 | grep -m $3 "$2" &) | grep -m $3 "$2" > /dev/null
}
echo "Waiting startup..."
findString $1 "$STRING_CONNECT" 1 $2 $3 $4 $5 $6 $7
$1 logs $2 $3 $4 $5 2>&1 | grep -q "Initializing database"
if [ $? -eq 0 ] ; then
echo "Almost there..."
findString $1 "$STRING_CONNECT" 2 $2 $3 $4 $5 $6 $7
fi
echo "Server is up!"
The script can be used in Docker Compose or in Docker itself. I hope the examples bellow make the the usage clear:
Example 01: Using with Docker Compose
SERVICE_NAME="mysql" && \
docker-compose up -d $SERVICE_NAME && \
./wait_db.sh docker-compose --no-color $SERVICE_NAME
Example 02: Using with Docker
CONTAINER_NAME="wait-db-test" && \
ISO_NOW=$(date -uIs) && \
docker run --rm --name $CONTAINER_NAME \
-e MYSQL_ROOT_PASSWORD=$ROOT_PASSWORD \
-d mysql:5.7 && \
./wait_db.sh docker --since "$ISO_NOW" $CONTAINER_NAME
Example 3: A full example (the test-case)
A full example can be found on the test case of the repository. This test-case will startup a new MySQL, create a dummy database, wait until everything is started and then fires a select to check if everything goes fine. After that it'll going restart the container and wait it to be started and then fires a new select to check if it's ready for connection.
Here's how I incorporated Adams solution into my docker-compose based project:
Created a bash file titled db-ready.sh in my server container folder (the contents of which are copied in to my container - server):
#!bin/bash
until nc -z -v -w30 $MYSQL_HOST 3306
do
echo "Waiting a second until the database is receiving connections..."
# wait for a second before checking again
sleep 1
done
I can then run docker-compose run server sh ./db-ready.sh && docker-compose run server yarn run migrate to ensure that when I run my migrate task within my server container, I know the DB will be accepting connections.
I like this approach as the bash file is separate to any command I want to run. I could easily run the db-ready.sh before any other DB using task I run.
i can recommend you to use /usr/bin/mysql --user=root --password=root --execute "SHOW DATABASE;" in healthcheck script instead of mysqladmin ping. This wait for real initialization and service is ready for client connection.
Example:
docker run -d --name "test-mysql-client" -p 0.0.0.0:3306:3306 -e MYSQL_PASSWORD=password -e MYSQL_USER=user -e MYSQL_ROOT_PASSWORD=root --health-cmd="/usr/bin/mysql --user=root --password=root --execute \"SHOW DATABASE;\"" --health-interval=1s --health-retries=60 --health-timeout=10s -e MYSQL_DATABASE=db mysql:latest```
Combining flamemyst‘s answer and Nathan Arthur's comment, I believe this answer would be the most convenient one:
CONTAINER_MYSQL='' # name of the MySQL container
CONTAINER_DB_HOST='127.0.0.1'
CONTAINER_DB_PORT=3306
MYSQL_USER='' # user name if there is, normally 'root'
MYSQL_PWD='' # password you set
is_mysql_alive() {
docker exec -it ${CONTAINER_MYSQL} \
mysqladmin ping \
--user=${MYSQL_USER} \
--password=${MYSQL_PWD} \
--host=${CONTAINER_DB_HOST} \
--port=${CONTAINER_DB_PORT} \
> /dev/null
returned_value=$?
echo ${returned_value}
}
until [ "$(is_mysql_alive)" -eq 0 ]
do
sleep 2
echo "Waiting for MySQL to be ready..."
done
anything_else_to_do
Basically, it checks whether mysqladmin is alive in MySQL container, MySQL should be up if so.
Building a bit on Mihai Crăiță excellent answer above, I added in the CURL option to enable 0.9 (which is disabled by default now) and to hide the output to reduce log "noise" during startup:
server="MyServerName"
echo "Waiting for MySQL at ${server}"
while ! curl --http0.9 -o - "${server}:3306" &> /dev/null; do sleep 1; done
https://github.com/docker-library/mysql/blob/master/5.7/docker-entrypoint.sh
docker-entrypoint.sh doesn't support merging customized .sql yet.
I think you can modify docker-entrypoint.sh to merge your sql so it can be executed once mysql instance is ready.
On your ENTRYPOINT script, you have to check if you have a valid MySQL connection or not.
This solution does not require you to install a MySQL Client on the container and while running the container with php:7.0-fpm running nc was not an option, because it had to be installed as well. Also, checking if the port is open does not necessarily mean that the service is running and exposed correctly. [more of this]
So in this solution, I will show you how to run a PHP script to check if a MySQL Container is able to take connection. If you want to know why I think this is a better approach check my comment here.
File entrypoint.sh
#!/bin/bash
cat << EOF > /tmp/wait_for_mysql.php
<?php
\$connected = false;
while(!\$connected) {
try{
\$dbh = new pdo(
'mysql:host=mysql:3306;dbname=db_name', 'db_user', 'db_pass',
array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION)
);
\$connected = true;
}
catch(PDOException \$ex){
error_log("Could not connect to MySQL");
error_log(\$ex->getMessage());
error_log("Waiting for MySQL Connection.");
sleep(5);
}
}
EOF
php /tmp/wait_for_mysql.php
# Rest of entry point bootstrapping
By running this, you are essentially blocking any bootstrapping logic of your container UNTIL you have a valid MySQL Connection.
I use the following code ;
export COMPOSE_PROJECT_NAME=web;
export IS_DATA_CONTAINER_EXISTS=$(docker volume ls | grep ${COMPOSE_PROJECT_NAME}_sqldata);
docker-compose up -d;
docker-compose ps;
export NETWORK_GATEWAY=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.Gateway}}{{end}}' ${COMPOSE_PROJECT_NAME}_webserver1_con);
Having issues running a shell script that invokes mysqdump to import the database. This is for the importing of a WordPress database from production to local dev machine. It worked in the past. But something must have changed on my local MacOs Mac Mini running OSX Sierra.
This is the script:
# sync-prod.sh
read -r -p "Do you solemnly swear that you have had fewer than 2 alcoholic beverages in the last hour and that you would really like to reset your development database and pull the latest from production? [y/N] " response
if [[ $response =~ ^([yY][eE][sS]|[yY])$ ]]
then
wp #development db reset --yes
wp #production db export - > sql-dump-production.sql
wp #development db import sql-dump-production.sql
wp #development search-replace https://example.com http://example.test
else
exit 0
fi
when I run it I get:
./sync-production.sh
Do you solemnly swear that you have had fewer than 2 alcoholic beverages in the last hour and that you would really like to reset your development database and pull the latest from production? [y/N] y
Success: Database reset.
/usr/bin/env: ‘mysqldump’: No such file or directory
Success: Imported from 'sql-dump-production.sql'.
Error: The site you have requested is not installed.
When I run mysqldump
jasper#~/webdesign/example.com/site $ which mysqldump
/usr/local/bin/mysqldump
from the terminal it starts fine:
$ mysqldump
Usage: mysqldump [OPTIONS] database [tables]
OR mysqldump [OPTIONS] --databases [OPTIONS] DB1 [DB2 DB3...]
OR mysqldump [OPTIONS] --all-databases [OPTIONS]
For more options, use mysqldump --help
Read env: mysql: No such file or directory after `wp import` and tried adjusting the $PATH and now have
echo $PATH
/usr/local/sbin:/usr/local/mysql/bin:/usr/local/sbin:/usr/local/sbin:/usr/local/sbin:/usr/local/sbin:/Users/jasper/.rvm/gems/ruby-2.3.3/bin:/Users/jasper/.rvm/gems/ruby-2.3.3#global/bin:/Users/jasper/.rvm/rubies/ruby-2.3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/jasper/.composer/vendor/bin:/usr/local/bin:/Users/jasper/.rvm/bin:/Users/jasper/.composer/vendor/bin:/usr/local/bin:/usr/local/mysql/bin:/Users/jasper/.composer/vendor/bin:/usr/local/bin:/usr/local/bin/mysql:/Users/jasper/.composer/vendor/bin:/usr/local/bin:/usr/local/bin/mysql:/Users/jasper/.composer/vendor/bin:/usr/local/bin:/usr/local/bin/mysql:/Users/jasper/.composer/vendor/bin:/usr/local/bin:/usr/local/bin/mysql
In .bash_profile I have
alias ll='ls -lGaf'
export PATH="/usr/local/sbin:$PATH"
export PATH="$PATH:$HOME/.composer/vendor/bin"
export PATH=$PATH:/usr/local/bin
export PATH=$PATH:/usr/local/bin/mysql
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*
###-tns-completion-start-###
if [ -f /Users/jasper/.tnsrc ]; then
source /Users/jasper/.tnsrc
fi
###-tns-completion-end-###
export PS1="\u#\w $ "
When I run wp #production db check things do connect and are fine. I also can connect to the database over ssh using the same username and password using SequelPro..
Any ideas why I still get no such file or directory for running mysql as part of this script?
What happens when you run env mysqldump command in shell? You can install and use strace for example and run the script or just the failing command like strace -F <command>. Maybe you will reveal more details about this error.
The remote server did (no longer) have mysql-client installed. Once I did apt install mariadb-client-10 things started working again:
wp #production db export sql-dump-production.sql
Success: Exported to 'sql-dump-production.sql'.
and I could also run:
./sync-production.sh
Do you solemnly swear that you have had fewer than 2 alcoholic beverages in the last hour and that you would really like to reset your development database and pull the latest from production? [y/N] y
Success: Database reset.
Success: Imported from 'sql-dump-production.sql'.
.........
I want a bash shell script that i can run using a cron job to check if mysql on a remote server is running. If it is, then do nothing, other start the server.
The cronjob will be checking the remote server for a live (or not) mysql every minute. I can write the cron job myself, but i need help with the shell script that checks if a remote mysql is up or down. The response after a check if up or down is not important. But the check is important.
You can use below script
#!/bin/bash
USER=root
PASS=root123
mysqladmin -h remote_server_ip -u$USER -p$PASS processlist ###user should have mysql permission on remote server. Ideally you should use different user than root.
if [ $? -eq 0 ]
then
echo "do nothing"
else
ssh remote_server_ip ###remote server linux root server password should be shared with this server.
service mysqld start
fi
The script in the selected answer works great, but requires that you have the MySQL client installed on the local host. I needed something similar for a Docker container and didn't want to install the MySQL client. This is what I came up with:
# check for a connection to the database server
#
check=$(wget -O - -T 2 "http://$MYSQL_HOST:$MYSQL_PORT" 2>&1 | grep -o mariadb)
while [ -z "$check" ]; do
# wait a moment
#
sleep 5s
# check again
#
check=$(wget -O - -T 2 "http://$MYSQL_HOST:$MYSQL_PORT" 2>&1 | grep -o mariadb)
done
This is a little different, in that it will loop until a database connection can be made. I am also using MariaDB instead of the stock MySQL database. You can change this by changing the grep -o mariadb to something else - I'm not sure what MySQL returns on a successful connection, so you'll have to play with it a bit.
I am deploying a few different docker containers, mysql being the first one. I want to run scripts as soon as database is up and proceed to building other containers. The script has been failing because it was trying to run when the entrypoint script, which sets up mysql (from this official mysql container), was still running.
sudo docker run --name mysql -e MYSQL_ROOT_PASSWORD=MY_ROOT_PASS -p 3306:3306 -d mysql
[..] wait for mysql to be ready [..]
mysql -h 127.0.0.1 -P 3306 -u root --password=MY_ROOT_PASS < MY_SQL_SCRIPT.sql
Is there a way to wait for a signal of an entrypoiny mysql setup script finishing inside the docker container? Bash sleep seems like a suboptimal solution.
EDIT: Went for a bash script like this. Not the most elegant and kinda brute force but works like a charm. Maybe someone will find that useful.
OUTPUT="Can't connect"
while [[ $OUTPUT == *"Can't connect"* ]]
do
OUTPUT=$(mysql -h $APP_IP -P :$APP_PORT -u yyy --password=xxx < ./my_script.sql 2>&1)
done
You can install mysql-client package and use mysqladmin to ping target server. Useful when working with multiple docker container. Combine with sleep and create a simple wait-loop:
while ! mysqladmin ping -h"$DB_HOST" --silent; do
sleep 1
done
This little bash loop waits for mysql to be open, shouldn't require any extra packages to be installed:
until nc -z -v -w30 $CFG_MYSQL_HOST 3306
do
echo "Waiting for database connection..."
# wait for 5 seconds before check again
sleep 5
done
This was more or less mentioned in comments to other answers, but I think it deserves its own entry.
First of all you can run your container in the following manner:
docker run --name mysql --health-cmd='mysqladmin ping --silent' -d mysql
There is also an equivalent in the Dockerfile.
With that command your docker ps and docker inspect will show you health status of your container. For mysql in particular this method has the advantage of mysqladmin being available inside the container, so you do not need to install it on the docker host.
Then you can simply loop in a bash script to wait on the status to become healthy. The following bash script is created by Dennis.
function getContainerHealth {
docker inspect --format "{{.State.Health.Status}}" $1
}
function waitContainer {
while STATUS=$(getContainerHealth $1); [ $STATUS != "healthy" ]; do
if [ $STATUS == "unhealthy" ]; then
echo "Failed!"
exit -1
fi
printf .
lf=$'\n'
sleep 1
done
printf "$lf"
}
Now you can do this in your script:
waitContainer mysql
and your script will wait until the container is up and running. The script will exit if the container becomes unhealthy, which is possible, if for example docker host is out of memory, so that the mysql cannot allocate enough of it for itself.
I've found that using the mysqladmin ping approach isn't always reliable, especially if you're bringing up a new DB. In that case, even if you're able to ping the server, you might be unable to connect if the user/privilege tables are still being initialized. Instead I do something like the following:
while ! docker exec db-container mysql --user=foo --password=bar -e "SELECT 1" >/dev/null 2>&1; do
sleep 1
done
So far I haven't encountered any problems with this method. I see that something similar was suggested by VinGarcia in a comment to one of the mysqladmin ping answers.
Some times the problem with the port is that the port could be open, but the database is not ready yet.
Other solutions require that you have installed the mysql o a mysql client in your host machine, but really you already have it inside the Docker container, so I prefer to use something like this:
Option 1:
while ! docker exec mysql mysqladmin --user=root --password=pass --host "127.0.0.1" ping --silent &> /dev/null ; do
echo "Waiting for database connection..."
sleep 2
done
Option 2 (from #VinGarcia):
while ! docker exec container_name mysql --user=root --password=pass -e "status" &> /dev/null ; do
echo "Waiting for database connection..."
sleep 2
done
One liner using curl, found on all linux distributions:
while ! curl -o - db-host:3306; do sleep 1; done
The following health-check works for all my mysql containers:
db:
image: mysql:5.7.16
healthcheck:
test: ["CMD-SHELL", 'mysql --database=$$MYSQL_DATABASE --password=$$MYSQL_ROOT_PASSWORD --execute="SELECT count(table_name) > 0 FROM information_schema.tables;" --skip-column-names -B']
interval: 30s
timeout: 10s
retries: 4
extends:
file: docker-compose-common-config.yml
service: common_service
So I am not sure if any one has posted this. It doesn't look like any one has, so... there is a command in mysqladmin that features a wait, it handles testing of the connection, then retries internally and returns a success upon completion.
sudo docker run --name mysql -e MYSQL_ROOT_PASSWORD=MY_ROOT_PASS -p 3306:3306 -d mysql
mysqladmin ping -h 127.0.0.1 -u root --password=MY_ROOT_PASS --wait=30 && mysql -h 127.0.0.1 -P 3306 -u root --password=MY_ROOT_PASS < MY_SQL_SCRIPT.sql
The important piece is mysqladmin ping -h 127.0.0.1 -u root --password=MY_ROOT_PASS --wait=30 -v with the --wait being the flag to wait until the connection is successful and the number being the amount of attempts to retry.
Ideally you would run that command from inside the docker container, but I didn't want to modify the original posters command too much.
When used in my make file for initialization
db.initialize: db.wait db.initialize
db.wait:
docker-compose exec -T db mysqladmin ping -u $(DATABASE_USERNAME) -p$(DATABASE_PASSWORD) --wait=30 --silent
db.initialize:
docker-compose exec -T db mysql -u $(DATABASE_USERNAME) -p$(DATABASE_PASSWORD) $(DATABASE_NAME) < dev/sql/base_instance.sql
I had the same issue when my Django container tried to connect the mysql container just after it started. I solved it using the vishnubob's wait-for.it.sh script. Its a shell script which waits for an IP and a host to be ready before continuing. Here is the example I use for my applicaction.
./wait-for-it.sh \
-h $(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $MYSQL_CONTAINER_NAME) \
-p 3306 \
-t 90
In that script I'm asking to the mysql container to wait maximum 90 seconds (it will run normally when ready) in the port 3306 (default mysql port) and the host asigned by docker for my MYSQL_CONTAINER_NAME. The script have more variables but for mw worked with these three.
If the docker container waiting for a mysql container is based on a python image (for instance for a Django application), you can use the code below.
Advantages are:
It's not based on wait-for-it.sh, which does wait for the IP and port of mysql to be ready, but this doesn't automatically mean also that the mysql initialization has finished.
It's not a shell script based on a mysql or mysqladmin executable that must be present in your container: since your container is based on a python image, this would require installing mysql on top of that image. With the below solution, you use the technology that is already present in the container: pure python.
Code:
import time
import pymysql
def database_not_ready_yet(error, checking_interval_seconds):
print('Database initialization has not yet finished. Retrying over {0} second(s). The encountered error was: {1}.'
.format(checking_interval_seconds,
repr(error)))
time.sleep(checking_interval_seconds)
def wait_for_database(host, port, db, user, password, checking_interval_seconds):
"""
Wait until the database is ready to handle connections.
This is necessary to ensure that the application docker container
only starts working after the MySQL database container has finished initializing.
More info: https://docs.docker.com/compose/startup-order/ and https://docs.docker.com/compose/compose-file/#depends_on .
"""
print('Waiting until the database is ready to handle connections....')
database_ready = False
while not database_ready:
db_connection = None
try:
db_connection = pymysql.connect(host=host,
port=port,
db=db,
user=user,
password=password,
charset='utf8mb4',
connect_timeout=5)
print('Database connection made.')
db_connection.ping()
print('Database ping successful.')
database_ready = True
print('The database is ready for handling incoming connections.')
except pymysql.err.OperationalError as err:
database_not_ready_yet(err, checking_interval_seconds)
except pymysql.err.MySQLError as err:
database_not_ready_yet(err, checking_interval_seconds)
except Exception as err:
database_not_ready_yet(err, checking_interval_seconds)
finally:
if db_connection is not None and db_connection.open:
db_connection.close()
Usage:
Add this code into a python file (wait-for-mysql-db.py for instance) inside your application's source code.
Write another python script (startup.py for instance) that first executes the above code, and afterwards starts up your application.
Make sure your application container's Dockerfile packs these two python scripts together with the application's source code into a Docker image.
In your docker-compose file, configure your application container with: command: ["python3", "startup.py"].
Note that this solution is made for a MySQL database. You'll need to adapt it slightly for another database.
I developed a new solution for this issue based on a new approach. All approaches I found rely on a script that tries over and over to connect to the database, or try to establish a TCP connection with the container. The full details can be found on the waitdb repository, but, my solution is to rely on the retrieved log from the container. The script waits until the log fires the message ready for connections. The script can identify if the container is starting for the first time. In this case the script waits until the initial database script is executed and the database is restarted, waiting again for a new ready for connections message. I tested this solution on MySQL 5.7 and MySQL 8.0.
The script itself (wait_db.sh):
#!/bin/bash
STRING_CONNECT="mysqld: ready for connections"
findString() {
($1 logs -f $4 $5 $6 $7 $8 $9 2>&1 | grep -m $3 "$2" &) | grep -m $3 "$2" > /dev/null
}
echo "Waiting startup..."
findString $1 "$STRING_CONNECT" 1 $2 $3 $4 $5 $6 $7
$1 logs $2 $3 $4 $5 2>&1 | grep -q "Initializing database"
if [ $? -eq 0 ] ; then
echo "Almost there..."
findString $1 "$STRING_CONNECT" 2 $2 $3 $4 $5 $6 $7
fi
echo "Server is up!"
The script can be used in Docker Compose or in Docker itself. I hope the examples bellow make the the usage clear:
Example 01: Using with Docker Compose
SERVICE_NAME="mysql" && \
docker-compose up -d $SERVICE_NAME && \
./wait_db.sh docker-compose --no-color $SERVICE_NAME
Example 02: Using with Docker
CONTAINER_NAME="wait-db-test" && \
ISO_NOW=$(date -uIs) && \
docker run --rm --name $CONTAINER_NAME \
-e MYSQL_ROOT_PASSWORD=$ROOT_PASSWORD \
-d mysql:5.7 && \
./wait_db.sh docker --since "$ISO_NOW" $CONTAINER_NAME
Example 3: A full example (the test-case)
A full example can be found on the test case of the repository. This test-case will startup a new MySQL, create a dummy database, wait until everything is started and then fires a select to check if everything goes fine. After that it'll going restart the container and wait it to be started and then fires a new select to check if it's ready for connection.
Here's how I incorporated Adams solution into my docker-compose based project:
Created a bash file titled db-ready.sh in my server container folder (the contents of which are copied in to my container - server):
#!bin/bash
until nc -z -v -w30 $MYSQL_HOST 3306
do
echo "Waiting a second until the database is receiving connections..."
# wait for a second before checking again
sleep 1
done
I can then run docker-compose run server sh ./db-ready.sh && docker-compose run server yarn run migrate to ensure that when I run my migrate task within my server container, I know the DB will be accepting connections.
I like this approach as the bash file is separate to any command I want to run. I could easily run the db-ready.sh before any other DB using task I run.
i can recommend you to use /usr/bin/mysql --user=root --password=root --execute "SHOW DATABASE;" in healthcheck script instead of mysqladmin ping. This wait for real initialization and service is ready for client connection.
Example:
docker run -d --name "test-mysql-client" -p 0.0.0.0:3306:3306 -e MYSQL_PASSWORD=password -e MYSQL_USER=user -e MYSQL_ROOT_PASSWORD=root --health-cmd="/usr/bin/mysql --user=root --password=root --execute \"SHOW DATABASE;\"" --health-interval=1s --health-retries=60 --health-timeout=10s -e MYSQL_DATABASE=db mysql:latest```
Combining flamemyst‘s answer and Nathan Arthur's comment, I believe this answer would be the most convenient one:
CONTAINER_MYSQL='' # name of the MySQL container
CONTAINER_DB_HOST='127.0.0.1'
CONTAINER_DB_PORT=3306
MYSQL_USER='' # user name if there is, normally 'root'
MYSQL_PWD='' # password you set
is_mysql_alive() {
docker exec -it ${CONTAINER_MYSQL} \
mysqladmin ping \
--user=${MYSQL_USER} \
--password=${MYSQL_PWD} \
--host=${CONTAINER_DB_HOST} \
--port=${CONTAINER_DB_PORT} \
> /dev/null
returned_value=$?
echo ${returned_value}
}
until [ "$(is_mysql_alive)" -eq 0 ]
do
sleep 2
echo "Waiting for MySQL to be ready..."
done
anything_else_to_do
Basically, it checks whether mysqladmin is alive in MySQL container, MySQL should be up if so.
Building a bit on Mihai Crăiță excellent answer above, I added in the CURL option to enable 0.9 (which is disabled by default now) and to hide the output to reduce log "noise" during startup:
server="MyServerName"
echo "Waiting for MySQL at ${server}"
while ! curl --http0.9 -o - "${server}:3306" &> /dev/null; do sleep 1; done
https://github.com/docker-library/mysql/blob/master/5.7/docker-entrypoint.sh
docker-entrypoint.sh doesn't support merging customized .sql yet.
I think you can modify docker-entrypoint.sh to merge your sql so it can be executed once mysql instance is ready.
On your ENTRYPOINT script, you have to check if you have a valid MySQL connection or not.
This solution does not require you to install a MySQL Client on the container and while running the container with php:7.0-fpm running nc was not an option, because it had to be installed as well. Also, checking if the port is open does not necessarily mean that the service is running and exposed correctly. [more of this]
So in this solution, I will show you how to run a PHP script to check if a MySQL Container is able to take connection. If you want to know why I think this is a better approach check my comment here.
File entrypoint.sh
#!/bin/bash
cat << EOF > /tmp/wait_for_mysql.php
<?php
\$connected = false;
while(!\$connected) {
try{
\$dbh = new pdo(
'mysql:host=mysql:3306;dbname=db_name', 'db_user', 'db_pass',
array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION)
);
\$connected = true;
}
catch(PDOException \$ex){
error_log("Could not connect to MySQL");
error_log(\$ex->getMessage());
error_log("Waiting for MySQL Connection.");
sleep(5);
}
}
EOF
php /tmp/wait_for_mysql.php
# Rest of entry point bootstrapping
By running this, you are essentially blocking any bootstrapping logic of your container UNTIL you have a valid MySQL Connection.
I use the following code ;
export COMPOSE_PROJECT_NAME=web;
export IS_DATA_CONTAINER_EXISTS=$(docker volume ls | grep ${COMPOSE_PROJECT_NAME}_sqldata);
docker-compose up -d;
docker-compose ps;
export NETWORK_GATEWAY=$(docker inspect --format='{{range .NetworkSettings.Networks}}{{.Gateway}}{{end}}' ${COMPOSE_PROJECT_NAME}_webserver1_con);
I need a script that automatically makes a backup of a MySql Database. I know there are a lot of posts and scripts out there on this topic already but here is where mine differs.
The script needs to run on the machine hosting the MySql database (It is a linux machine).
The backups must be saved onto the same server that the database is on.
A backup needs to be made every 30 minutes.
When a backup is older than a week it is deleted unless it is the very first backup created that week. i.e out of these backups backup_1_12_2010_0-00_Mon.db, backup_1_12_2010_0-30_Mon.db, backup_1_12_2010_1-00_Mon.db ... backup_7_12_2010_23-30_Sun.db etc only backup_1_12_2010_0-00_Mon.db is kept.
Anyone have anything similar or any ideas where to start?
Answer: A cron
Description:
Try creating a file something.sh with this:
#!/bin/sh
mysqldump -u root -p pwd --opt db1.sql > /respaldosql/db1.sql
mysqldump -u root -p pwd --opt db2.sql > /respaldosql/db2.sql
cd /home/youuser/backupsql/
tar -zcvf backupsql_$(date +%d%m%y).tgz *.sql
find -name '*.tgz' -type f -mtime +2 -exec rm -f {} \;
Give the adequate permission to the file
chmod 700 mysqlrespaldo.sh
or
sudo chmod 700 something.sh
and then create a cron with
crontab -e
setting it like
**0 1 * * *** /home/youruser/coolscripts/something.sh
Remember that the numbers or '*' characters have this structure:
Minutes (range 0-59)
Hours (0-23)
Day of month (1-31)
Month (1-12)
Day of the week (0-6 being 0=Domingo)
Absolute path to script or program to run
You can also use the helper folder available in newer versions of linux distros, where you find /etc/cron.daily, /etc/cron.hourly, /etc/cron.weekly, etc. In this case, you can create a symlink to your script into the chosen folder and OS will take charge of running it with the promised recurrence (from a powerful comment by #Nick).
Create a shell script like the one below:
#!/bin/bash
mysqldump -u username -p'password' dbname > /my_dir/db_$(date+%m-%d-%Y_%H-%M-%S).sql
find /mydir -mtime +10 -type f -delete
Replace username, password and your backup directory(my_dir). Save it in a directory(shell_dir) as filename.sh
Schedule it to run everyday using crontab -e like:
30 8 * * * /shell_dir/filename.sh
This will run everyday at 8:30 AM and backup the database. It also deletes the backup which is older than 10 days. If you don't wanna do that just delete the last line from the script.
Doing pretty much the same like many people.
The script needs to run on the machine hosting the MySql database (It is a linux machine).
=> Create a local bash or perl script (or whatever) "myscript" on this machine "A"
The backups must be saved onto the same server that the database is on.
=> in the script "myscript", you can just use mysqldump. From the local backup, you may create a tarball that you send via scp to your remote machine. Finally you can put your backup script into the crontab (crontab -e).
Some hints and functions to get you started as I won't post my entire script, it does not fully do the trick but not far away :
#!/bin/sh
...
MYSQLDUMP="$(which mysqldump)"
FILE="$LOCAL_TARBALLS/$TARBALL/mysqldump_$db-$SNAPSHOT_DATE.sql"
$MYSQLDUMP -u $MUSER -h $MHOST -p$MPASS $db > $FILE && $GZIP $GZ_COMPRESSION_LEVEL $FILE
function create_tarball()
{
local tarball_dir=$1
tar -zpcvf $tarball_dir"_"$SNAPSHOT_DATE".tar.gz" $tarball_dir >/dev/null
return $?
}
function send_tarball()
{
local PROTOCOLE_="2"
local IPV_="4"
local PRESERVE_="p"
local COMPRESSED_="C"
local PORT="-P $DESTINATION_PORT"
local EXECMODE="B"
local SRC=$1
local DESTINATION_DIR=$2
local DESTINATION_HOST=$DESTINATION_USER"#"$DESTINATION_MACHINE":"$DESTINATION_DIR
local COMMAND="scp -$PROTOCOLE_$IPV_$PRESERVE_$COMPRESSED_$EXECMODE $PORT $SRC $DESTINATION_HOST &"
echo "remote copy command: "$COMMAND
[[ $REMOTE_COPY_ACTIVATED = "Yes" ]] && eval $COMMAND
}
Then to delete files older than "date", you can look at man find and focus on the mtime and newer options.
Edit: as said earlier, there is no particular interest in doing a local backup except a temproray file to be able send a tarball easily and delete it when sent.
You can do most of this with a one-line cronjob set to run every 30 minutes:
mysqldump -u<user> -p<pass> <database> > /path/to/dumps/db.$(date +%a.%H:%M).dump
This will create a database dump every 30 minutes, and every week it'll start overwriting the previous week's dumps.
Then have another cronjob that runs once a week that copies the most recent dump to a separate location where you're keeping snapshots.
After a brief reading the question and the good answers i would add few more points. Some of them are mentioned already.
The backup process can involve next steps:
Create a backup
Compress the backup file
Encrypt the compressed backup
Send the backup to a cloud (DropBox, OneDrive, GoogleDrive, AmazonS3,...)
Get a notification about results
Setup a schedule to run the backup process periodically
Delete the old backup files
To compound a script to cover all the backup steps you need an effort and knowledge.
I would like to share a link to an article (i'm one of the writers) which describes the most used ways to backup MySQL databases with some details:
Bash script
# Backup storage directory
backup_folder=/var/backups
# Notification email address
recipient_email=<username#mail.com>
# MySQL user
user=<user_name>
# MySQL password
password=<password>
# Number of days to store the backup
keep_day=30
sqlfile=$backup_folder/all-database-$(date +%d-%m-%Y_%H-%M-%S).sql
zipfile=$backup_folder/all-database-$(date +%d-%m-%Y_%H-%M-%S).zip
# Create a backup
sudo mysqldump -u $user -p$password --all-databases > $sqlfile
if [ $? == 0 ]; then
echo 'Sql dump created'
else
echo 'mysqldump return non-zero code' | mailx -s 'No backup was created!' $recipient_email
exit
fi
# Compress backup
zip $zipfile $sqlfile
if [ $? == 0 ]; then
echo 'The backup was successfully compressed'
else
echo 'Error compressing backup' | mailx -s 'Backup was not created!' $recipient_email
exit
fi
rm $sqlfile
echo $zipfile | mailx -s 'Backup was successfully created' $recipient_email
# Delete old backups
find $backupfolder -mtime +$keep_day -delete
Automysqlbackup
sudo apt-get install automysqlbackup
wget https://github.com/sixhop/AutoMySQLBackup/archive/master.zip
mkdir /opt/automysqlbackup
mv AutoMySQLBackup-master.zip
cd /opt/automysqlbackup
tar -zxvf AutoMySQLBackup-master.zip
./install.sh
sudo nano /etc/automysqlbackup/automysqlbackup.conf
CONFIG_configfile="/etc/automysqlbackup/automysqlbackup.conf"
CONFIG_backup_dir='/var/backup/db'
CONFIG_mysql_dump_username='root'
CONFIG_mysql_dump_password='my_password'
CONFIG_mysql_dump_host='localhost'
CONFIG_db_names=('my_db')
CONFIG_db_exclude=('information_schema')
CONFIG_mail_address='mail#google.com'
CONFIG_rotation_daily=6
CONFIG_rotation_weekly=35
CONFIG_rotation_monthly=150
automysqlbackup /etc/automysqlbackup/automysqlbackup.conf
Third party tools
Hope it would be helpful!
My preference is for AutoMySQLBackup which comes with Debian. It's really easy and creates daily backups, which can be configured. As well, it stores on weekly and then one monthly backup as well.
I have had this running for a while and it's super easy to configure and use!
You might consider this Open Source tool, matiri, https://github.com/AAFC-MBB/matiri which is a concurrent mysql backup script with metadata in Sqlite3. Features (more than you were asking for...):
Multi-Server: Multiple MySQL servers are supported whether they are co-located on the same or separate physical servers.
Parallel: Each database on the server to be backed up is done separately, in parallel (concurrency settable: default: 3)
Compressed: Each database backup compressed
Checksummed: SHA256 of each compressed backup file stored and the archive of all files
Archived: All database backups tar'ed together into single file
Recorded: Backup information stored in Sqlite3 database
Full disclosure: original matiri author.