Import multiple database schemas in mysql in docker container - mysql

want to upload multiple database schema using backup.sql. Then when try to migrate showing (1044, "Access denied for user 'pranay'#'%' to database 'core'")
I have added snapshot of my files for reference
***docker-compose.yml***
version: '3'
services:
db:
image: mysql:5.7
container_name: mirror_core
volumes:
- ./mirror/core.sql:/docker-entrypoint-initdb.d/core.sql:rw
- ./mysql:/var/lib/mysql:rw
expose:
- "3306"
restart: always
environment:
- MYSQL_ROOT_PASSWORD=mobigo#123
- MYSQL_USER=pranay
- MYSQL_PASSWORD=mobigo#123
web:
build: .
container_name: mirrorweb
command: bash -c "python manage.py collectstatic --no-input && gunicorn mirror.wsgi -b 0.0.0.0:8000"
links:
- db
volumes:
- ./mirror:/mirror
expose:
- "8000"
depends_on:
- db
core.sql
CREATE DATABASE `core` ;
CREATE DATABASE `murad` ;
CREATE DATABASE `mysqltest` ;
settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'core',
'USER':'pranay',
'PASSWORD':'mobigo#123',
'HOST':'db',
'PORT':'',
}
}
steps are as follows : docker-compose build >> docker-compose up >> docker-compose exec web bash >> python manage.py migrate (within docker container)
on migrate getting error as (1044, "Access denied for user 'pranay'#'%' to database 'core'")

The problem is the # in the password. You need to escape it in you docker-compose.yml Python uses password with #123 while compose treats it differently and therefore is not the correct password you've set up for mysql.
Check the env variable in the container to get what compose really set it as password.
For reference see:
https://symfony.com/doc/current/components/yaml/yaml_format.html:
Strings containing any of the following characters must be quoted. Although you can use double quotes, for these characters it is more convenient to use single quotes, which avoids having to escape any backslash :
:, {, }, [, ], ,, &, *, #, ?, |, -, <, >, =, !, %, #, `
http://yaml.org/spec/1.2/spec.html#id2772075:
The “#” (#x40, at) and “`” (#x60, grave accent) are reserved for future use.
Maybe also possible, but less likely:
This happens because the moment when service web executes python command, the mysqldb in db service is not yet set up.
See mysql docker readme (https://hub.docker.com/_/mysql/):
No connections until MySQL init completes
If there is no database initialized when the container starts, then a
default database will be created. While this is the expected behavior,
this means that it will not accept incoming connections until such
initialization completes. This may cause issues when using automation
tools, such as docker-compose, which start several containers
simultaneously.
Try starting up db service (docker-compose up db) and give it several seconds and then try to run the web service.
The depends_on directive only waits for the container to start - docker has no knowledge when the service inside container will be 'ready' - the developer needs to implement this on his own. Usually you just set the web container to start over and over again until it finally succeeds (the db will be ready).
Also, however less recommended, is just to give a sleep 10 before you execute your migration script.

Related

docker-compose cannot wait for mysql database

I am having real problems trying to get a docker-compose script to initiate a mysql database and a Django project, but get the Django project to wait until the mysql database is ready.
I have two files, a Dockerfile and a docker-compose.yml, which I have copied below.
When I run the docker-compose.yml, and check the logs of the web container, it says that it cannot connect to the database mydb. However the second time that I run it (without clearing the containers and images) it connects properly and the Django app works.
I have spent a whole day trying a number of things such as scripts, health checks etc, but I cannot get it to work.
Dockerfile
FROM python:3.6
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY ./ /code/
RUN pip install -r requirements.txt
RUN python manage.py collectstatic --noinput
docker-compose.yml
version: '3'
services:
mydb:
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_USER=django
- MYSQL_PASSWORD=secret
- MYSQL_DATABASE=dbMarksWebsite
image: mysql:5.7
ports:
# Map default mysql port 3306 to 3308 on outside so that I can connect
# to mysql using workbench localhost with port 3308
- "3308:3306"
web:
environment:
- DJANGO_DEBUG=1
- DOCKER_PASSWORD=secret
- DOCKER_USER=django
- DOCKER_DB=dbMarksWebsite
- DOCKER_HOST=mydb
- DOCKER_PORT=3306
build: .
command: >
sh -c "sleep 10 &&
python manage.py migrate &&
python manage.py loaddata myprojects_testdata.json &&
python manage.py runserver 0.0.0.0:8080"
ports:
- "8080:8080"
depends_on:
- mydb
First run (with no existing images or containers):
...
File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in __init__
super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2002, "Can't connect to MySQL server on 'mydb' (115)")
Second run:
System check identified no issues (0 silenced).
March 27, 2020 - 16:44:57
Django version 2.2.11, using settings 'ebdjango.settings'
Starting development server at http://0.0.0.0:8080/
Quit the server with CONTROL-C.
I solved it using the following function in my entrypoint.sh:
function wait_for_db()
{
while ! ./manage.py sqlflush > /dev/null 2>&1 ;do
echo "Waiting for the db to be ready."
sleep 1
done
}
For anybody who is interested, I found a solution to this:
1 - I wrote a python script to connect to the database every second,
but with a timeout. I set this timeout to be quite high at 60
seconds, but this seems to work on my computer.
2 - I added the command to wait into my compose file.
It should mean that I can bring up a set of test containers for my website, where I can specify the exact version of Python and MySQL used.
The relevant files are listed below:
Dockerfile:
FROM python:3.6
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY ./ /code/
RUN pip install -r requirements.txt
RUN python manage.py collectstatic --noinput
docker-compose.yml
version: '3'
services:
mydb:
container_name: mydb
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_USER=django
- MYSQL_PASSWORD=secret
- MYSQL_DATABASE=dbMarksWebsite
image: mysql:5.7
ports:
# Map default mysql port 3306 to 3308 on outside so that I can connect
# to mysql using workbench localhost with port 3308
- "3308:3306"
web:
container_name: web
environment:
- DJANGO_DEBUG=1
- DOCKER_PASSWORD=secret
- DOCKER_USER=django
- DOCKER_DB=dbMarksWebsite
- DOCKER_HOST=mydb
- DOCKER_PORT=3306
build: .
command: >
sh -c "python ./bin/wait-for.py mydb 3306 django secret dbMarksWebsite 60 &&
python manage.py migrate &&
python manage.py loaddata myprojects_testdata.json &&
python manage.py runserver 0.0.0.0:8080"
ports:
- "8080:8080"
depends_on:
- mydb
wait-for.py
'''
I don't like adding this in here, but I cannot get the typical wait-for scripts
to work with MySQL database in docker, so I hve written a python script that
either times out after ? seconds or successfully connects to the database
The input arguments for the script need to be:
HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT
'''
import sys, os
import time
import pymysql
def readCommandLineArgument():
'''
Validate the number of command line input arguments and return the
input filename
'''
# Get arguments
if len(sys.argv)!=7:
raise ValueError("You must pass in 6 arguments, HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT")
# return the arguments as a tuple
return (sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5], sys.argv[6])
def connectToDB(HOST, PORT, USERNAME, PASSWORD, DATABASE):
'''
for now, just try to connect to the database.
'''
con = pymysql.connect(host=HOST, port=PORT, user=USERNAME, password=PASSWORD, database=DATABASE)
with con:
cur = con.cursor()
cur.execute("SELECT VERSION()")
def runDelay():
'''
I don't like passing passwords in, but this is only used for a test docker
delay script
'''
# Get the database connection characteristics.
(HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT) = readCommandLineArgument()
# Ensure timeout is an integer greater than zero, otherwise use 15 secs a default
try:
TIMEOUT = int(TIMEOUT)
if TIMEOUT <= 0:
raise("Timeout needs to be > 0")
except:
TIMEOUT = 60
# Ensure port is an integer greater than zero, otherwise use 3306 as default
try:
PORT = int(PORT)
if PORT <= 0:
raise("Port needs to be > 0")
except:
PORT = 3306
# Try to connect to the database TIMEOUT times
for i in range(0, TIMEOUT):
try:
# Try to connect to db
connectToDB(HOST, PORT, USERNAME, PASSWORD, DATABASE)
# If an error hasn't been raised, then exit
return True
except Exception as Ex:
strErr=Ex.args[0]
print(Ex.args)
# Sleep for 1 second
time.sleep(1)
# If I get here, assume a timeout has occurred
raise("Timeout")
if __name__ == "__main__":
runDelay()
For testing/development purposes, you could use a version of the MySQL image that has health checks (I believe there's a healthcheck/mysql image), or configure your own (see example here: Docker-compose check if mysql connection is ready).
For production use, you don't want to upgrade the database schema on startup, nor do you want to assume the database is up. Upgrading schema automatically encourages you to not think about what happens when you deploy a bug and need to rollback, and parallel schema upgrades won't work. Longer version: https://pythonspeed.com/articles/schema-migrations-server-startup/
Another option is to use a script to control the startup order, and wrap the web service's command.
In the docker-compose's documentation "wait-for-it" is one of the recommended tools, but other exists.

Switch from docker mysql container to host's mysql without using network-mode="host"

My company has a docker-container based website (Ubuntu containers), deployed via GitHub => CircleCI => AWS. (This was set up by a consultant, who we are not currently working with. I'm trying to make sense out of everything on my own.)
I've copied the source files to my Windows 10 development PC.
Locally, the website is running successfully, until it tries to access MySQL data.
I've installed MySQL (8) locally, with default settings (port 3306) and location (C:\ProgramData\MySQL\MySQL Server 8.0\Data), and imported the database. I've created root user. Running test queries as root from MySQL Workbench works.
I've read Q&A's such as How to connect locally hosted MySQL database with the docker container.
I've successfully bridged to host's IP (on EthernetV; Docker running in Hyper-V) 172.26.92.81 for other purposes, specifically to have XDebug in my website container talk to phpstorm on my Windows 10 host. (I could use host.docker.internal for IP; but I'm making everything as "concrete" as possible, until everything works.)
I just don't understand what I have to change where, for redirecting MySQLi queries (in php as create web pages) to the (development pc) host. (Assume I'm clueless about both MySQL and Docker and Apache, until proven otherwise.)
Relevant parts of (original) docker-compose.yml:
mywebsite:
build:
context: ./mywebsite/
dockerfile: Dockerfile
...
depends_on:
- mysql
links:
- mysql
environment:
MYSQL_HOST: mysql
MYSQL_USER: root
MYSQL_PASS: xxx
MYSQL_SCHEMA: xxx
MYSQL_PORT: 3306
MYSQL_CHARSET: utf8
mysql:
image: mysql:5.7
ports:
- 3305:3306
command: mysqld --sql_mode=""
environment:
MYSQL_DATABASE: xxx
MYSQL_ROOT_PASSWORD: xxx
volumes:
- data:/var/lib/mysql
NOTE: Not going for "best practices" above; I need to see a simple approach working. Security comes later.
I'd prefer a solution that doesn't involve "network_mode: "host" -- solving it that way would avoid details that I need to understand. [The linked Q&A shows that host mode is the simplest solution, but that is "too simple" - obscures some important considerations.]
Host has MySQL on its port 3306. The mysql container shown above: is it using its own copy of MySQL? Or is that already attempting to connect to host's MySQL? And why does it have the port mapping 3305:3306? (I can't change that to 3306:3306; if do so, it is unable to assign the port number. I assume that is because host's MySQL already uses 3306, and that port line is exposing the mysql container's MySQL?)
The volume mapping data:... is where I should move the data to, if I want to use the mysql container as is? That's probably what I will do for now - which may make this SO question moot - but I'd still like to know how to do what I ask.
I'm assuming it is its own database, because it has its own version number (5.7).
From php inside mywebsite:
$connection = new \mysqli(
$_ENV["MYSQL_HOST"],
$_ENV["MYSQL_USER"],
$_ENV["MYSQL_PASS"],
'INFORMATION_SCHEMA',
$_ENV["MYSQL_PORT"]
);
$result = $connection->query("SELECT VERSION();");
# breakpoint: $result > one row > ['VERSION()'] = 5.7.25
which is the version number of that mysql image, not the host's mysql.
similarly, list of databases doesn't include some databases I see on the host mysql from Workbench.
What changes do I need to make in docker-compose.yml?
Do I also need to make changes in mywebsite's Dockerfile?
mywebsite is based on apache2 + php 7.3. Dockerfile:
FROM php:7.3.3-apache-stretch
...
RUN apt-get update && ... docker-php-ext-install ... mysqli \
&& docker-php-ext-enable mysqli ...
...
COPY /php.ini /usr/local/etc/php/
COPY /src/ /var/www/html/
First of all, the declaration:
links:
- mysql
in the mywebiste service is not needed and actually stays in the way of you solving your issue. You should remove it.
After doing that, you can try 2 things (both of them worked for me):
change MYSQL_HOST to point to your computer's IP (not 'localhost', not '127.0.0.1' but your local network IP address) and start only the mywebsite service.
in version 3.0 or higher of docker-compose file you can add extra hosts:
mywebsite:
...
extra_hosts:
- "mysql:<your ip>"
I hope this helps

Docker mysql - run update command/script on image startup

I wish to build a docker image with an initialised database.
This initial data will contain a default client ref value 'XXX'.
Here's my Dockerfile:
FROM mysql/mysql-server:5.7
COPY data.sql /docker-entrypoint-initdb.d/
When starting up the container of this image, I need to replace the ref value with the user's particular value, 'ABCD', which will come from an environment variable set by a docker compose file.
So the update query to run is something like:
update client set ref='ABCD' where ref='XXX'
How do I get the Dockerfile to do this? I don't think it can be a RUN command as I don't want the update to be part of the image build, but part of the startup of the image (it's fine if it runs this update on every startup).
I have all the usual mysql env varibles set (MYSQL_ROOT_PASSWORD/MYSQL_ROOT_HOST/MYSQL_DATABASE/MYSQL_USER/MYSQL_PASSWORD) and hope to refer to another env var for the desired ref to be setting. Keen to see this could be done as raw commands as well as a script.
You're right,
Everything that you want to be persistent should be in the Dockerfile or in build: section of your docker-compose file.
As you want to have the update only in docker run or docker-compose up phase, you can use CMD or ENTRYPOINT to execute it.
There are several ways, but let me recommend the following one:
docker-compose.yml
version: '3'
services:
your-service:
env_file: ./your-mysql-env-file.env
image: mysql/mysql-server:5.7
volumes:
- ./data.sql:/docker-entrypoint-initdb.d/data.sql
- ./your-init-sql-commands.sql:/docker-entrypoint-initdb.d/your-init-sql-commands.sql
container_name: your-container-name
command:
- /bin/bash
- -c
- |
/etc/init.d/mysqld start
mysql -u ${MYSQL_USER} -p ${MYSQL_PASSWORD} -h ${MYSQL_ROOT_HOST} ${MYSQL_DATABASE} < your-init-sql-commands.sql
[other commands if needed...]
Note that your update client set ref='ABCD' where ref='XXX' should be defined inside your-init-sql-commands.sql. So, if you want to change updates, you won't need to rebuild image.
your-mysql-env-file.env just define inside it your MYSQL env variables if you need them inside.
Note that I've changed your dockerfile by image:section and COPY by volumes: in docker-compose.yml, but all this steps can be also done with Dockerfile and docker run.
If you don't need a multiple command, just change all lines by a single one.
I hope this can be helpful for you.

Import into dockerized mariadb on initial build with script

I'm using MariaDB, but I think this could probably apply to MySQL as well.
I have a project that works off of MariaDB, and there is some initial setup for the database that needs to be done to create tables, insert initial data, etc. Based on other answers, I could normally do ADD dump.sql /docker-entrypoint-initdb.d, but I don't have a dump.sql -- instead what I have is a python script that connects to MariaDB directly and creates the tables and data.
I have a docker-compose.yml
version: '3'
services:
db:
build: ./db
ports:
- "3306:3306"
container_name: db
environment:
- MYSQL_ROOT_PASSWORD=root
web:
build: ./web
command: node /app/src/index.js
ports:
- "3000:3000"
links:
- db
"Web" is not so important right now since I just want to get db working.
The Dockerfile I've attempted for DB is:
# db/Dockerfile
FROM mariadb:10.3.2
RUN apt-get update && apt-get install -y python python-pip python-dev libmariadbclient-dev
RUN pip install requests mysql-python
ADD pricing_import.py /scripts/
RUN ["/bin/sh", "-c", "python /scripts/pricing_import.py"]
However this doesn't work for various reasons. I've gotten up to the point where pip install mysql-python doesn't compile:
_mysql.c:2005:41: error: 'MYSQL' has no member named 'reconnect'
if ( reconnect != -1 ) self->connection.reconnect = reconnect;
I think this has to do with the installation of mysql-python.
However before I go down the hole too far, I want to make sure my approach even makes sense since I don't even think the database will be started once I get to the ./pricing_import.py script and since it tries to connect to the database and runs queries, it probably won't work.
Since I can't get the python installation to work on the mariadb container anyway, I was also thinking about creating another docker-compose entry that depends on db and runs the python script on build to do the initial import during docker-compose build.
Are either of these approaches correct, or is there a better way to handle running an initialization script against MariaDB?
We use docker-compose healthcheck with combination of makefile and bash to handle running an initialization script. So your docker-compose.yml would look something like that:
version: '3'
services:
db:
build: ./db
ports:
- "3306:3306"
container_name: db
environment:
- MYSQL_ROOT_PASSWORD=root
healthcheck:
test: mysqlshow --defaults-extra-file=./database/my.cnf
interval: 5s
timeout: 60s
Where ./database/my.cnf is the config with credentials:
[client]
host = db
user = root
password = password
Then you can use this health-check.bash script to check the health:
#!/usr/bin/env bash
DATABASE_DOCKER_CONTAINER=$1
# Check on health database server before continuing
function get_service_health {
# https://gist.github.com/mixja/1ed1314525ba4a04807303dad229f2e1
docker inspect -f '{{if .State.Running}}{{ .State.Health.Status }}{{end}}' $DATABASE_DOCKER_CONTAINER
}
until [[ $(get_service_health) != starting ]];
do echo "database: ... Waiting on database Docker Instance to Start";
sleep 5;
done;
# Instance has finished starting, will be unhealthy until database finishes startup
MYSQL_HEALTH_CHECK_ATTEMPTS=12
until [[ $(get_service_health) == healthy ]]; do
echo "database: ... Waiting on database service"
sleep 5
if [[ $MYSQL_HEALTH_CHECK_ATTEMPTS == 0 ]];
then echo $DATABASE_DOCKER_CONTAINER ' failed health check (not running or unhealthy) - ' $(get_service_mysql_health)
exit 1
fi;
MYSQL_HEALTH_CHECK_ATTEMPTS=$((MYSQL_HEALTH_CHECK_ATTEMPTS-1))
done;
echo "Database is healthy"
Finally, you can use makefile to connect all things together. Something like that:
docker-up:
docker-compose up -d
db-health-check:
db/health-check.bash db
load-database:
docker run --rm --interactive --tty --network your_docker_network_name -v `pwd`:/application -w /application your_docker_db_image_name python /application/pricing_import.py
start: docker-up db-health-check load-database
Then start your app with make start.

Adding Flyway to a MySQL Docker Container

I'm building an derivative to this Docker container for mysql (using it as a starting point): https://github.com/docker-library/mysql
I've amended the Dockerfile to add in Flyway. Everything is set up to edit the config file to connect to the local DB instance, etc. The intent is to call this command from inside the https://github.com/docker-library/mysql/blob/master/5.7/docker-entrypoint.sh file (which runs as the ENTRYPOINT) around line 186:
flyway migrate
I get a connection refused when this is run from inside the shell script:
Flyway 4.1.2 by Boxfuse
ERROR:
Unable to obtain Jdbc connection from DataSource
(jdbc:mysql://localhost:3306/db-name) for user 'root': Could not connect to address=(host=localhost)(port=3306)(type=master) : Connection refused
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 08
Error Code : -1
Message : Could not connect to address=(host=localhost)(port=3306)(type=master) : Connection refused
But, if I remove the command from the shell script, rebuild and log in to the container, and run the same command manually, it works with no problems.
I suspect that there may be some differences with how the script connects to the DB to do its thing (it has a built in SQL "runner"), but I can't seem to hunt it down. The container restarts the server during the process, which is what may be the difference here.
Since this container is intended for development, one alternative (a work-around, really) is to use the built in SQL "runner" for this container, using the filename format that Flyway expects, then use Flyway to manage the production DB's versions.
Thanks in advance for any help.
I mean it's the good way to start from the ready image (for start).
You may start from image docker "mysql"
FROM mysql
If you start the finished image - when creating new version your docker then
will only update the difference.
Next, step you may install java and net-tools
RUN apt-get -y install apt-utils openjdk-8-jdk net-tools
Config mysql
ENV MYSQL_DATABASE=mydb
ENV MYSQL_ROOT_PASSWORD=root
Add flyway
ADD flyway /opt/flyway
Add migrations
ADD sql /opt/flyway/sql
Add config flyway
ADD config /opt/flyway/conf
Add script to start
ADD start /root/start.sh
Check start mysql
RUN netstat -ntlp
Check java version
RUN java -version
Example file: /opt/flyway/conf/flyway.conf
flyway.driver=com.mysql.jdbc.Driver
flyway.url=jdbc:mysql://localhost:3306/mydb
flyway.user=root
flyway.password=root
Example file: start.sh
#!/bin/bash
cd /opt/flyway
flyway migrate
# may change to start.sh to start product migration or development.
Flyway documentation
I mean that you in next step may use flyway as service:
For example:
docker run -it -p 3307:3306 my_docker_flyway /root/start << migration_prod.sh
docker run -it -p 3308:3306 my_docker_flayway /root/start << migration_dev.sh
etc ...
services:
# Standard Mysql Box, we have to add tricky things else logging by workbench is hard
supermonk-mysql:
image: mysql
command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
environment:
- MYSQL_ROOT_PASSWORD=P#ssw0rd
- MYSQL_ROOT_HOST=%
- MYSQL_DATABASE=test
ports:
- "3306:3306"
healthcheck:
test: ["CMD-SHELL", "nc -z 127.0.0.1 3306 || exit 1"]
interval: 1m30s
timeout: 60s
retries: 6
# Flyway is best for mysql schema migration history.
supermonk-flyway:
container_name: supermonk-flyway
image: boxfuse/flyway
command: -url=jdbc:mysql://supermonk-mysql:3306/test?verifyServerCertificate=false&useSSL=true -schemas=test -user=root -password=P#ssw0rd migrate
volumes:
- "./sql:/flyway/sql"
depends_on:
- supermonk-mysql
mkdir ./sql
vi ./sql/V1.1__Init.sql # and paste below
CREATE TABLE IF NOT EXISTS test.USER (
id VARCHAR(64),
fname VARCHAR(256),
lname VARCHAR(256),
CONSTRAINT pk PRIMARY KEY (id));
save and close
docker-compose up -d
wait for 2 minutes
docker-compose run supermonk-flyway
Ref :
https://github.com/supermonk/webapp/tree/branch-1/docker/docker-database
Thanks to docker community and mysql community
docker-compose logs -f