I am having real problems trying to get a docker-compose script to initiate a mysql database and a Django project, but get the Django project to wait until the mysql database is ready.
I have two files, a Dockerfile and a docker-compose.yml, which I have copied below.
When I run the docker-compose.yml, and check the logs of the web container, it says that it cannot connect to the database mydb. However the second time that I run it (without clearing the containers and images) it connects properly and the Django app works.
I have spent a whole day trying a number of things such as scripts, health checks etc, but I cannot get it to work.
Dockerfile
FROM python:3.6
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY ./ /code/
RUN pip install -r requirements.txt
RUN python manage.py collectstatic --noinput
docker-compose.yml
version: '3'
services:
mydb:
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_USER=django
- MYSQL_PASSWORD=secret
- MYSQL_DATABASE=dbMarksWebsite
image: mysql:5.7
ports:
# Map default mysql port 3306 to 3308 on outside so that I can connect
# to mysql using workbench localhost with port 3308
- "3308:3306"
web:
environment:
- DJANGO_DEBUG=1
- DOCKER_PASSWORD=secret
- DOCKER_USER=django
- DOCKER_DB=dbMarksWebsite
- DOCKER_HOST=mydb
- DOCKER_PORT=3306
build: .
command: >
sh -c "sleep 10 &&
python manage.py migrate &&
python manage.py loaddata myprojects_testdata.json &&
python manage.py runserver 0.0.0.0:8080"
ports:
- "8080:8080"
depends_on:
- mydb
First run (with no existing images or containers):
...
File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 84, in Connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 179, in __init__
super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2002, "Can't connect to MySQL server on 'mydb' (115)")
Second run:
System check identified no issues (0 silenced).
March 27, 2020 - 16:44:57
Django version 2.2.11, using settings 'ebdjango.settings'
Starting development server at http://0.0.0.0:8080/
Quit the server with CONTROL-C.
I solved it using the following function in my entrypoint.sh:
function wait_for_db()
{
while ! ./manage.py sqlflush > /dev/null 2>&1 ;do
echo "Waiting for the db to be ready."
sleep 1
done
}
For anybody who is interested, I found a solution to this:
1 - I wrote a python script to connect to the database every second,
but with a timeout. I set this timeout to be quite high at 60
seconds, but this seems to work on my computer.
2 - I added the command to wait into my compose file.
It should mean that I can bring up a set of test containers for my website, where I can specify the exact version of Python and MySQL used.
The relevant files are listed below:
Dockerfile:
FROM python:3.6
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY ./ /code/
RUN pip install -r requirements.txt
RUN python manage.py collectstatic --noinput
docker-compose.yml
version: '3'
services:
mydb:
container_name: mydb
environment:
- MYSQL_ROOT_PASSWORD=password
- MYSQL_USER=django
- MYSQL_PASSWORD=secret
- MYSQL_DATABASE=dbMarksWebsite
image: mysql:5.7
ports:
# Map default mysql port 3306 to 3308 on outside so that I can connect
# to mysql using workbench localhost with port 3308
- "3308:3306"
web:
container_name: web
environment:
- DJANGO_DEBUG=1
- DOCKER_PASSWORD=secret
- DOCKER_USER=django
- DOCKER_DB=dbMarksWebsite
- DOCKER_HOST=mydb
- DOCKER_PORT=3306
build: .
command: >
sh -c "python ./bin/wait-for.py mydb 3306 django secret dbMarksWebsite 60 &&
python manage.py migrate &&
python manage.py loaddata myprojects_testdata.json &&
python manage.py runserver 0.0.0.0:8080"
ports:
- "8080:8080"
depends_on:
- mydb
wait-for.py
'''
I don't like adding this in here, but I cannot get the typical wait-for scripts
to work with MySQL database in docker, so I hve written a python script that
either times out after ? seconds or successfully connects to the database
The input arguments for the script need to be:
HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT
'''
import sys, os
import time
import pymysql
def readCommandLineArgument():
'''
Validate the number of command line input arguments and return the
input filename
'''
# Get arguments
if len(sys.argv)!=7:
raise ValueError("You must pass in 6 arguments, HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT")
# return the arguments as a tuple
return (sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5], sys.argv[6])
def connectToDB(HOST, PORT, USERNAME, PASSWORD, DATABASE):
'''
for now, just try to connect to the database.
'''
con = pymysql.connect(host=HOST, port=PORT, user=USERNAME, password=PASSWORD, database=DATABASE)
with con:
cur = con.cursor()
cur.execute("SELECT VERSION()")
def runDelay():
'''
I don't like passing passwords in, but this is only used for a test docker
delay script
'''
# Get the database connection characteristics.
(HOST, PORT, USERNAME, PASSWORD, DATABASE, TIMEOUT) = readCommandLineArgument()
# Ensure timeout is an integer greater than zero, otherwise use 15 secs a default
try:
TIMEOUT = int(TIMEOUT)
if TIMEOUT <= 0:
raise("Timeout needs to be > 0")
except:
TIMEOUT = 60
# Ensure port is an integer greater than zero, otherwise use 3306 as default
try:
PORT = int(PORT)
if PORT <= 0:
raise("Port needs to be > 0")
except:
PORT = 3306
# Try to connect to the database TIMEOUT times
for i in range(0, TIMEOUT):
try:
# Try to connect to db
connectToDB(HOST, PORT, USERNAME, PASSWORD, DATABASE)
# If an error hasn't been raised, then exit
return True
except Exception as Ex:
strErr=Ex.args[0]
print(Ex.args)
# Sleep for 1 second
time.sleep(1)
# If I get here, assume a timeout has occurred
raise("Timeout")
if __name__ == "__main__":
runDelay()
For testing/development purposes, you could use a version of the MySQL image that has health checks (I believe there's a healthcheck/mysql image), or configure your own (see example here: Docker-compose check if mysql connection is ready).
For production use, you don't want to upgrade the database schema on startup, nor do you want to assume the database is up. Upgrading schema automatically encourages you to not think about what happens when you deploy a bug and need to rollback, and parallel schema upgrades won't work. Longer version: https://pythonspeed.com/articles/schema-migrations-server-startup/
Another option is to use a script to control the startup order, and wrap the web service's command.
In the docker-compose's documentation "wait-for-it" is one of the recommended tools, but other exists.
Performing database migrations using flyway. Using the dockerized version and using conf files for mirgations configurations.
Below is my config file
flyway.url = jdbc:mysql://${MYSQLHOST}:3306/myschema
flyway.user = myusername
flyway.password = mypassword
flyway.schemas = myschema
flyway.cleanDisabled = true
Am running the below command to perform migration
sudo docker run -e "MYSQLHOST=myhostip" --rm -it -v `pwd`/path/to/confi/:/flyway/conf/ -v `pwd`/path/to/migrations:/flyway/sql boxfuse/flyway:5.1.4 -configFiles=/flyway/conf/flyway.conf migrate
Am getting the below error
ERROR:
Unable to obtain connection from database (jdbc:mysql://${MYSQLHOST}:3306/myschema) for user 'myuser': Could not connect to address=(host=${MYSQLHOST})(port=3306)(type=master) : ${MYSQLHOST}
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 08
Error Code : -1
Message : Could not connect to address=(host=${MYSQLHOST})(port=3306)(type=master) : ${MYSQLHOST}
If I change the config file with my host ip details the migrations are successful without any errors. What am I doing wrongly?
Flyway allows you to pass in configuration parameters via a specific set of environment variables; it doesn't perform a complete variable substitution in the config file. See https://flywaydb.org/documentation/envvars.
What you could do is delete the first line of the config file, then
sudo docker run -e "FLYWAY_URL=jdbc:mysql://myhostip:3306/myschema" ...
I found it not very helpful to only see a page explaining that there are envVars -- but no mention of the exact env variable names to actually use.
Here is the definition of the env names / especially for the gradle plugin but I suspect that they are generally valid:
https://github.com/flyway/flyway/blob/master/flyway-core/src/main/java/org/flywaydb/core/internal/configuration/ConfigUtils.java#L139
ALERT: WAIT BEFORE MARKING THIS AS A DUPLICATE, THERE ARE A LOT OF SIMILIAR QUESTIONS HERE, BUT NOTHING THAT WORKED FOR ME.
I'm starting to learn the php framework Symfony and I have some problems with Entities and migration. When i try to run the command bin/console make:migration on my git Bush, it gives me these errors:
In AbstractMySQLDriver.php line 106:
An exception occurred in driver: could not find driver
In PDOConnection.php line 31:
could not find driver
In PDOConnection.php line 27:
could not find driver
I am working with Symfony 4. My PC is a Windows 10 and the code is running in an Apache server made with XAMPP. The DB is local (always made with XAMPP) and it is MySQL (server MariaDB 10.4.6).The PHP version is 7.3.9.
This is my .env Symfony file:
###> symfony/framework-bundle ###
APP_ENV=dev
APP_SECRET=b9c673023606f26343aee9c3d24a8aba
#TRUSTED_PROXIES=127.0.0.1,127.0.0.2
#TRUSTED_HOSTS='^localhost|example\.com$'
###< symfony/framework-bundle ###
###> doctrine/doctrine-bundle ###
# Format described at https://www.doctrine-project.org/projects/doctrine-dbal/en/latest/reference/configuration.html#connecting-using-a-url
# For an SQLite database, use: "sqlite:///%kernel.project_dir%/var/data.db"
# Configure your db driver and server_version in config/packages/doctrine.yaml
DATABASE_URL=mysql://root#localhost:3306/symfony
###< doctrine/doctrine-bundle ###
###> symfony/swiftmailer-bundle ###
# For Gmail as a transport, use: "gmail://username:password#localhost"
# For a generic SMTP server, use: "smtp://localhost:25?encryption=&auth_mode="
# Delivery is disabled by default via "null://localhost"
MAILER_URL=null://localhost
###< symfony/swiftmailer-bundle ###
this is my doctrine.yaml:
doctrine:
dbal:
# configure these for your database server
driver: 'pdo_mysql'
server_version: '5.7'
charset: utf8mb4
default_table_options:
charset: utf8mb4
collate: utf8mb4_unicode_ci
url: '%env(resolve:DATABASE_URL)%'
orm:
auto_generate_proxy_classes: true
naming_strategy: doctrine.orm.naming_strategy.underscore
auto_mapping: true
mappings:
App:
is_bundle: false
type: annotation
dir: '%kernel.project_dir%/src/Entity'
prefix: 'App\Entity'
alias: App
In this file I tried to change the driver with php_pdo_mysql (in c:\xampp\php\ext I have a file with this name), the charset, the db version (in mariadb-10.4.6) but nothing changed.
In my php.ini (the once at c:\xampp\php ) I already removed the semicolon at extension=pdo_mysql but nothing changed.
I've also tried to uninstall and re-install doctrine, to create a new project, to search online for some sort of "mysql drivers" (maybe I searched the wrong things) and also to uninstall and re-install the entire xampp.
Nothing worked.
How can I solve this issue?
Seems that you don't use php.ini inside xampp but C:\Program Files (x86)\php\php.ini
Your php.ini try to search settings in C:\Program Files (x86)\php\php.ini
do you have another istance of php installed on your pc?
maybe in PATH environment php.exe path is out of folder of xampp
you can try to remove semicolon from C:\Program Files (x86)\php\php.ini to pdo_mysql and should fix your problem
but better solution is to fix PATH environment in windows to search php.exe in c:\xampp\php
I try to connect to MySQL using LUA however an error occurs such as:
Module 'luasql.mysql' not found:
no field package.preload['luasql.mysql']
luasql = require "luasql.mysql"
env = assert (luasql.mysql())
con=assert(env:connect("db_name","username","pass","host",port))
How can I connect successfully to MySQL database?
Are you using lua-jdbc ?
Try luarocks install luasql-jdbc
Then install lua-mysql luarocks install luasql-mysql
Run again.
Check the Project: https://github.com/keplerproject/luasql
I have installed hue on my local ubuntu system and installed hadoop muti cluster system on two system.
Hadoop Version : 2.7.3
Hue Version : 3.12.0
Ozzie Version : 4.3.0
I am facing issue when I am running sqoop job process from mysql to import data from HDFS. I am getting following error.
Caused by: java.net.NoRouteToHostException: No Route to Host from Developer4/127.0.0.1 to cm:10020 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost
HDFS url hdfs://master:9000
My /etc/hosts file like
192.168.1.149 master
127.0.0.1 developer4
192.168.1.161 slave
Please suggest me where I am doing wrong. Even ozzie command for start and stop command work properly on command line.
You have set Hadoop in your localhost system then you need to remove or modified below things in core-site.xml file.
mapreduce.jobhistory.address 0.0.0.0:10020 Host and port for Job History Server (default 0.0.0.0:10020)
After that you need to run jobhistoryservice with below command.
sbin/mr-jobhistory-daemon.sh --config /home/developer4/hadoop-2.7.3/etc start historyserver
After this command port is enable on your localhost and hope this will help you.