Django docker could not access MYSQL in host server (using docker-compose) - mysql

I have a working django application run via docker to be used for production. Currently, its also using a mysql database which is also hosted using docker. This actually works fine.
However, as what I learned, hosting mysql database on docker may not be a preferable way to do it in production. Which is why I wanted to use my host server, which has mysql running instead. My problem is, I can't seem to make it my django app connect to the host server's mysql server.
This is my docker-compose file.
version: '3.9'
networks:
default:
external: true
name: globalprint-shared-network
services:
#db:
# image: mysql:5.7
# ports:
# - "3307:3306"
# hostname: globalprint-db
# restart: always
# volumes:
# - production_db_volume:/var/lib/mysql
# env_file:
# - .env.prod
app:
build:
context: .
ports:
- "8001:8000"
volumes:
- production_static_data:/vol/web
hostname: globalprint-backend
restart: always
env_file:
- .env.prod
# depends_on:
# - db
proxy:
build:
context: ./proxy
hostname: globalprint-backend-api
volumes:
- production_static_data:/vol/static
restart: always
ports:
- "81:80"
depends_on:
- app
volumes:
production_static_data:
production_db_volume:
I actually have tried adding this in app service and made but still it did not work:
extra_hosts:
host.docker.internal:host-gateway
My django settings for database is also this one. Its actually referencing to an environment variable but the value for host is host.docker.internal and port is 3306:
DATABASES = {
'default': {
'ENGINE': env('DATABASE_ENGINE_MYSQL'),
'NAME': env('MYSQL_DATABASE'),
'USER': env('MYSQL_USER'),
'PASSWORD': env('MYSQL_PASSWORD'),
'HOST': env('DATABASE_HOST'), # localhost Or an IP Address that your DB is hosted on
'PORT': env('DATABASE_PORT'),
}
}
Can anyone tell what did I do wrong here? Appreciate any help on this. Thank you.

Related

Connect to MariaDB in Docker via R

I have a few Docker containers up and running. The main ones are RStudio (for the app that connects to the database) and a database that's been synced with github container registry (ghcr.io). The end goal will be transitioning the R Shiny app to a Dash app in Python. Right now I'm having trouble connecting the database to either the Shiny app in another container or Python (the Python app has yet to be developed). See below for the docker-compose.yml
version: '3'
services:
frontapp:
container_name: frontend
build:
context: .
dockerfile: shinyserver-dockerfile
ports:
- "3838:3838"
volumes:
- type: bind
source: ../
target: /XXXX/XX_front_end_R
networks:
- frontend
rstudio:
container_name: rstudio
build:
context: .
dockerfile: rstudio-dockerfile
ports:
- "8787:8787"
volumes:
- type: bind
source: ../
target: /home/rstudio
networks:
- frontend
db:
image: ghcr.io/XXXX/XX_db:main
container_name: db
ports:
- "3306:3306"
volumes:
- dbdata:/var/lib/mysql
phpmyadmin:
image: phpmyadmin/phpmyadmin
container_name: pma
links:
- db
environment:
PMA_HOST: db
PMA_PORT: 3306
PMA_ARBITRARY: 1
restart: always
ports:
- "8081:80"
networks:
- frontend
volumes:
dbdata:
networks:
frontend:
driver: bridge
Below is the result from docker ps
Then when I log into R Studio and try to run the app (in http://localhost:8787/), there's an error trying to connect to the database.
mydb <- dbConnect(MariaDB(), user = "XXXX", password = "YYYY",
dbname = "prototype", host = "db", port="3306")
I then get the following error:
Error: Failed to connect: Unknown MySQL server host 'db' (-2)
It looks like the app should be running on http://localhost:3838 although that page just returns an error.
Any help with connecting to the database is helpful. I'm not sure if there are other dependencies or installations that need to be done or if I'm not correcting in the right way. Thanks.
Note: I'm coming in fresh to this project. A previous developer has currently created what I have access to now.

Is there a way to use a generic hostname that will work in both Docker and localhost?

I’m on Mac OS Big Sur and running the following Docker versions
$ docker -v
Docker version 20.10.12, build e91ed57
$ docker-compose -v
Docker Compose version v2.2.3
I’m running a Rails 6 app, whose database is configured like so
development:
adapter: mysql2
encoding: utf8
host: host.docker.internal
database: cfs
pool: 5
username: myuser
password: bypass
I have to use “host.docker.internal” so that the Rails app can access the Docker db when running inside of Docker, set up in my docker-compose.yml like so
services:
db:
image: mysql:5.7
command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
restart: always
volumes:
- ./docker/provision/mysql/init:/docker-entrypoint-initdb.d
environment:
MYSQL_ROOT_PASSWORD: bypass
ports:
# <Port exposed> : < MySQL Port running inside container>
- '3306:3306'
expose:
# Opens port 3306 on the container
- '3306'
…
web:
#restart: always
build: ./my-project
ports:
- "3000:3000"
expose:
- '3000'
command: foreman start
volumes:
- ./my-project/:/app
depends_on:
- db
However if I run the app locally without Docker, I have to change my config files to remove “host.docker.internal” and use “127.0.0.1” instead (or localhost).
Is there a way I can set things up so that I have a single database config file that works in both Docker and without Docker such that I don’t have to change the host around?
You can use ERB markup in the database.yml file, which lets you use an environment variable here. In general I'd suggest making default values for things be whatever default will work in a plain non-Docker development environment.
# config/database.yml
development:
host: <%= ENV['DATABASE_HOST'] || 'localhost' %>
If you're running this in a setup where the database hostname isn't localhost – it's in a sibling container, or you're using a cloud-hosted database like Amazon RDS – you can set that environment variable.
# docker-compose.yml
version: '3.8'
services:
db:
image: 'mysql:5.7'
et: cetera
web:
build: .
ports:
- '3000:3000'
depends_on:
- db
environment:
DATABASE_HOST: db # <-- will be read in database.yml

Can't connect to MySQL container from other container

I'm trying to connect my FASTAPI app container to a MySQL database container using the docker-compose file. In the Docker documentation it says that docker creates a default network for both containers. However, I would like to use a pre-existing network that I've created(app-net).
This is my docker-compose file:
version: '3.4'
services:
mysql:
image: mysql:latest
command: mysqld --default-authentication-plugin=mysql_native_password
container_name: mysql
restart: always
ports:
- 3307:3306
environment:
MYSQL_ROOT_PASSWORD: password
volumes:
- mysql-data:/var/lib/mysql
app:
build: .
image: app:1.0
container_name: app
ports:
- 8000:8000
environment:
PASSWORD: password
USERNAME: root
networks:
default:
external: true
name: app-net
volumes:
mysql-data:
driver: local
this is the output I get when i run docker inspect mysql -f "{{json .NetworkSettings.Networks }}":
{"app-net":{"IPAMConfig":null,"Links":null,"Aliases":["mysql","5e998f9fb646"],"NetworkID":"7f60c83e4c88d25e674461521446ec9fa98baca8639c782c79671c4fcb77ba88","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"","DriverOpts":null}}
However, when I run each container individually using CMD with the --network app-net the output is different:
{"app-net":{"IPAMConfig":null,"Links":null,"Aliases":["46157e588c87"],"NetworkID":"7f60c83e4c88d25e674461521446ec9fa98baca8639c782c79671c4fcb77ba88","EndpointID":"6a6922a9a6ea8f9d113447decbbb927cb93ddffd3b9563ee882fa2e44970cde5","Gateway":"172.20.0.1","IPAddress":"172.20.0.2","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:ac:14:00:02","DriverOpts":null}}
In my app code in order to connect the mysql server, I specified the container name as the hostname since they are supposed to share the same network. But, as I mentioned it seems both containers can't talk to each other.
I'm pretty sure that is the reason I can't connect the database through my app and get that error when I run:
docker-compose -f docker-compose.yml up
I get this error:
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2005, "Unknown MySQL server host 'mysql' (0)")
What am I missing?
If the error you're getting is specifically "unknown host" or something similar, this can happen if your application container starts before the database container exists. You can work around this situation by telling Compose about the dependency:
version: '3.8'
services:
mysql: { ... }
app:
depends_on:
- mysql
This has two key effects. If you try to start only the application container docker-compose up app, it will also start the mysql container, following the depends_on: chain. When Compose does start containers, it at least creates the mysql container before creating the app container.
Note that this doesn't guarantee the database will actually be running by the time your application starts up. If you do encounter this, you will get a different error like "connection refused". You can work around this by embedding a script in your image that waits for the database to be available, or by using a version 2 Compose file with health-check support; see Docker Compose wait for container X before starting Y for more details on this specific problem.
You could add the key networks to both containers, this way:
version: '3.4'
services:
mysql:
image: mysql:latest
command: mysqld --default-authentication-plugin=mysql_native_password
container_name: mysql
restart: always
ports:
- 3307:3306
environment:
MYSQL_ROOT_PASSWORD: password
volumes:
- mysql-data:/var/lib/mysql
networks:
- app-net
app:
build: .
image: app:1.0
container_name: app
ports:
- 8000:8000
environment:
PASSWORD: password
USERNAME: root
networks:
- app-net
networks:
default:
external: true
name: app-net
volumes:
mysql-data:
driver: local

Spring Boot + MySQL + Docker Compose - Cannot make Spring Boot connect to MySQL

I've been trying to set up a connection between a backend (runs on Spring Boot) container and a pre-built MySQL container. However, I cannot get it to connect. My docker compose file is:
version: '3.7'
services:
test-mysql:
image: mysql
restart: always
volumes:
- db_data:/var/lib/mysql
environment:
MYSQL_DATABASE: testdb
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
backend:
depends_on:
- test-mysql
build:
context: backend
dockerfile: Dockerfile
ports:
- "8080:8080"
restart: always
volumes:
db_data: {}
my application.properties:
spring.jpa.hibernate.ddl-auto=update
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQL5InnoDBDialect
spring.datasource.url=jdbc:mysql://test-mysql:3306/testdb?autoReconnect=true&failOverReadOnly=false&maxReconnects=10
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.username=test
spring.datasource.password=test
When I use docker-compose up, Spring Boot is not able to recognize the container name test-mysql. It throws: java.net.UnknownHostException
When I change it to an IP, it says connection refused. I have been looking everywhere and couldn't come with a fix. I hope anyone can help me out. Thank you!
You have to mention the backend mysql properties in the composer file like below,
backend:
depends_on:
- test-mysql
build:
context: backend
dockerfile: Dockerfile
ports:
- "8080:8080"
restart: always
environment:
SPRING_DATASOURCE_URL: jdbc:mysql://test-
mysql:3306/testdbautoReconnect=true&failOverReadOnly=false&maxReconnects=10
SPRING_DATASOURCE_USERNAME: test
SPRING_DATASOURCE_PASSWORD: test
links:
- test-mysql:test-mysql
If this wouldn't work try to create a common docker network and add it to your composer file like below,
backend:
depends_on:
- test-mysql
build:
context: backend
dockerfile: Dockerfile
ports:
- "8080:8080"
restart: always
environment:
SPRING_DATASOURCE_URL: jdbc:mysql://test-
mysql:3306/testdbautoReconnect=true&failOverReadOnly=false&maxReconnects=10
SPRING_DATASOURCE_USERNAME: test
SPRING_DATASOURCE_PASSWORD: test
networks:
-common-network
test-mysql:
image: mysql
restart: always
volumes:
- db_data:/var/lib/mysql
environment:
MYSQL_DATABASE: testdb
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
networks:
-common-network
#Docker Networks
networks:
common-network:
driver: bridge
#Volumes
volumes:
dbdata:
driver: local
You can define a common network on which both the application server and the database can connect. Please check the file (docker-compose.yml) below where I have defined a common network: backend
# Docker Compose file Reference (https://docs.docker.com/compose/compose-file/)
version: '3.7'
# Define services
services:
# App backend service
app-server:
# Configuration for building the docker image for the backend service
build:
context: . # Use an image built from the specified dockerfile in the `springboot-app-server` directory.
dockerfile: Dockerfile
ports:
- "8080:8080" # Forward the exposed port 4000 on the container to port 4000 on the host machine
restart: always
depends_on:
- db # This service depends on mysql. Start that first.
environment: # Pass environment variables to the service
SPRING_DATASOURCE_URL: jdbc:mysql://test-mysql:3306/testdb?useSSL=false&serverTimezone=UTC&useLegacyDatetimeCode=false
SPRING_DATASOURCE_USERNAME: test
SPRING_DATASOURCE_PASSWORD: test
networks: # Networks to join (Services on the same network can communicate with each other using their name)
- backend
# Database Service (Mysql)
db:
image: mysql:5.7
ports:
- "3306:3306"
restart: always
environment:
MYSQL_DATABASE: testdb
MYSQL_USER: test
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
volumes:
- db-data:/var/lib/mysql
networks:
- backend
# Volumes
volumes:
db-data:
# Networks to be created to facilitate communication between containers
networks:
backend:
I have written a blog and a simple working Spring Boot MySQL application on GitHub which tells about using Docker Compose. Please check: http://softwaredevelopercentral.blogspot.com/2020/10/spring-boot-mysql-docker-compose-example.html
If you would like to use this test-mysql in your spring config
spring.datasource.url=jdbc:mysql://test-mysql:3306/testdb?autoReconnect=true&failOverReadOnly=false&maxReconnects=10
Then add the hostname attribute at service test-mysql
version: '3.7'
services:
test-mysql:
image: mysql
hostname: test-mysql
...
I hope this has been already solved but in case it hasn't yet, the problem lies in mysql docker container lagging behind the start up.
Another problem is that you might need to build the jar file and then copy it the container. That's a big problem because when you build the jar file, the database with db as hostname is unavailable. So when you are building the jar file, skip the test.
This is bash script i created but you can run command one by one:
#!/bin/bash
cd storage-service
rm -rf target/
mvn clean compile package -Dmaven.test.skip=true
cd ..
docker-compose up
In case you want to initialize db in the container. That file is in the folder env where i have a file database.env
-- create the databases
CREATE DATABASE IF NOT EXISTS model_storage;
-- create the users for each database
CREATE USER 'arsene'#'localhost' IDENTIFIED BY 'arsene';
GRANT CREATE, ALTER, INDEX, LOCK TABLES, REFERENCES, UPDATE, DELETE, DROP, SELECT, INSERT ON `model_storage`.* TO 'arsene'#'localhost';
FLUSH PRIVILEGES;
The backend service Dockerfile looks like this:
FROM adoptopenjdk/openjdk11
COPY target/*.jar storage.jar
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /storage.jar" ]
EXPOSE 8089
The database env file looks like this:
MYSQL_ROOT_PASSWORD=arsene
MYSQL_DATABASE=model_storage
MYSQL_USER=arsene
MYSQL_PASSWORD=arsene
DATABASE_HOST=model_storage
DATABASE_USER=arsene
DATABASE_PASSWORD=arsene
DATABASE_NAME=model_storage
DATABASE_PORT=3306
In case you intend to pass JAVA_OPTS env in the image. These can be used later as seen in docker-compose.yml below
Your backend (the service that depends on the mysql db) needs to restart until the docker-compose is able to resolve the the container name of mysql, in my case its name is db. And don't forget to include datasource connection properties in docker-compose backend service image as i did below. I am not an expert in spring boot and neither in docker but for now it works!
Below is the way mine is structured:
I am using docker version: "3.8"
Storage service
storage-service:
container_name: storage-service
restart: always
build:
context: storage-service
image: "service_storage_image"
depends_on:
- db
ports:
- "8089:8089"
links:
- db
env_file:
- env/database.env
environment:
WAIT_HOSTS: db:3306
SPRING_DATASOURCE_URL: jdbc:mysql://db:3306/model_storage?allowPublicKeyRetrieval=true&useSSL=false
SPRING_DATASOURCE_USERNAME: root
SPRING_DATASOURCE_PASSWORD: arsene
healthcheck:
test: "/usr/bin/mysql --user=arsene --password=arsene--execute \"SHOW DATABASES;\""
interval: 2s
timeout: 20s
retries: 10
environment:
- JAVA_OPTS=
-DEUREKA_SERVER=http://eureka-registry-server:7070/eureka
-DZIPKIN_SERVER=http://zipkin:9411/
networks:
- private-network-mms
My db in docker-compose is structured this way:
Mysql database
db:
hostname: db
container_name: db
image: "mysql:latest"
env_file:
- env/database.env
volumes:
- type: bind
source: ./env/setup.sql
target: /docker-entrypoint-initdb.d/setup.sql
- db_volume:/var/lib/mysql
ports:
- 3307:3306
networks:
- private-network-mms

docker-compose run two instance of mysql

I want to run two instances of mysql using docker-compose.
I'm running MySQL in a Docker container and I have another Docker container running a python script to access the MySQL database.
One works fine on port 3306. In order to get two working, I thought I would just run the other one on a different port. But when I change it to a different port (e.g. 6603), but when I do, I get the below error:
mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'mysql:6603' (111 Connection refused)
I have read every question on s.o. I can find that seems relevant but none of the solutions work. I feel certain the fix will involve changing a line or two of configuration but I've spent many hours on this so far so any help would be greatly appreciated.
The docker-compose script is below (works fine if 6603 is replaced with 3306).
server:
build:
context: ../
dockerfile: Docker/ServerDockerfile
ports:
- "8080:8080"
links:
- mysql:mysql
volumes:
- ../py:/app
tty: true
mysql:
image: mysql
expose:
- "6603"
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: project
volumes:
- ./MySQL:/docker-entrypoint-initdb.d
- ./MySQL/data:/var/lib/mysql
And it is being accessed like this:
cnx = mysql.connector.connect(user='root', password='password',
host='mysql',
port="6603",
database='project')
Try to specify another port for MySQL by modifying its my.cnf file.
Have eventually found a couple of ways that work. The neatest one is for each app to create a network and connect the containers to it.
If each app uses a different network then mysql can run on 3306 on that Docker network and can be accessed on mysql://3306 from app1 and mysql2://3306 from app2. (Assuming you name you give the mysql service for app 2 is mysql2).
The Docker file with the new lines is below:
server:
build:
context: ../
dockerfile: Docker/ServerDockerfile
ports:
- "8080:8080"
volumes:
- ../py:/app
tty: true
networks:
-net
mysql2:
image: mysql
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: project
volumes:
- ./MySQL:/docker-entrypoint-initdb.d
- ./MySQL/data:/var/lib/mysql
networks:
-net
networks:
net:
driver: bridge
The Docker file for the second app is identical except the names are different (I put a 2 after each for simplicity).
server2:
build:
context: ../
dockerfile: Docker/ServerDockerfile
ports:
- "8081:8080"
volumes:
- ../py:/app
tty: true
networks:
-net2
mysql2:
image: mysql
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: project
volumes:
- ./MySQL:/docker-entrypoint-initdb.d
- ./MySQL/data:/var/lib/mysql
networks:
-net2
networks:
net2:
driver: bridge