I'm new to Docker. I managed to build an image with the things I wanted (CentOS with Apache, PHP, MySQL, MailHog and supervisord.) It works fine.
Now, what I'm trying to do is turn my image (one container) into multiple images/containers: one for web, one for db, etc.
I managed to build those different images, but I'm having trouble linking web and db together via docker-composer.yml. Here is what I have:
web:
container_name: centosweb
image: fab/centosweb
ports:
- "80:80"
volumes:
# Single files
- ./config/httpd.conf:/etc/httpd/conf/httpd.conf
# Directories
- ./vhosts:/var/www/html
- /Users/fabien/Dropbox/AppData/XAMPP/web/bilingueanglais/public_html:/var/www/html/bilingueanglais
- ./logs/apache:/etc/httpd/logs # This will include access_log(s) and error_log(s), including PHP errors.
links:
- db
db:
container_name: centosdb
image: fab/centosdb
volumes:
# Single files
- ./config/my.cnf:/etc/my.cnf
# Directories
- ./mysqldata:/var/lib/mysql
- ./logs/mysql:/var/log/mysql
The output of docker ps is this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
95048de7a6c4 fab/centosweb "supervisord -n" 14 minutes ago Up 14 minutes 22/tcp, 8025/tcp, 0.0.0.0:80->80/tcp centosweb
eab3047a2dde fab/centosdb "supervisord -n" 14 minutes ago Up 14 minutes 22/tcp, 80/tcp, 8025/tcp centosdb
Trying to connect to my vhosts, I get a database error:
Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'
However, I can connect to centosdb, enter MySQL, and confirm the database is there (i.e.: the db server itself runs fine.)
My understanding so far is that I'm missing a way to tell MySQL to allow connections from the web app instead of the default behavior (connections coming from localhost.) However, I'm confused as to just how to do that.
This is very similar to this question but the latter doesn't contain the specifics I'm looking for.
EDIT: the source of the Dockerfile for the db container.
FROM centos:6.9
# Install MySQL (MariaDB)
# Warning: the repo is super slow in my experience (e.g.: 15 min for 191 MB.)
RUN yum -y update
ADD MariaDB.repo /etc/yum.repos.d/MariaDB.repo
RUN yum install -y MariaDB-server MariaDB-client
# Install supervisord
# EPEL = Extra Packages for Enterprise Linux -- used for python
RUN rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
RUN yum install -y python-pip && pip install "pip>=1.4,<1.5" --upgrade
RUN pip install supervisor
##
# START SERVER
# port 22: SSH
# port 80: TCP, HTTP
# port 8025: MailHog UI (web)
##
ADD supervisord-db.conf /etc/supervisord.conf
EXPOSE 22 80 8025
CMD ["supervisord", "-n"]
# MEMO · BUILD THE IMAGE:
# docker build -t fab/centosdb .
How is your application configured to talk to MySQL? Instead of localhost:3306 you need to tell your application to connect to MySQL at db:3306. The link sets a DNS name for the MySQL container as db and your application should be able to resolve this DNS name to get the correct IP address of the MySQL container.
Related
I am newbie on using Ubuntu, I am trying to install apache-superset and successfully installed it by using Docker by directly sudo docker pull apache/superset, but I am stuck at adding Database Connector to the running superset
As for my local database I am using MySQL and I happen to use SSH Tunnel in localhost to access it in server. So I think at Docker container perspective this must be an "external" databases
What I have tried:
I installed mysqlclient from pip3
By following this references: https://devopsheaven.com/docker/devops/add-host/link/2017/10/04/connect-external-services-from-docker-container.html
I tried to type: sudo docker run -it mysql -h 192.168.100.1 -P 33063 -u czjovan --password=mypw cz_payment_merged but then i get:
2021-03-04 11:34:53+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.23-1debian10 started.
2021-03-04 11:34:53+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
command was: mysqld -h 192.168.100.1 -P 33063 -u czjovan --password=mypw cz_payment_merged --verbose --help --log-bin-index=/tmp/tmp.EV6L0jrspQ
2021-03-04T11:34:53.402148Z 0 [ERROR] [MY-010124] [Server] Fatal error: Can't change to run as user 'czjovan' ; Please check that the user exists!
2021-03-04T11:34:53.403355Z 0 [ERROR] [MY-010119] [Server] Aborting
By Following this also: From inside of a Docker container, how do I connect to the localhost of the machine?
I tried to type: sudo docker run --rm it --network=host mysql mysql -h 127.0.0.1 but got error 2003 (hy000) can't connect to mysql server on 127.0.0.1
I tried to add mysqlconnector to sqlalchemy uri, but the driver not found
I am not an expert by setting these, I lack of Docker mechanism.. appreciate it if anyone willingly to direct me how to step by step resolving this..
UPDATE-------------------------------------------------:
-> Following Mustafa Guler to add -p 3306:3306, the mysql container now starts..
but I still cannot add database in Superset, what should I do next?
You need to provide the IP of the docker host machine. If you're using default networking, use the static IP 172.17.0.1 for the host.
Alternatively you can try host.docker.internal as the hostname.
You need to define user and pass as environment to create new user during start of mysql container. Maybe problem is user and pass MYSQL_USER, MYSQL_PASSWORD Also you do not expose 3306 port on docker run. Please use -p 3306:3306
I found out that installing Superset manually from scratch than using Superset from Docker Container are more a solution to me, since configuring Superset from docker manually can be a little cumbersome, what I did:
#-- Install Superset and MySQL Locally from Scratch ---#
sudo pip install apache-superset (
sudo apt install mysql-client-core-8.0
sudo apt install mysql-server
sudo pip install mysqlclient (for Superset Database Connectors)
#NOTE: there is also a package called 'superset' in pip, in my case i uninstalled this to ensure that only apache-superset is used
service mysql start
when mysql start, try to set password for the first time:
sudo mysqladmin -u root password
then to test it: mysql -u root -p, enter a created password
if it can enter mysql normally, the password set is successful,
CREATE DATABASE superset (this will save all config that superset progress will be saved)
Editing config.py in apache superset, which usually located on /usr/local/lib/python3.8/dist-packages/superset/config.py (this depends on pip installation from no 1),
8a) Edit the sqlalchemy uri part in config.py, so it can connect to a local installed mysql in ubuntu, to something like 'mysql://root:#localhost:3306/superset'
8b) Ensure that the database part in config.py is superset, or the same name with database name created in MySQL
After all database, sqlalchemy uri, and database name is prepared, its good to go to follow with Superset configs stated from: https://superset.apache.org/docs/installation/installing-superset-from-scratch
A) sudo superset db upgrade
B) sudo superset fab create-admin
C) sudo superset load examples
D) sudo superset init
E) sudo superset run -p 8088 --with-threads --reload --debugger (run this to start superset daily)
Run a Private SSH Tunnel Connecting to Database (Optional, in my case I use SSH Tunnel to access database)
And then I am able to add connection locally in Superset defining sqlalchemy uri
A) towards the SSH Tunnel mysql://(server_user):(server_password)#127.0.0.1:33063/(database_name)
(this is outside from local MySQL)
B) towards Local Installed MySQL 'mysql://root:rootpwd#127.0.0.1:3306/database_name'
I will recommend to use docker than manual installation by pip. Package dependencies especially flask are real problems.
Do not follow docker compose as beginner.
I had a problems using docker compose. Port is closed was annoying problem due to networking. Host.docker.internal doesn’t worked for me on Ubuntu 22 on Google cloud. I switched to manual installation and it was package versions hell on python 3.8. I would like to recommend to not follow official doc and use better approach with single docker image to start. Instead of running 5 containers by compose, run everything in one. Use official docker image, here image. Than modify docker file as follows to install custom db driver:
FROM apache/superset
USER root
RUN pip install mysqlclient
RUN pip install sqlalchemy-redshift
USER superset
Second step is to build new image based on docker file description. To avoid networking problems start both containers on same network (superset, your db) easier is to use host network. I used this on Google cloud, example as follow:
docker run -d --network host --name superset supers
The same command to start container with your database. —network host. This solved my problems. More about in whole step to step tutorial: medium or here blog
I can create a docker containing my project code. It has unit tests that fail because there is no database connection.
I can log in to a server with a central database that contains our test data using SSH key and credentials.
I cannot get the docker and DB communicating.
I've tried several different suggestions, scratching and restarting this portion of the Dockerfile over the past two days. I've searched Youtube for tutorials, Stackexchange for answers and the docker forums for reference.
If there's a step by step tutorial, that is tucked away I would love to see that too!
The docker-compose has the following:
services:
app:
build:
context: .
dockerfile: .docker/Dockerfile
args:
APP_PATH: ${APP_PATH}
image: laravel-docker
env_file: .env
ports:
- 8080:80
# We need to expose 443 port for SSL certification.
- "443:443"
volumes:
- .:/var/www/jumbledown
Inside the container, I can contact the host of the remote DB with the following:
ssh -4 -R 8888:localhost:8888 [devname]#NN.NN.NN.NN -i ~/ident -p [portnumber]
where:
- devname is my log in.
- NN.NN.NN.NN is the IP address of the host of the DB.
- ident is a file containing the SSH key that is copied in by a copy command contained in the Dockerfile.
The Docker file is built off FROM php:7.1.8-apache and installs a LOT of extra stuff now, including Xdebug. It's too long to just copy and paste and I'm not sure what parts are relevant; I can expose at request.
Ideally, I'd like to be able to use Dockerfile to set up an SSH tunnel serving the DB to the docker container. Right now, I'd settle for being able to manually set up the connection inside the container.
Update As per questions in the answer, the end result I need to create is for several developers to have local dockers and each have a tunnel to a central database that contains testing data, for our use while we code throughout the day.
If you want the PHP container to have a permanent SSH tunnel to your remote DB, you can change you Dockerfile's COMMAND statement (assuming the ENTRYPOINT is a shell) to use a script that creates the SSH tunnel in the background, similar to what you manually, wait for the SSH tunnel and then proceed to run whatever it is you want to run.
Your question lacks the details of what you're trying to achieve (permanent tunnel? only while testing? etc.)
An example to such script:
# run ssh in background (notice the "&" at the end)
ssh -4 -R 8888:localhost:8888 $DB_USERNAME#$DB_HOST -i ~/ident -p $DB_PORT &
# wait for the ssh tunnel if needed
# ...
# run the main command here
# ...
I'd suggest to consider a different path -
Create a new service in the docker-compose file that is dedicated to open a tunnel, and then connect to that service from your PHP service.
Win 10
Composer version 1.4.1 2017-03-10 09:29:45
PHP 7
npm/Node
Docker CE
Apache 2.4
Powershell
git BASH shell
drush (installed via composer)
Noob Composer/Docker skills
I have a docker config yml specifying how mysql service can start:
version: "2"
services:
mysql:
image: mysql:5.6
ports:
- 3306:3306
volumes:
- /data/nbif_mysql:/var/lib/mysql
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
And when I call
#!/bin/bash
docker-compose up -d mysql
I see that the container is running:
PS C:\dev\appname> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f1a0ecab8af6 mysql:5.6 "docker-entrypoint..." 2 hours ago Up 5 seconds 0.0.0.0:3306->3306/tcp appname_mysql_1
But, notice that the reported IP is 0.0.0.0:3306->3306/tcp
So when I try to connect with the expected IP, it fails:
ERROR 2003 (HY000): Can't connect to MySQL server on '172.17.0.1' (10060)
How to I tell docker-compose to use that expected IP for docker?
Is this a setup issue, or some config tweak I need to do?
When you bind a port to your host, you have to use localhost instead of the container's IP address, because you're not assigning any local IP address.
Every container always runs in a isolated network (bridge), the container in your compose file will be able to find the others by their hostname, but inside of those containers, they are isolated from the local network so that's why you can't reach them.
In your compose file you only have a mysql container and you're binding that port in your host, so the only way to reach that container is by using localhost:3306
Remember, when you run a docker container it isn't like a server with an IP in your host network, it's more like a virtual machine with an isolated network configuration.
Take a look on the docker-compose docs in this specific topic:
https://docs.docker.com/compose/networking
UPDATE:
The link that finally answered the question was:
https://docs.docker.com/engine/userguide/networking/default_network/custom-docker0/
I'm building an derivative to this Docker container for mysql (using it as a starting point): https://github.com/docker-library/mysql
I've amended the Dockerfile to add in Flyway. Everything is set up to edit the config file to connect to the local DB instance, etc. The intent is to call this command from inside the https://github.com/docker-library/mysql/blob/master/5.7/docker-entrypoint.sh file (which runs as the ENTRYPOINT) around line 186:
flyway migrate
I get a connection refused when this is run from inside the shell script:
Flyway 4.1.2 by Boxfuse
ERROR:
Unable to obtain Jdbc connection from DataSource
(jdbc:mysql://localhost:3306/db-name) for user 'root': Could not connect to address=(host=localhost)(port=3306)(type=master) : Connection refused
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 08
Error Code : -1
Message : Could not connect to address=(host=localhost)(port=3306)(type=master) : Connection refused
But, if I remove the command from the shell script, rebuild and log in to the container, and run the same command manually, it works with no problems.
I suspect that there may be some differences with how the script connects to the DB to do its thing (it has a built in SQL "runner"), but I can't seem to hunt it down. The container restarts the server during the process, which is what may be the difference here.
Since this container is intended for development, one alternative (a work-around, really) is to use the built in SQL "runner" for this container, using the filename format that Flyway expects, then use Flyway to manage the production DB's versions.
Thanks in advance for any help.
I mean it's the good way to start from the ready image (for start).
You may start from image docker "mysql"
FROM mysql
If you start the finished image - when creating new version your docker then
will only update the difference.
Next, step you may install java and net-tools
RUN apt-get -y install apt-utils openjdk-8-jdk net-tools
Config mysql
ENV MYSQL_DATABASE=mydb
ENV MYSQL_ROOT_PASSWORD=root
Add flyway
ADD flyway /opt/flyway
Add migrations
ADD sql /opt/flyway/sql
Add config flyway
ADD config /opt/flyway/conf
Add script to start
ADD start /root/start.sh
Check start mysql
RUN netstat -ntlp
Check java version
RUN java -version
Example file: /opt/flyway/conf/flyway.conf
flyway.driver=com.mysql.jdbc.Driver
flyway.url=jdbc:mysql://localhost:3306/mydb
flyway.user=root
flyway.password=root
Example file: start.sh
#!/bin/bash
cd /opt/flyway
flyway migrate
# may change to start.sh to start product migration or development.
Flyway documentation
I mean that you in next step may use flyway as service:
For example:
docker run -it -p 3307:3306 my_docker_flyway /root/start << migration_prod.sh
docker run -it -p 3308:3306 my_docker_flayway /root/start << migration_dev.sh
etc ...
services:
# Standard Mysql Box, we have to add tricky things else logging by workbench is hard
supermonk-mysql:
image: mysql
command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
environment:
- MYSQL_ROOT_PASSWORD=P#ssw0rd
- MYSQL_ROOT_HOST=%
- MYSQL_DATABASE=test
ports:
- "3306:3306"
healthcheck:
test: ["CMD-SHELL", "nc -z 127.0.0.1 3306 || exit 1"]
interval: 1m30s
timeout: 60s
retries: 6
# Flyway is best for mysql schema migration history.
supermonk-flyway:
container_name: supermonk-flyway
image: boxfuse/flyway
command: -url=jdbc:mysql://supermonk-mysql:3306/test?verifyServerCertificate=false&useSSL=true -schemas=test -user=root -password=P#ssw0rd migrate
volumes:
- "./sql:/flyway/sql"
depends_on:
- supermonk-mysql
mkdir ./sql
vi ./sql/V1.1__Init.sql # and paste below
CREATE TABLE IF NOT EXISTS test.USER (
id VARCHAR(64),
fname VARCHAR(256),
lname VARCHAR(256),
CONSTRAINT pk PRIMARY KEY (id));
save and close
docker-compose up -d
wait for 2 minutes
docker-compose run supermonk-flyway
Ref :
https://github.com/supermonk/webapp/tree/branch-1/docker/docker-database
Thanks to docker community and mysql community
docker-compose logs -f
I'm new in docker and i'm trying to make my nodejs express run inside it.
I'm trying to install the dependencies using shellscript and its working but in the end I can't connect to mysql.
My docker file install mysql, create an user and a database, and install nodejs too.
Then it runs npm install and try to start my app but knex says it can't connect to the mysql with the message:
Knex:Error Pool2 - Error: connect ECONNREFUSED /var/run/mysqld/mysqld.sock
Here's a gist with the code i'm using. (nodejs part is incomplete, just with the important part):
https://gist.github.com/jradesenv/527f6e59ab2e7985c38fbed3a2084c83
I hope anyone will have a good ideia on how to resolve or debbug this.
The best practice is to keep the components of a micro-service separate in their own container.
See for instance "Learn Docker by building a Microservice" from Dave Kerr.
He does declare two services:
version: '2'
services:
users-service:
build: ./users-service
ports:
- "8123:8123"
depends_on:
- db
environment:
- DATABASE_HOST=db
db:
build: ./test-database
With a dedicated Dockerfile for the database:
FROM mysql:5
ENV MYSQL_ROOT_PASSWORD 123
ENV MYSQL_DATABASE users
ENV MYSQL_USER users_service
ENV MYSQL_PASSWORD 123
ADD setup.sql /docker-entrypoint-initdb.d
Docker containers are designed to run a single command. The mysql installer expects the service it registered to automatically be started on the OS bootup, but that's not the case inside of a container.
The proper solution is to split these into two separate containers, one db container, and another nodejs/app container. Link and run the two together with a docker-compose configuration that automatically sets up the host names.
The less ideal option is supervisord which you can use to run and manage multiple processes inside of the container. You install it just like any other app, configure your db and node app as two services for supervisord to manage, and then launch supervisord as your container's run command.
Use docker-compose and create a dockerfile for your nodejs and one for mysql. Each container is responsible for doing their thing. In your compose, link them. Then point your nodejs db connection to the mysql container.