I have the following docker-compose file:
version: '3.7'
services:
web:
build: ./project
command: uvicorn app.main:app --reload --workers 1 --host 0.0.0.0 --port 8000
volumes:
- ./project:/usr/src/app
ports:
- 8002:8000
environment:
- ENVIRONMENT=dev
- TESTING=0
- DATABASE_URL=postgres://postgres:postgres#web-db:5432/web_dev # new
- DATABASE_TEST_URL=postgres://postgres:postgres#web-db:5432/web_test # new
depends_on: # new
- web-db
Within ./project there is a .dockerignore that is used to ignore files within ./project. However these files are not being ignored. Does the volume take precedent and ignores .dockerignore? ./project:/usr/src/app.
How can I prevent files within ./project from being mounted onto the image?
Thanks!
.dockerignore does not work like that: it is mean for your build process, so that it does not consider certain files as the build context of your image, and are not used in your Dockerfile. Volumes will always include all files in the directory, hidden ones too.
Related
I'm trying to dockerize an existing adonis.js app and MySQL through docker-compose.
Here is my Dockerfile
FROM node:12.18.2-alpine3.9
ENV HOME=/app
RUN mkdir /app
COPY package.json $HOME
WORKDIR $HOME
RUN npm i -g #adonisjs/cli && npm install
CMD ["npm", "start"]
And here is my docker-compose.yml file
version: '3'
services:
adonis-mysql:
image: mysql:5.7
ports:
- '3307:3306'
volumes:
- $PWD/data:/var/lib/mysql
environment:
MYSQL_USER: ${DB_USER}
MYSQL_DATABASE: ${DB_DATABASE}
MYSQL_PASSWORD: ${DB_PASSWORD}
MYSQL_ALLOW_EMPTY_PASSWORD: ${DB_ALLOW_EMPTY_PASSWORD}
networks:
- api-network
adonis-api:
container_name: "${APP_NAME}-api"
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/app
- /app/node_modules
ports:
- "3333:3333"
depends_on:
- adonis-mysql
networks:
- api-network
networks:
api-network:
When running docker-compose up everything goes smoothly and the adonis-api container says that the app is running but I'm am unable to reach it, I always get:
This site can’t be reached
127.0.0.1 refused to connect.
or
This site can’t be reached
The connection was reset.
I tried with different docker-compose settings, and different dockerfiles, almost always everything starts ok but I'm just unable to access the server.
Also tried different IP and ports, but still nothing.
Container logs:
testProject-api |
testProject-api | > adonis-fullstack-app#4.1.0 start /app
testProject-api | > node server.js
testProject-api |
adonis-mysql_1 | 2020-07-09T09:56:35.960082Z 1 [Warning] root#localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
testProject-api | info: serving app on http://127.0.0.1:80
docker ps
dan#dan-Nitro-AN515-54:~/Documents/Tests/testProject$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
45f3dd21ef93 testproject_adonis-api "docker-entrypoint.s…" 20 seconds ago Up 19 seconds 0.0.0.0:3333->3333/tcp testProject-api
7b40bc7c75c3 mysql:5.7 "docker-entrypoint.s…" 2 minutes ago Up 20 seconds 33060/tcp, 0.0.0.0:3307->3306/tcp testproject_adonis-mysql_1
There's two things that jump out in this setup.
First of all, when the container startup prints:
info: serving app on http://127.0.0.1:80
That's usually an indication of a configuration issue that will make the process inaccessible. In Docker each container has its own localhost interface, so a process that's "listening on 127.0.0.1" will only be reachable from the container-private localhost interface, but not from other containers or the host (regardless of what ports: options you have). You generally need to set processes to "bind" or "listen" to the special 0.0.0.0 all-interfaces address.
Within Adonis it looks like this is controlled by the $HOST environment variable; the Adonis templates set this to 127.0.0.1. Adonis documents itself as using the dotenv library, and that in turn gives precedence to environment variables over the .env file, so it should be enough to set an environment variable HOST=0.0.0.0.
(None of the previous paragraph is discussed in the Adonis documentation!)
The second thing from that error message is that the second number in ports: needs to match the port number the container process is using. The Adonis templates all seem to default this to port 3333 but that startup message says port 80, so you need to change your ports: to be port 80 on the right-hand side. You can pick any port you want for the left-hand side.
Adding in some routine cleanups, you could replace your docker-compose.yml service block with:
adonis-api:
build: . # context directory only; use default Dockerfile
environment:
- HOST=0.0.0.0 # listen on all interfaces
ports:
- "3333:80" # matches actual listener message
depends_on:
- adonis-mysql
# Use "default" network (also delete other networks: blocks in the file)
# Use Compose default container name
# Use code from the Docker image; don't overwrite with volumes
# (and don't tell Docker to use arbitrarily old node_modules)
I have a requirement where I need to wait for a few commands before I seed the data for the database:
I have some Migration scripts that create the schema in the database (this command runs from my app container). After this executes, I want to seed data to the database.
As I read, the docker-entrypoint-initdb scripts is executed when the container is initialized. If I mount my seed.sql script to it, the data is seeded before the Migrate scripts. (The Migrate scripts actually drop all tables and create them from scratch). The seeded data is therefore lost.
How can I achieve this? (I cannot change the Migrate scripts)
Here's my docker-compose.yml file
version: '3'
services:
app:
build: .
# mount the current directory (on the host) to /usr/src/app on the container, any changes in either would be reflected in both the host and the container
volumes:
- .:/usr/src/app
# expose application on localhost:36081
ports:
- "36081:36081"
# application restarts if stops for any reason - required for the container to restart when the application fails to start due to the database containers not being ready
restart: always
environment:
MIGRATE: Y
<some env variables here>
config-dev:
image: mysql/mysql-server:5.7
environment:
MYSQL_DATABASE: config_dev
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
volumes:
# to persist data
- config-dev-volume:/var/lib/mysql
restart: always
# to connect locally from SequelPro
ports:
- "1200:3306"
<other database containers>
My Dockerfile for app container has the following ENTRYPOINT
# start the application
ENTRYPOINT /usr/src/app/docker-entrypoint.sh
Here's the docker-entrypoint.sh file
#!/bin/bash
if [ "$MIGRATE" = "Y" ];
then
<command to start migration scripts>
echo "------------starting application--------------"
<command to start application>
else
echo "------------starting application--------------"
<command to start application>
fi
Edit: Is there a way I can run a script in config-db container from the docker-entrypoint.sh file in app container?
This can be solved in two steps:
You need to wait until your db container is started and is ready.
Wait until started can be handled by adding depends_on in docker-compose file:
version: '3'
services:
app:
build: .
# mount the current directory (on the host) to /usr/src/app on the container, any changes in either would be reflected in both the host and the container
depends_on:
- config-dev
- <other containers (if any)>
volumes:
- .:/usr/src/app
# expose application on localhost:36081
ports:
- "36081:36081"
# application restarts if stops for any reason - required for the container to restart when the application fails to start due to the database containers not being ready
restart: always
environment:
MIGRATE: Y
<some env variables here>
config-dev:
image: mysql/mysql-server:5.7
environment:
MYSQL_DATABASE: config_dev
MYSQL_ALLOW_EMPTY_PASSWORD: 'yes'
volumes:
# to persist data
- config-dev-volume:/var/lib/mysql
restart: always
# to connect locally from SequelPro
ports:
- "1200:3306"
<other database containers>
Wait until db is ready is another case because sometimes it takes time for the db process to start listening on the tcp port.
Unfortunately, Docker does not provide a way to hook onto container state. There are many tools and scripts to have a workaround this.
You can go through this to implement the workaround.
https://docs.docker.com/compose/startup-order/
TL;DR
Download https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh inside the container and delete the ENTRYPOINT field (Not required for your use case) and use CMD field instead:
CMD ["./wait-for-it.sh", "<db_service_name_as_per_compose_file>:<port>", "--", "/usr/src/app/docker-entrypoint.sh"]
Now, That this is complete. Next part is to execute your seed.sql script.
That is easy and can be executed by adding following line into your /usr/src/app/docker-entrypoint.sh script.
sqlcmd -S -U -P -i inputquery_file_name -o outputfile_name
Place above command after migrate script in /usr/src/app/docker-entrypoint.sh
I have a docker-compose.yml:
```
version: '2'
volumes:
db-data:
vendor:
var:
services:
mysql:
container_name: test-mysql-5.7
image: mysql:5.7
ports:
- '3306:3306'
volumes:
- db-data:/var/lib/mysql
- /home/ubuntu/www/prestashop-shops/test-ps/seeds:/docker-entrypoint-initdb.d
environment:
MYSQL_ROOT_PASSWORD: test123
restart: always
```
I run docker-compose up -d and it will bring up the container with the associated volumes. I have been playing with this and needed to tear it down to test the docker-compose.yml again.
I teardown by running docker-compose rm -fsv in the folder, which works in bringing down the container and some of the volumes.
I then run docker volume ls | grep test- | awk '{print $2}' | xargs docker volume rm the test_db-data, test_var, test_vendor volumes also created by the compose file.
I then run docker-compose up -d again, but when I go into the mysql instance and check the databases with a SHOW DATABASES; command. All the databases that were created before are still there.
I've checked the filesystem /var/lib/docker/volumes/ to see if there are any physical files remaining for this and there are none.
I've tried renaming the db-data part of my compose file to something else, and it's still there.
I think this may be cached in the image but I don't remember building anything.
Wondering if volume data persists anywhere or is cached anywhere because I cannot get a clean instance of mysql deployed on my server now.
I'm new in docker, so cant understand - if I want to build container of mysql/postgresql/clickhouse etc - how to create database and schema of database/table? Maybe in Dockerfile or i can do it from docker-compose.yml?
I mean, that I dont know when and where to use CREATE DATABASE; CREATE TABLE ...; queries if I use docker containers of popular databases
You can use both docker and docker-compose. For example with docker compose.
Create a file called docker-compose.yml like:
version: '3'
services:
db:
image: percona:5.7
container_name: whatever_you_want
environment:
- MYSQL_DATABASE=${DATABASE}
- MYSQL_ROOT_PASSWORD=${ROOT_PASSWORD}
- MYSQL_USER=${USER}
- MYSQL_PASSWORD=${PASSWORD}
volumes:
- ./data:/docker-entrypoint-initdb.d
ports:
- "3306:3306"
Additionally you need a file under ./data with whatever SQL commands you want to run and and .env file where you define the environmental variables I used in the docker-compose.yml file above like: ${DATABASE}
Your .env file:
# MySQL
DATABASE=db_name_here
ROOT_USER=root
ROOT_PASSWORD=root
USER=dev
PASSWORD=dev
Your file with SQL commands to execute ./data/init.sql (you can name the file whatever you want)
CREATE DATABASE 'whatever';
DROP DATABASE 'whatever';
-- you can do whatever you want here
This file will be executed each time you do:
docker-compose up -d db
At first you need to create docker a image for your db server, or use an already existing image.
Bellow is an example of mysql docker image.
version: "3"
services:
****************
mysql:
container_name: mysql
image: mysql:5.7
restart: on-failure
environment:
- MYSQL_DATABASE=YOUR_DB_NAME
- MYSQL_ROOT_PASSWORD=YOUR_ROOT_USER_PASSWORD
- MYSQL_USER=YOUR_USER
- MYSQL_PASSWORD=YOUR_USER_PASSWORD
ports:
- "33060:3306"
volumes:
- "./data/db/mysql:/var/lib/mysql"
Let's describe some sections:
volumes:
- "./data/db/mysql:/var/lib/mysql"
This is like "mounting" container's /var/lib/mysql to system's ./data/db/mysql. So your data will be on your system drive, because in debian the default path to MySQL data is /var/lib/mysql.
ports:
- "33060:3306"
This will map port 3306 from container to system's 33060 port, to avoid conflicts if you have installed MySQL server on system as well.
environment:
- MYSQL_DATABASE=YOUR_DB_NAME
- MYSQL_ROOT_PASSWORD=YOUR_ROOT_USER_PASSWORD
- MYSQL_USER=YOUR_USER
- MYSQL_PASSWORD=YOUR_USER_PASSWORD
This will create a database with the defined parameters: name, root password, ..., or if a database already exists it will try to access with the defined credentials. Functionality to check/create database is already defined in the image.
If you want to define your own functionality you can define your image (e.g. dockerfile: ./Dockerfile instead of image: mysql:5.7). Dockerfile can be something like this:
FROM mysql:5.7
ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_ROOT_PASSWORD
ENV MYSQL_DATABASE=${MYSQL_DATABASE}
ENV MYSQL_USER=${MYSQL_USER}
ENV MYSQL_PASSWORD=${MYSQL_PASSWORD}
ENV MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
# copy predefined config file
COPY configs/default.cnf /etc/mysql/conf.d/
# To be sure that MySQL will not ignore configs
RUN chmod og-w /etc/mysql/conf.d/default.cnf
# DO SOMETHING ELSE YOU WANT
EXPOSE 3306
CMD ["mysqld"]
So you can build and up your container with command docker-compose up -d --build
Here is an example I used to initialise SQL Server 2017 database using container.
https://www.handsonarchitect.com/2018/01/build-custom-sql-server-2017-linux.html
The trick is to use a shell script to run which will invoke the database initialisation script. You might have to wait for few seconds for the database engine service to start before executing the initialisation script.
Hello I have a configuration that builds docker containers for a flask app and a mysql instance.
I create a new VM with
docker-machine create -d amazonec2 --....... production
and then (after setting the correct environment)
docker-compose build -> docker-compose up -d
The problem is that all these happen whilst CWD is a local repo with the files I need. It turns out these files are not copied over.
I have looked at docker cp and docker scp but it seems they do not solve the problem. E.g. with SCP I cannot reference the specific machine I need to copy the repo over (xow_web_1)
Here is the .yml
web:
restart: always
volumes:
- .:/xow
build: .
ports:
- "80:80"
links:
- db
hostname: xowflask
command: python xow.py
db:
restart: always
hostname: xowmysql
image: mysql:latest
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: somepasswordhere
MYSQL_DATABASE: somedatabase
data:
restart: always
image: mysql:latest
volumes:
- /var/lib/mysql
command: "true"
How would be the most appropriate way to solve this? Is docker-compose the right approach? Looks awesome, but it doesn't solve an issue like this
The way we solved it in our organization is by using the COPY command to copy all of the data in the folder to the container.
For example, copying all of the files from the current dir to the container /src folder will look like this -
### Copy Code
COPY . /src
It looks like you should add this line into the web container in your docker-compose configuration.