newbies for airflow and mysql settings here. I am trying to set up airflow in docker-compose with MySQL as backend.
I have my mySQL connection string as sql_alchemy_conn_cmd=mysql://localuser:localpassword#mock_mysql/metastore in airflow.cfg
I have my docker-compose.yml as
version: "3.9"
networks:
airflow:
services:
redis_local:
image: redis:latest
container_name: redis_server
ports:
- 6379:6379
command: redis-server
restart: on-failure
networks:
- airflow
mysql_local:
image: mysql:5.7
container_name: mysql_local
environment:
- MYSQL_ROOT_HOST=%
- MYSQL_USER=localuser
- MYSQL_PASSWORD=localpassword
- MYSQL_ROOT_PASSWORD=localpassword
- MYSQL_DATABASE=metastore
volumes:
- ./script/init.sql:/data/application/init.sql
- ./dbdata:/var/lib/mysql
ports:
- 3306:3306
restart: on-failure
healthcheck:
test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"]
timeout: 1s
retries: 10
command:
--init-file /data/application/init.sql
--explicit_defaults_for_timestamp=1
networks:
- airflow
airflow_init_db:
depends_on:
- redis_local
- mysql_local
container_name: airflow_init_db
image: apache/airflow:2.0.0-python3.8
command: airflow initdb
since it's connecting to the metastore database, it will required to set up the database in mysql when the database is starting. i have my init.sql as
CREATE DATABASE IF NOT EXISTS metastore;
CREATE USER 'localuser'#'localhost' IDENTIFIED BY 'localpassword';
GRANT ALL PRIVILEGES ON metastore . * TO 'localuser'#'localhost';
FLUSH PRIVILEGES;
However from the healthcheck, mysql itself was failing with
mysql_local | 2021-07-06T04:29:04.011037Z 3 [Note] Access denied for user 'root'#'localhost' (using password: NO)
init db and scheduler both failed with
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2005, "Unknown MySQL server host 'mock_mysql' (-2)")
I wasn't really sure what are missing at this point.
really appreciate any help!
Thanks
Two problems.
you should use mysql_local as hostname in your connection string not mock_mysql as this is the name of your MySQL container (and this.is the name defined in DNS by docker-compose)
the healthecheck you use assumes password-less authentication, and likely in default db image of MySQL it is.disabled for security. You should pass user/password as.parameters to your healthecheck command or enable root password-less authentication
Related
I have a project with a mysql database in a container. I use docker-compose to set my project up. And I want to run the mysql command to inspect te database.
So I did, and get:
docker-compose run --rm database mysql
Creating myproject_database_run ... done
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
However when I tried this it works:
docker exec -it myproject_database_1 mysql
ERROR 1045 (28000): Access denied for user 'root'#'localhost' (using password: NO)
Can anybody explain me this?
My docker-compose file:
version: "3.7"
services:
database:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
ports:
- "127.0.0.1:3306:3306"
env_file: .env
volumes:
- type: volume
source: db_data
target: /var/lib/mysql
- type: bind
source: ./my.cnf
target: /etc/my.cnf
read_only: true
volumes:
db_data:
testing_images:
docker-compose run creates a new container. That's perfectly normal, but if your mysql client is configured to connect via a Unix socket, the new container will have a new filesystem and won't be able to see the main database container's /var/run directory.
When you use docker-compose run, you need to specify a TCP connection, using the setup described in Networking in Compose in the Docker documentation. For example,
docker-compose run --rm database \
mysql -h database
Since you publish ports: out of the container, you should be able to reach it from the host, without Docker. The trick here is that the mysql command-line client interprets localhost as a magic term to use a Unix socket and not a normal host name, so you need to specifically use the IP address 127.0.0.1 instead.
# From the same host, without anything Docker-specific
mysql -h 127.0.0.1
Try adding MYSQL_ROOT_PASSWORD in the environment.
environment:
MYSQL_ROOT_PASSWORD: root
This is from one of my working compose file
services:
## -----------------------------------------------
## MySql database
## -----------------------------------------------
db_mysql:
image: mysql:8.0
restart: always
volumes:
- db_mysql:/var/lib/mysql
- ./mysql:/docker-entrypoint-initdb.d
command: --default-authentication-plugin=mysql_native_password
environment:
MYSQL_ROOT_PASSWORD: root
networks:
- app-network
deploy:
mode: global
ports:
- "3306:3306"
## map volume
volumes:
db_mysql:
## in network, we can define any name under networks:
networks:
app-network:
driver: bridge
FYI: Official MySQL docker image - Docker Hub
Using docker compose i have a mysql 5.7 database container, i set the root password and a user password but they don't work, the docker-compose:
version: '3.8'
services:
viprs-proxy:
platform: linux/amd64
image: nginx:alpine
container_name: viprs-proxy
depends_on:
- viprs-website
volumes:
- ./nginx/proxy.conf:/etc/nginx/nginx.conf
ports:
- 80:80
networks:
- viprs-net
viprs-website:
platform: linux/amd64
image: nginx
container_name: viprs-website
depends_on:
- php
- viprs-website-database
volumes:
- ./website/nginx/site.conf:/etc/nginx/conf.d/default.conf
- ./website:/usr/share/nginx/html
- ./website/logs:/var/log/nginx
- viprs-uploads:/usr/share/nginx/html/wp-content/uploads
ports:
- 80
links:
- php
networks:
- viprs-net
php:
platform: linux/amd64
#image: php:7-fpm
image: viprs-php
container_name: php
volumes:
- ./website:/usr/share/nginx/html
ports:
- 9000
networks:
- viprs-net
viprs-website-database:
platform: linux/amd64
image: mysql:5.7
container_name: viprs-db
command: --init-file /usr/share/nginx/website.sql
volumes:
- ./website.sql:/usr/share/nginx/website.sql
- viprs-db:/var/lib/mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: testpassword
MYSLQ_DATABASE: viprs
MYSQL_USER: viprs
MYSQL_PASSWORD: testpassword
networks:
- viprs-net
networks:
viprs-net:
volumes:
viprs-uploads:
viprs-db:
Now if i log into bash:
docker exec -it viprs-db bash
and try to log into mysql:
mysql -u root -p
It just says access is denied, it doesn't matter if i am using the root user or the viprs user.
ERROR 1045 (28000): Access denied for user 'root'#'localhost' (using password: YES)
If i output $MYSQL_ROOT_PASSWORD it gives:
echo $MYSQL_ROOT_PASSWORD
testpassword
I have seen loads of people with this issue but can't find any solution that works. The environment is set as per the image documentation on docker hub so i'm confused.
In fact, i can log in as root without a password, so something isn't right.
The environment variables are only used if there is no database present when then container starts and MySQL has to create one.
If there already is a database, the users defined in that database are used and no new users are created. Since you have a volume mapping on /var/lib/mysql chances are that you already have a database.
To verify if that's the issue, you can try removing the /var/lib/mysql mapping. That will cause the container to create a new database when it starts using the environment variable values.
I'm trying to dockerize monitoring of mysql using prom/mysqld-exporter and prom/prometheus. I have configured the docker-compose.yml file like the following:
version: '3'
services:
mysql:
image: mysql
container_name: mysql
restart: always
volumes:
- mysql:/var/lib/mysql
# command:
# - CREATE USER 'root'#'localhost' IDENTIFIED BY 'password' WITH MAX_USER_CONNECTIONS 3;
# - GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'root'#'localhost';
environment:
- MYSQL_ROOT_PASSWORD= password
- MYSQL_DATABASE= db #Defining a new Database
- MYSQL_USER= mostafa
- MYSQL_PASSWORD= ghadimi
ports:
- 3306:3306
- 33060:33060
prometheus:
image: prom/prometheus
container_name: prometheus
ports:
- 9090:9090
volumes:
- prometheus:/prometheus
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- --config.file=/etc/prometheus/prometheus.yml
mysql-exporter:
image: prom/mysqld-exporter
container_name: mysql-exporter
ports:
- 9104:9104
volumes:
- ./mysql-exporter/.my.cnf:/root/.my.cnf
depends_on:
- mysql
volumes:
mysql:
prometheus:
and here is my ./mysql-exporter/.my.cnf file:
[client]
user=mostafa
password=ghadimi
After running docker-compose up command, everything works properly except the following error!:
mysqld: dial tcp 127.0.0.1:3306: connect: connection refused
I don't know how can I fix it!
PS: I have also tried to connect to mysql through its container and docker exec -it <container-id> bash command and I have attempted all the possible passwords (like empty string and the password I have set in docker-compose file), but I face with another error:
ERROR 1045 (28000): Access denied for user 'root'#'localhost' (using password: YES)
Mysql-d exporter is looking for mysql instance on localhost, but it is not running on the localhost inside mysqld container. You will have to pass external connection string with the host of mysql because this is the name of mysql service in docker-compose. From mysqld-exporter docs - you will have to pass environment variable to mysqld-exporter container with this datasource connection string :
mysql-exporter:
image: prom/mysqld-exporter
container_name: mysql-exporter
ports:
- 9104:9104
volumes:
- ./mysql-exporter/.my.cnf:/root/.my.cnf
environment:
- DATA_SOURCE_NAME="user:password#mysql:3306/database"
depends_on:
- mysql
And change user, password and database name to your desired ones.
I'm trying to connect to my mysql database using official adminer and mysql images from docker hub.
Here is my docker-compose.yml file configuration:
version: '3'
services:
mysql:
image: mysql
restart: always
volumes:
- mysql:/var/lib/mysql
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD= 1
- MYSQL_DATABASE= db
ports:
- 3306:3306
- 33060:33060
adminer:
image: adminer
restart: always
ports:
- 8080:8080
depends_on:
- mysql
volumes:
mysql:
Whenever I want to login to the MySQL using Adminer I face with the following problem:
SQLSTATE[HY000] [2054] The server requested authentication method unknown to the client
SQLSTATE[HY000] [2002] No such file or directory
Here is the inputs I've used trying to connect to MySQL from Adminer interface:
#first try
System: MySQL
Server: localhost
Username: root
Database: db
#second try
System: MySQL
Server: mysql #container-name
Username: root
Database: db
You have to add the default authentication plugin in compose file
command: --default-authentication-plugin=mysql_native_password
Here is the complete docker-compose.yml
services:
mysql:
image: mysql
restart: always
command: --default-authentication-plugin=mysql_native_password
volumes:
- mysql:/var/lib/mysql
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD= 1
- MYSQL_DATABASE= db
ports:
- 3306:3306
- 33060:33060
adminer:
image: adminer
restart: always
ports:
- 8080:8080
depends_on:
- mysql
volumes:
mysql:
In adminer
System: MySQL <DB System to connect>
Server: mysql <should match the starting tag before image tag > (in your case mysql but you can change it to any name )
Username: root <username>
Password: <password>
Database: db <database name>
I wanna deploy MySQL+PHPMyAdmin. My docker-compose.yml:
version: "3"
services:
db:
image: mysql:5.7
restart: always
container_name: db
volumes:
- ./~mysql:/var/lib/mysql
- ./mysql.cnf:/etc/mysql/conf.d/my.cnf
environment:
MYSQL_DATABASE: "dbtest"
MYSQL_ROOT_PASSWORD: "123456"
MYSQL_ROOT_HOST: "%"
networks:
- db
command: --default-authentication-plugin=mysql_native_password
healthcheck:
test: "mysqladmin ping -h localhost"
interval: 1s
timeout: 1s
retries: 60
phpmyadmin:
image: phpmyadmin/phpmyadmin:4.7
restart: always
container_name: phpmyadmin
ports:
- 8080:80
networks:
- external-net
- db
environment:
PMA_HOST: db
depends_on:
- db
networks:
external-net:
external:
name: external-net
db:
driver: bridge
After some time later I getting subject error. MYSQL_ROOT_HOST don't helped. When I trying to connect to mysql from db-container:
ERROR 1045 (28000): Access denied for user 'root'#'localhost' (using password: YES)
I really don't know what to do with this magic... Thx.
This problem might occur, if the files on your local system were created in a corrupted way or are not correctly accessible by the Docker daemon. This might be due to the following reasons:
Docker is lacking the access rights on your local hard drive, on Windows e.g. C.
While building up your containers Docker didn't have the access rights to your local hard drive. Even though Docker asks during the first --build process to allow an access to C on Windows these files might still be corrupted.
The solution could be do delete the according local files after the access to Docker has been granted, in your case these are files in /~mysql and the file mysql.cnf.
You can pass an extra environment variable when starting the MySQL container MYSQL_ROOT_HOST= this will create a root user with permission to login from given IP address. In case where you want to allow login from any IP you can specify MYSQL_ROOT_HOST=%.
This will work only for a newly created containers.
When spinning new container:
docker run --name some-mysql -e MYSQL_ROOT_HOST=% -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:latest
In compose file it would be
version: '2'
services:
### Mysql container
mysql:
image: mysql:latest
ports:
- "3306:3306"
volumes:
- /var/lib/mysql:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_DATABASE: test_db
MYSQL_USER: test
MYSQL_PASSWORD: test_pass
MYSQL_ROOT_HOST: '%' # needs to be enclosed with quotes
I've recreated your setup and was just adding some ENV configuration to do the trick, I've removed volumes section because there was no problem with it:
docker-compose.yml
version: "3"
services:
db:
image: mysql:5.7
restart: always
container_name: db
environment:
- MYSQL_ROOT_PASSWORD=rootpasswd
- MYSQL_DATABASE=phpmyadmin
- MYSQL_USER=user
- MYSQL_PASSWORD=userpasswd
networks:
- db
command: --default-authentication-plugin=mysql_native_password
healthcheck:
test: "mysqladmin ping -h localhost"
interval: 1s
timeout: 1s
retries: 60
phpmyadmin:
image: phpmyadmin/phpmyadmin:4.7
restart: always
container_name: phpmyadmin
ports:
- 8080:80
networks:
- external-net
- db
environment:
PMA_HOST: db
depends_on:
- db
networks:
external-net:
external:
name: external-net
db:
driver: bridge
Accessing PHPMyadmin with root:rootpasswd works fine.
For some reason using "~" before the volume path solves the problem for me.
volumes:
- ./~mysql:/var/lib/mysql
- ./mysql.cnf:/etc/mysql/conf.d/my.cnf
Change this code to
volumes:
- ~/mysql:/var/lib/mysql
for MySQL container.
In Genetal settings of Docker enable daemon without TLS. I think It works.
Docker image