docker compose mysql works differently on remote server than on windows docker desktop - access denied for user 'root'#'172.18.0.2' - mysql

The bounty expires in 4 days. Answers to this question are eligible for a +50 reputation bounty.
Lou K wants to draw more attention to this question.
I have a docker compose set of images with one of them based on mysql. The stack was adapted from https://github.com/docker/awesome-compose/tree/c2f8036fd353dae457eba7b9b436bf3a1c85d937/nginx-flask-mysql
This runs fine on my ("local") Windows 10 docker desktop (docker v20.10.22), but when I try to run it on my ("remote") centos7 server (Docker version 23.0.0, build e92dd87, remote server hosted at digitalocean), I see Access denied for user 'root'#'172.18.0.3' (using password: YES) when trying to connect with the database.
I should note I've seen answers like https://stackoverflow.com/a/59839180/799921, and I have (repeatedly) tried removing the volume on the remote server.
This failure happens whether I use root and MYSQL_ROOT_PASSWORD or MYSQL_USER (user) and MYSQL_PASSWORD. I've verified I can connect to the database from the mysql container, but the app container does not have mysql installed so I haven't been able to manually test the connection from the app container.
Following https://hub.docker.com/_/mysql, "Connect to MySQL from the MySQL command line client", I'm able to connect, so this is a problem with the app according to https://dev.mysql.com/doc/refman/8.0/en/problems-connecting.html, "If you have access problems with a Perl, PHP, Python, or ODBC program, try to connect to the server with mysql -u user_name db_name or mysql -u user_name -ppassword db_name. If you are able to connect using the mysql client, the problem lies with your program", but I'm not sure what.
$ docker run -it --network webmodules_backend-network --rm mysql mysql -hdb -uroot -p
Enter password:
Welcome to the MySQL monitor.
...
I've looked at the mysql.user table on both running versions and nothing jumps out at me as being incorrect.
mysql.user on local:
mysql> select * from mysql.user where user='root';
+-----------+------+-------------+-------------+-------------+-------------+-------------+-----------+-------------+---------------+--------------+-----------+------------+-----------------+------------+------------+--------------+------------+-----------------------+------------------+--------------+-----------------+------------------+------------------+----------------+---------------------+--------------------+------------------+------------+--------------+------------------------+----------+------------------------+--------------------------+----------------------------+---------------+-------------+-----------------+----------------------+-----------------------+-------------------------------------------+------------------+-----------------------+-------------------+----------------+------------------+----------------+------------------------+---------------------+--------------------------+-----------------+
| Host | User | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Reload_priv | Shutdown_priv | Process_priv | File_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Repl_slave_priv | Repl_client_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Create_user_priv | Event_priv | Trigger_priv | Create_tablespace_priv | ssl_type | ssl_cipher | x509_issuer | x509_subject | max_questions | max_updates | max_connections | max_user_connections | plugin | authentication_string | password_expired | password_last_changed | password_lifetime | account_locked | Create_role_priv | Drop_role_priv | Password_reuse_history | Password_reuse_time | Password_require_current | User_attributes |
+-----------+------+-------------+-------------+-------------+-------------+-------------+-----------+-------------+---------------+--------------+-----------+------------+-----------------+------------+------------+--------------+------------+-----------------------+------------------+--------------+-----------------+------------------+------------------+----------------+---------------------+--------------------+------------------+------------+--------------+------------------------+----------+------------------------+--------------------------+----------------------------+---------------+-------------+-----------------+----------------------+-----------------------+-------------------------------------------+------------------+-----------------------+-------------------+----------------+------------------+----------------+------------------------+---------------------+--------------------------+-----------------+
| % | root | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | 0x | 0x | 0x | 0 | 0 | 0 | 0 | mysql_native_password | *52594A4243313A7447185F38CB9D3859DDC5FF77 | N | 2023-02-14 21:20:23 | NULL | N | Y | Y | NULL | NULL | NULL | NULL |
| localhost | root | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | 0x | 0x | 0x | 0 | 0 | 0 | 0 | mysql_native_password | *52594A4243313A7447185F38CB9D3859DDC5FF77 | N | 2023-02-14 21:20:23 | NULL | N | Y | Y | NULL | NULL | NULL | NULL |
+-----------+------+-------------+-------------+-------------+-------------+-------------+-----------+-------------+---------------+--------------+-----------+------------+-----------------+------------+------------+--------------+------------+-----------------------+------------------+--------------+-----------------+------------------+------------------+----------------+---------------------+--------------------+------------------+------------+--------------+------------------------+----------+------------------------+--------------------------+----------------------------+---------------+-------------+-----------------+----------------------+-----------------------+-------------------------------------------+------------------+-----------------------+-------------------+----------------+------------------+----------------+------------------------+---------------------+--------------------------+-----------------+
2 rows in set (0.00 sec)
mysql.user on remote:
mysql> select * from mysql.user where user='root';
+-----------+------+-------------+-------------+-------------+-------------+-------------+-----------+-------------+---------------+--------------+-----------+------------+-----------------+------------+------------+--------------+------------+-----------------------+------------------+--------------+-----------------+------------------+------------------+----------------+---------------------+--------------------+------------------+------------+--------------+------------------------+----------+------------------------+--------------------------+----------------------------+---------------+-------------+-----------------+----------------------+-----------------------+-------------------------------------------+------------------+-----------------------+-------------------+----------------+------------------+----------------+------------------------+---------------------+--------------------------+-----------------+
| Host | User | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Reload_priv | Shutdown_priv | Process_priv | File_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Repl_slave_priv | Repl_client_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Create_user_priv | Event_priv | Trigger_priv | Create_tablespace_priv | ssl_type | ssl_cipher | x509_issuer | x509_subject | max_questions | max_updates | max_connections | max_user_connections | plugin | authentication_string | password_expired | password_last_changed | password_lifetime | account_locked | Create_role_priv | Drop_role_priv | Password_reuse_history | Password_reuse_time | Password_require_current | User_attributes |
+-----------+------+-------------+-------------+-------------+-------------+-------------+-----------+-------------+---------------+--------------+-----------+------------+-----------------+------------+------------+--------------+------------+-----------------------+------------------+--------------+-----------------+------------------+------------------+----------------+---------------------+--------------------+------------------+------------+--------------+------------------------+----------+------------------------+--------------------------+----------------------------+---------------+-------------+-----------------+----------------------+-----------------------+-------------------------------------------+------------------+-----------------------+-------------------+----------------+------------------+----------------+------------------------+---------------------+--------------------------+-----------------+
| % | root | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | 0x | 0x | 0x | 0 | 0 | 0 | 0 | mysql_native_password | *CFBC0A14FD2027A55F04E2A65FAF93B5D528800B | N | 2023-02-14 21:22:12 | NULL | N | Y | Y | NULL | NULL | NULL | NULL |
| localhost | root | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | | 0x | 0x | 0x | 0 | 0 | 0 | 0 | mysql_native_password | *CFBC0A14FD2027A55F04E2A65FAF93B5D528800B | N | 2023-02-14 21:22:12 | NULL | N | Y | Y | NULL | NULL | NULL | NULL |
+-----------+------+-------------+-------------+-------------+-------------+-------------+-----------+-------------+---------------+--------------+-----------+------------+-----------------+------------+------------+--------------+------------+-----------------------+------------------+--------------+-----------------+------------------+------------------+----------------+---------------------+--------------------+------------------+------------+--------------+------------------------+----------+------------------------+--------------------------+----------------------------+---------------+-------------+-----------------+----------------------+-----------------------+-------------------------------------------+------------------+-----------------------+-------------------+----------------+------------------+----------------+------------------------+---------------------+--------------------------+-----------------+
2 rows in set (0.01 sec)
This is started with the following on local:
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d
and on remote (from the local machine):
docker --context webmodules compose -f docker-compose.yml -f docker-compose-prod.yml up -d
where
docker-compose.yml:
version: "3.8"
services:
db:
# https://github.com/docker-library/mysql/issues/275#issuecomment-636831964
image: mysql:8.0.32 # 32 gives access denied on centos7 server for both root and user
# command: '--default-authentication-plugin=mysql_native_password'
command: '--default-authentication-plugin=mysql_native_password --log_error_verbosity=3' # mysql
# restart: always
secrets:
- db-password
- user-password
volumes:
- db-data:/var/lib/mysql
networks:
- backend-network
environment:
- MYSQL_DATABASE=webmodules
- MYSQL_ROOT_PASSWORD_FILE=/run/secrets/db-password
- MYSQL_USER=user
- MYSQL_PASSWORD_FILE=/run/secrets/user-password
app:
build: app
restart: always
secrets:
- db-password
- user-password
networks:
- backend-network
- frontend-network
web:
build: web
restart: always
ports:
- 8000:80
networks:
- frontend-network
volumes:
db-data:
secrets:
db-password:
file: db/password.txt
user-password:
file: db/userpassword.txt
networks:
backend-network:
frontend-network:
docker-compose.dev.yml:
version: '3.8'
services:
app:
ports:
- 5678:5678
volumes:
- ./app/src:/app
environment:
- FLASK_DEBUG=True
docker-compose-prod.yml:
version: '3.8'
secrets:
db-password:
file: /home/appuser/.docker/webmodules-db-password.txt
user-password:
file: /home/appuser/.docker/webmodules-user-password.txt
app/Dockerfile:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.9-slim
EXPOSE 5000
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# set the working directory in the container
WORKDIR /app
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
# copy the content of the local src directory to the working directory
# this isn't needed when developing as there's a bind under volumes: in the docker-compose.dev.yml file
COPY src .
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["gunicorn", "--reload", "--bind", "0.0.0.0:5000", "app:app"]
The full log file from the mysql container on remote:
$ docker logs webmodules-db-1
2023-02-15 12:35:06+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
2023-02-15 12:35:07+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-02-15 12:35:07+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2023-02-15T12:35:08.185441Z 0 [Note] [MY-013667] [Server] Error-log destination "stderr" is not a file. Can not restore error log messages from previous run.
2023-02-15T12:35:08.174478Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2023-02-15T12:35:08.182644Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2023-02-15T12:35:08.182660Z 0 [Note] [MY-013932] [Server] BuildID[sha1]=6b049f17400f850658b2eb3ff165ec9a085d9655
2023-02-15T12:35:08.182673Z 0 [Note] [MY-010949] [Server] Basedir set to /usr/.
2023-02-15T12:35:08.182694Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.32) starting as process 1
2023-02-15T12:35:08.195754Z 0 [Note] [MY-012366] [InnoDB] Using Linux native AIO
2023-02-15T12:35:08.195970Z 0 [Note] [MY-010747] [Server] Plugin 'FEDERATED' is disabled.
2023-02-15T12:35:08.196046Z 0 [Note] [MY-010747] [Server] Plugin 'ndbcluster' is disabled.
2023-02-15T12:35:08.196062Z 0 [Note] [MY-010747] [Server] Plugin 'ndbinfo' is disabled.
2023-02-15T12:35:08.196069Z 0 [Note] [MY-010747] [Server] Plugin 'ndb_transid_mysql_connection_map' is disabled.
2023-02-15T12:35:08.197842Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-02-15T12:35:08.197887Z 1 [Note] [MY-013546] [InnoDB] Atomic write enabled
2023-02-15T12:35:08.197923Z 1 [Note] [MY-012932] [InnoDB] PUNCH HOLE support available
2023-02-15T12:35:08.197942Z 1 [Note] [MY-012944] [InnoDB] Uses event mutexes
2023-02-15T12:35:08.197948Z 1 [Note] [MY-012945] [InnoDB] GCC builtin __atomic_thread_fence() is used for memory barrier
2023-02-15T12:35:08.197956Z 1 [Note] [MY-012948] [InnoDB] Compressed tables use zlib 1.2.13
2023-02-15T12:35:08.206838Z 1 [Note] [MY-012951] [InnoDB] Using hardware accelerated crc32 and polynomial multiplication.
2023-02-15T12:35:08.207494Z 1 [Note] [MY-012203] [InnoDB] Directories to scan './'
2023-02-15T12:35:08.207577Z 1 [Note] [MY-012204] [InnoDB] Scanning './'
2023-02-15T12:35:08.212659Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 4 files.
2023-02-15T12:35:08.213682Z 1 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 128.000000M, instances = 1, chunk size =128.000000M
2023-02-15T12:35:08.228723Z 1 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2023-02-15T12:35:08.385456Z 0 [Note] [MY-011952] [InnoDB] If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2023-02-15T12:35:08.386138Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2023-02-15T12:35:08.386668Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2023-02-15T12:35:08.432730Z 1 [Note] [MY-013566] [InnoDB] Double write buffer files: 2
2023-02-15T12:35:08.432776Z 1 [Note] [MY-013565] [InnoDB] Double write buffer pages per instance: 4
2023-02-15T12:35:08.432817Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_0.dblwr' for doublewrite
2023-02-15T12:35:08.432849Z 1 [Note] [MY-013532] [InnoDB] Using './#ib_16384_1.dblwr' for doublewrite
2023-02-15T12:35:08.531818Z 1 [Note] [MY-013883] [InnoDB] The latest found checkpoint is at lsn = 31919058 in redo log file ./#innodb_redo/#ib_redo9.
2023-02-15T12:35:08.532211Z 1 [Note] [MY-013086] [InnoDB] Starting to parse redo log at lsn = 31918620, whereas checkpoint_lsn = 31919058 and start_lsn = 31918592
2023-02-15T12:35:08.585369Z 1 [Note] [MY-013083] [InnoDB] Log background threads are being started...
2023-02-15T12:35:08.760113Z 1 [Note] [MY-012532] [InnoDB] Applying a batch of 0 redo log records ...
2023-02-15T12:35:08.760147Z 1 [Note] [MY-012535] [InnoDB] Apply batch completed!
2023-02-15T12:35:08.760387Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'.
2023-02-15T12:35:08.763735Z 1 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'.
2023-02-15T12:35:08.768232Z 1 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.
2023-02-15T12:35:08.768318Z 1 [Note] [MY-011980] [InnoDB] GTID recovery trx_no: 2832
2023-02-15T12:35:08.788140Z 1 [Note] [MY-013777] [InnoDB] Time taken to initialize rseg using 1 thread: 19811 ms.
2023-02-15T12:35:08.788273Z 1 [Note] [MY-012923] [InnoDB] Creating shared tablespace for temporary tables
2023-02-15T12:35:08.788341Z 1 [Note] [MY-012265] [InnoDB] Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-02-15T12:35:08.835413Z 1 [Note] [MY-012266] [InnoDB] File './ibtmp1' size is now 12 MB.
2023-02-15T12:35:08.835602Z 1 [Note] [MY-013627] [InnoDB] Scanning temp tablespace dir:'./#innodb_temp/'
2023-02-15T12:35:09.003141Z 1 [Note] [MY-013018] [InnoDB] Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.
2023-02-15T12:35:09.037574Z 1 [Note] [MY-012976] [InnoDB] 8.0.32 started; log sequence number 31919068
2023-02-15T12:35:09.038169Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-02-15T12:35:09.046561Z 1 [Note] [MY-011089] [Server] Data dictionary restarting version '80023'.
2023-02-15T12:35:09.184823Z 1 [Note] [MY-012357] [InnoDB] Reading DD tablespace files
2023-02-15T12:35:09.185738Z 1 [Note] [MY-012356] [InnoDB] Scanned 6 tablespaces. Validated 6.
2023-02-15T12:35:09.246031Z 1 [Note] [MY-010006] [Server] Using data dictionary with version '80023'.
2023-02-15T12:35:09.252593Z 0 [Note] [MY-011332] [Server] Plugin mysqlx reported: 'IPv6 is available'
2023-02-15T12:35:09.254590Z 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. bind-address: '::' port: 33060'
2023-02-15T12:35:09.254626Z 0 [Note] [MY-011323] [Server] Plugin mysqlx reported: 'X Plugin ready for connections. socket: '/var/run/mysqld/mysqlx.sock''
2023-02-15T12:35:09.278358Z 0 [Note] [MY-010902] [Server] Thread priority attribute setting in Resource Group SQL shall be ignored due to unsupported platform or insufficient privilege.
2023-02-15T12:35:09.310954Z 0 [Note] [MY-013911] [Server] Crash recovery finished in binlog engine. No attempts to commit, rollback or prepare any transactions.
2023-02-15T12:35:09.311015Z 0 [Note] [MY-013911] [Server] Crash recovery finished in InnoDB engine. No attempts to commit, rollback or prepare any transactions.
2023-02-15T12:35:09.316319Z 0 [Note] [MY-012487] [InnoDB] DDL log recovery : begin
2023-02-15T12:35:09.316414Z 0 [Note] [MY-012488] [InnoDB] DDL log recovery : end
2023-02-15T12:35:09.322395Z 0 [Note] [MY-011946] [InnoDB] Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2023-02-15T12:35:09.322785Z 0 [Note] [MY-011946] [InnoDB] Buffer pool(s) load completed at 230215 12:35:09
2023-02-15T12:35:09.433353Z 0 [Note] [MY-010913] [Server] You have not provided a mandatory server-id. Servers in a replication topology must have unique server-ids. Please refer to the proper server start-up parameters documentation.
2023-02-15T12:35:09.435029Z 0 [Note] [MY-010182] [Server] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2023-02-15T12:35:09.435065Z 0 [Note] [MY-010304] [Server] Skipping generation of SSL certificates as certificate files are present in data directory.
2023-02-15T12:35:09.438539Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2023-02-15T12:35:09.438584Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2023-02-15T12:35:09.438630Z 0 [Note] [MY-010308] [Server] Skipping generation of RSA key pair through --sha256_password_auto_generate_rsa_keys as key files are present in data directory.
2023-02-15T12:35:09.438643Z 0 [Note] [MY-010308] [Server] Skipping generation of RSA key pair through --caching_sha2_password_auto_generate_rsa_keys as key files are present in data directory.
2023-02-15T12:35:09.438802Z 0 [Note] [MY-010252] [Server] Server hostname (bind-address): '*'; port: 3306
2023-02-15T12:35:09.438849Z 0 [Note] [MY-010253] [Server] IPv6 is available.
2023-02-15T12:35:09.438857Z 0 [Note] [MY-010264] [Server] - '::' resolves to '::';
2023-02-15T12:35:09.438885Z 0 [Note] [MY-010251] [Server] Server socket created on IP: '::'.
2023-02-15T12:35:09.440185Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2023-02-15T12:35:09.461816Z 0 [Note] [MY-011025] [Repl] Failed to start slave threads for channel ''.
2023-02-15T12:35:09.463840Z 5 [Note] [MY-010051] [Server] Event Scheduler: scheduler thread started with id 5
2023-02-15T12:35:09.464101Z 0 [Note] [MY-011240] [Server] Plugin mysqlx reported: 'Using SSL configuration from MySQL Server'
2023-02-15T12:35:09.464743Z 0 [Note] [MY-011243] [Server] Plugin mysqlx reported: 'Using OpenSSL for TLS connections'
2023-02-15T12:35:09.464917Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.32' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL.
2023-02-15T12:35:09.464973Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2023-02-15T12:36:06.574103Z 8 [Note] [MY-010926] [Server] Access denied for user 'root'#'172.18.0.2' (using password: YES)
And not sure if this is relevant, but the python code is in app.py:
import mysql.connector
from flask import Flask, jsonify
from ptvsd import enable_attach
# enable python visual studio debugger
enable_attach(address=('0.0.0.0', 5678))
app = Flask(__name__)
conn = None
# adapted from https://github.com/aiordache/demos/blob/c7aa37cc3e2f8800296f668138b4cf208b27380a/dockercon2020-demo/app/src/server.py
# similar to https://github.com/docker/awesome-compose/blob/e6b1d2755f2f72a363fc346e52dce10cace846c8/nginx-flask-mysql/backend/hello.py
class DBManager:
def __init__(self, database='example', host="db", user="root", password_file=None):
pf = open(password_file, 'r')
self.connection = mysql.connector.connect(
user=user,
password=pf.read(),
host=host, # name of the mysql service as set in the docker compose file
database=database,
auth_plugin='mysql_native_password'
)
pf.close()
self.cursor = self.connection.cursor()
def populate_db(self):
self.cursor.execute('DROP TABLE IF EXISTS blog')
self.cursor.execute('CREATE TABLE blog (id INT AUTO_INCREMENT PRIMARY KEY, title VARCHAR(255))')
self.cursor.executemany('INSERT INTO blog (id, title) VALUES (%s, %s);', [(i, 'Blog post #%d'% i) for i in range (1,5)])
self.connection.commit()
def query_titles(self):
self.cursor.execute('SELECT title FROM blog')
rec = []
for c in self.cursor:
rec.append(c[0])
return rec
#app.route('/')
def hello_world():
return 'Hello, Docker!'
#app.route('/blogs')
def listBlog():
global conn
if not conn:
conn = DBManager(host='db', database='webmodules', user='root', password_file='/run/secrets/db-password')
conn.populate_db()
rec = conn.query_titles()
result = []
for c in rec:
result.append(c)
return jsonify({"response": result})
if __name__ == "__main__":
app.run(host ='0.0.0.0', port=5000)
my.cnf (local and remote identical):
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/8.0/en/server-configuration-defaults.html
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
# Remove leading # to revert to previous value for default_authentication_plugin,
# this will increase compatibility with older clients. For background, see:
# https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_default_authentication_plugin
# default-authentication-plugin=mysql_native_password
skip-host-cache
skip-name-resolve
datadir=/var/lib/mysql
socket=/var/run/mysqld/mysqld.sock
secure-file-priv=/var/lib/mysql-files
user=mysql
pid-file=/var/run/mysqld/mysqld.pid
[client]
socket=/var/run/mysqld/mysqld.sock
!includedir /etc/mysql/conf.d/

You should replace the following line in app.py
password=pf.read(),
by this line:
password=pf.read().strip(),
The reason is that python will also read the newline at the end of the file as part of the password string, and mysql will cut that off.
On Windows you might have used an editor which doesn't place a newline at the end of the file or it might have added a carriage return whereas on Linux the line ending is just a linefeed character.
Alternatively you can make sure you password file doesn't contain a newline at the end, but most editors will not allow this.
Using the 'echo -n' command you can achieve this:
> echo -n "bla" >password_no_lf.txt
> od -c < password_no_lf.txt
0000000 b l a
0000003
> echo "bla" >password_with_lf.txt
> od -c < password_with_lf.txt
0000000 b l a \n
0000004

Related

Metadata lock on MySQL 5.7, can't find locking process?

So I have a python server with a (currently asleep) connection to the MySQL database. I go to phpMyAdmin and just try to truncate the "tools" table, which is part of the "organize" database. But it doesn't work. Problem is, I can't seem to find which query is actually locking it.
mysql> show full processlist;
+-----+------+-----------+----------+---------+------+---------------------------------+------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+------+-----------+----------+---------+------+---------------------------------+------------------------+
| 175 | user | localhost | organize | Sleep | 1235 | | NULL |
| 244 | user | localhost | NULL | Query | 0 | starting | show full processlist |
| 307 | user | localhost | organize | Query | 272 | Waiting for table metadata lock | TRUNCATE TABLE `tools` |
| 308 | user | localhost | NULL | Sleep | 272 | | NULL |
+-----+------+-----------+----------+---------+------+---------------------------------+------------------------+
4 rows in set (0.00 sec)
mysql>
While the SHOW ENGINE INNODB STATUS gives the following:
mysql> SHOW ENGINE INNODB STATUS;
---------------------------------------------------------------------------+
| InnoDB | |
=====================================
2017-03-16 14:18:08 0x7fa0da508700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 6 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 46 srv_active, 0 srv_shutdown, 4115 srv_idle
srv_master_thread log flush and writes: 4161
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 2
OS WAIT ARRAY INFO: signal count 2
RW-shared spins 0, rounds 4, OS waits 2
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 4.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 5384
Purge done for trx's n:o < 0 undo n:o < 0 state: running but idle
History list length 0
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421804127426384, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
229 OS file reads, 390 OS file writes, 13 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 69257, node heap has 0 buffer(s)
Hash table size 69257, node heap has 0 buffer(s)
Hash table size 69257, node heap has 0 buffer(s)
Hash table size 69257, node heap has 0 buffer(s)
Hash table size 69257, node heap has 0 buffer(s)
Hash table size 69257, node heap has 0 buffer(s)
Hash table size 69257, node heap has 0 buffer(s)
Hash table size 69257, node heap has 0 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 2232766
Log flushed up to 2232766
Pages flushed up to 2232766
Last checkpoint at 2232757
0 pending log flushes, 0 pending chkp writes
10 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 274857984
Dictionary memory allocated 317906
Buffer pool size 16384
Free buffers 16129
Database pages 255
Old database pages 0
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 203, created 52, written 370
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 255, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=25594, Main thread ID=140328678156032, state: sleeping
Number of rows inserted 2782, updated 0, deleted 0, read 2792
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
And looking at performance_schema.meta_locks gives this:
mysql> select * from metadata_locks;
+-------------+--------------------+------------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | OBJECT_INSTANCE_BEGIN | LOCK_TYPE | LOCK_DURATION | LOCK_STATUS | SOURCE | OWNER_THREAD_ID | OWNER_EVENT_ID |
+-------------+--------------------+------------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
| TABLE | organize | AccessMatrixView | 72627408 | SHARED_READ | TRANSACTION | GRANTED | sql_parse.cc:6030 | 334 | 4 |
| TABLE | organize | access_matrix | 80700416 | SHARED_READ | TRANSACTION | GRANTED | sql_parse.cc:6030 | 334 | 4 |
| TABLE | organize | people | 81091984 | SHARED_READ | TRANSACTION | GRANTED | sql_parse.cc:6030 | 334 | 4 |
| TABLE | organize | tools | 79476128 | SHARED_READ | TRANSACTION | GRANTED | sql_parse.cc:6030 | 334 | 7 |
| GLOBAL | NULL | NULL | 140327657064288 | INTENTION_EXCLUSIVE | STATEMENT | GRANTED | sql_base.cc:5496 | 335 | 170 |
| SCHEMA | organize | NULL | 140327657360976 | INTENTION_EXCLUSIVE | TRANSACTION | GRANTED | sql_base.cc:5481 | 335 | 170 |
| TABLE | organize | tools | 140327657150944 | EXCLUSIVE | TRANSACTION | PENDING | sql_parse.cc:6030 | 335 | 170 |
| TABLE | performance_schema | metadata_locks | 140327923115360 | SHARED_READ | TRANSACTION | GRANTED | sql_parse.cc:6030 | 351 | 94 |
+-------------+--------------------+------------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
8 rows in set (0.00 sec)
mysql>
Can anyone help me diagnose the issue here?
So to expand upon the other answer, this is the day I figure out TRUNCATE isn't equivalent to DELETE FROM table. Silly me.
Per the documentation:
Logically, TRUNCATE TABLE is similar to a DELETE statement that deletes all rows, or a sequence of DROP TABLE and CREATE TABLE statements. To achieve high performance, it bypasses the DML method of deleting data. Thus, it cannot be rolled back, it does not cause ON DELETE triggers to fire, and it cannot be performed for InnoDB tables with parent-child foreign key relationships.
Although TRUNCATE TABLE is similar to DELETE, it is classified as a DDL statement rather than a DML statement.`
Essentially, a process in the SLEEP state was reading the table, but the TRUNCATE was trying to do a DDL statement, hence it being blocked until the other SLEEP-ing process was killed.
It's thread 335.
mysql> select * from metadata_locks;
+-------------+--------------------+------------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | OBJECT_INSTANCE_BEGIN | LOCK_TYPE | LOCK_DURATION | LOCK_STATUS | SOURCE | OWNER_THREAD_ID | OWNER_EVENT_ID |
+-------------+--------------------+------------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
| GLOBAL | NULL | NULL | 140327657064288 | INTENTION_EXCLUSIVE | STATEMENT | GRANTED | sql_base.cc:5496 | 335 | 170 |
| SCHEMA | organize | NULL | 140327657360976 | INTENTION_EXCLUSIVE | TRANSACTION | GRANTED | sql_base.cc:5481 | 335 | 170 |
| TABLE | organize | tools | 140327657150944 | EXCLUSIVE | TRANSACTION | PENDING | sql_parse.cc:6030 | 335 | 170 |
+-------------+--------------------+------------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
>>>>>> here >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> ^^^
The lock on organize.tools itself appears to actually be waiting for the other two locks held by the same thread. Anything else needing a conflicting lock would wait behind these.
You can't see this thread in SHOW PROCESSLIST because the user you're logged in as does not have the PROCESS privilege, which allows you to see threads owned by other users and what they are doing.
The PROCESS privilege pertains to display of information about the threads executing within the server (that is, information about the statements being executed by sessions). The privilege enables use of SHOW PROCESSLIST or mysqladmin processlist to see threads belonging to other accounts; you can always see your own threads
https://dev.mysql.com/doc/refman/5.7/en/privileges-provided.html#priv_process
Note that mysqladmin processlist (mentioned in the quote) is not a query you run -- it's a shell command that allows you to see the same information provided in response to the query SHOW PROCESSLIST.
I was having the same issue, I was executing: ALTER TABLE Version MODIFY COLUMN version VARCHAR(15);
Version is a table with only 1 row, without foreign keys, is a very simple table to store my current version of the code.
I spent multiple hours modifying the previous command but the problem was with the last SQL instruccion
I was asking in python with mysql:
cur = execute("SELECT * FROM Version")
Then if my version is old
execute(ALTER TABLE Version MODIFY COLUMN version VARCHAR(15);)
commit
Seems like the SELECT was locking the table, it was waiting for me to release, so I had to add the commit after the execute. This is the first time that I add a commit after a consult, I was committing only for instructions that modify the DB.
So the solution for me was to commit the last SQL instruction before the instruction that was getting locked.

ERROR 1123 (HY000): Can't initialize function 'mysql_firewall'; Plugin initialization function failed

I am trying to install and enable the mysql enterprise firewall on one of my linux servers. I need to run in manually, and am executing the linux_install_firewall.sql script.
When I do, when executing the line
mysql> INSTALL PLUGIN mysql_firewall SONAME 'firewall.so';
I get the following error message.
ERROR 1123 (HY000): Can't initialize function 'mysql_firewall'; Plugin initialization function failed.
In reviewing the mysqld.log, i see the following:
2017-02-09T19:40:17.033605Z 5 [ERROR] Plugin MYSQL_FIREWALL reported: 'Disable query cache before using the firewall.'
2017-02-09T19:40:17.033655Z 5 [ERROR] Plugin 'MYSQL_FIREWALL' init function returned error.
2017-02-09T19:40:17.033667Z 5 [ERROR] Plugin 'MYSQL_FIREWALL' registration as a AUDIT failed.
2017-02-09T19:40:17.033686Z 5 [Note] Shutting down plugin 'MYSQL_FIREWALL'
However, I have set query_cache_size = 0 in both my.cnf and directly in global variables. I've also restart the mysqld service.
I still see:
mysql> show variables like '%query_cache%';
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| have_query_cache | YES |
| query_cache_limit | 2097152 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 0 |
| query_cache_type | ON |
| query_cache_wlock_invalidate | OFF |
+------------------------------+---------+
I'm still left without being able to install the firewall. Any ideas?

After upgrading to MariaDB 5.5 connections are being dropped after a period of inactivity

We have upgraded our database server from MySQL 5.1 to MariaDB 5.5 (5.5.40-MariaDB-1~wheezy-log).
After this upgrade, some long running processes mysql connection si being dropped.
Common scenario for those processes is:
Connect to MySQL
Run some queries
Do some heavy lifting without connecting to MySQL for at least one minute
Try to query against the original connection
Exception giving a 2600 error - MySQL server has gone away
This does happen in PHP CLI scripts (php 5.3), but also in a Ruby application (Redmine 2.5.1). It was not happening with MySQL 5.1 and there were no changes on the applications side, so it should not be app-related.
The %timeout% variables in MariaDB are:
+----------------------------+----------+
| Variable_name | Value |
+----------------------------+----------+
| connect_timeout | 5 |
| deadlock_timeout_long | 50000000 |
| deadlock_timeout_short | 10000 |
| delayed_insert_timeout | 300 |
| innodb_lock_wait_timeout | 50 |
| innodb_rollback_on_timeout | OFF |
| interactive_timeout | 28800 |
| lock_wait_timeout | 31536000 |
| net_read_timeout | 30 |
| net_write_timeout | 60 |
| slave_net_timeout | 3600 |
| thread_pool_idle_timeout | 60 |
| wait_timeout | 28800 |
+----------------------------+----------+
We are not using thread pooling:
+---------------------------+---------------------------+
| Variable_name | Value |
+---------------------------+---------------------------+
| thread_cache_size | 128 |
| thread_concurrency | 10 |
| thread_handling | one-thread-per-connection |
| thread_pool_idle_timeout | 60 |
| thread_pool_max_threads | 500 |
| thread_pool_oversubscribe | 3 |
| thread_pool_size | 12 |
| thread_pool_stall_limit | 500 |
| thread_stack | 294912 |
+---------------------------+---------------------------+
When the thing happens, there is also an event logged in syslog, everytime looking the same:
Dec 16 13:00:14 DB01 mysqld: 141216 13:00:14 [Warning] Aborted connection 9202885 to db: 'some_db_name' user: 'user' host: 'app' (Unknown error)
Besides that, there are also weird root account disconnection messages:
Dec 16 13:05:02 DB01 mysqld: 141216 13:05:02 [Warning] Aborted connection 9225621 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:10:00 DB01 mysqld: 141216 13:10:00 [Warning] Aborted connection 9218291 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:10:12 DB01 mysqld: 141216 13:10:12 [Warning] Aborted connection 9232561 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:17:01 DB01 /USR/SBIN/CRON[41343]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Dec 16 13:20:02 DB01 mysqld: 141216 13:20:02 [Warning] Aborted connection 9248777 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:20:02 DB01 mysqld: 141216 13:20:02 [Warning] Aborted connection 9248788 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Dec 16 13:20:12 DB01 mysqld: 141216 13:20:12 [Warning] Aborted connection 9248798 to db: 'unconnected' user: 'root' host: 'localhost' (Unknown error)
Out of those settings is there any, that should be changed to fix the weird server has gone away errors?
In the end we have found out that the DB is not the cause of exceptions dropping, as the drops are appearing also in other unrelated systems.

How can I distinguish mysql master binlog dump threads when there are 2 mysql slave instance on same server?

There are 2 mysql slave instance setup on a server, and one master instance setup on another server.
When "show processlist;" on master, there are 2 different binlog dump threads.
How can I distinguish between one binlog dump threads from the other (which slave use which thread)?
Thanks.
I made a mysql plugin for it.
Demo:
mysql> select * from information_schema.binlog_dump_list;
+----+----------------+------+------+-----------------------------------------------------------------------+
| ID | HOST | PORT | USER | STATUS |
+----+----------------+------+------+-----------------------------------------------------------------------+
| 2 | 192.168.56.202 | 3306 | repl | Master has sent all binlog to slave; waiting for binlog to be updated |
| 3 | 192.168.56.202 | 3406 | repl | Master has sent all binlog to slave; waiting for binlog to be updated |
+----+----------------+------+------+-----------------------------------------------------------------------+
2 rows in set (0.00 sec)

InnoDB Error in Mysql

Last night the server was upgraded from a MySQL server with InnoDB Cpanel have much of a problem when MySQL server will shut down!
Because many of the sites on vps in their database to InnoDB Engine Table Does not have put up their site and say
error: Unknown table engine 'InnoDB'
Error
It will enable the mysql server but it did not put the InnoDB!
The mysql command
show engines;
I noticed the following result which is not InnoDB Result
mysql> show engines;
| Engine | Support | Comment | Transactions | XA | Savepoints |
| MyISAM | DEFAULT | Default engine as of MySQL 3.23 with great performance | NO | NO | NO |
| MRG_MYISAM | YES | Collection of identical MyISAM tables | NO | NO | NO |
| BLACKHOLE | YES | / dev / null storage engine (anything you write to it disappears) | NO | NO | NO |
| CSV | YES | CSV storage engine | NO | NO | NO |
| MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO |
| FEDERATED | NO | Federated MySQL storage engine | NULL | NULL | NULL |
| ARCHIVE | YES | Archive storage engine | NO | NO | NO |
7 rows in set (0.00 sec)
Log into mysql server gives an error ...
121105 18:26:50 mysqld_safe Starting mysqld daemon with databases from / var / lib / mysql
121105 18:26:50 [Note] Plugin 'FEDERATED' is disabled.
121105 18:26:50 InnoDB: Initializing buffer pool, size = 8.0M
121105 18:26:50 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 268435456 bytes
InnoDB: than specified in the. Cnf file 0 536870912 bytes!
121105 18:26:50 [ERROR] Plugin 'InnoDB' init function returned error.
121105 18:26:50 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
121105 18:26:50 [Note] Event Scheduler: Loaded 0 events
121105 18:26:50 [Note] / usr / sbin / mysqld: ready for connections.
Version: '5 .1.65-cll 'socket:' / var / lib / mysql / mysql.sock 'port: 3306 MySQL Community Server (GPL)
in my.cnf innodb settings for both are:
innodb_fast_shutdown = 0
innodb_log_buffer_size = 8M
innodb_log_file_size = 512M
Thanks, plz help me to solve the problem.
From what i see in your logs your InnoDB is broken due to its file ib_logfile0is messed up. I had same problem when hard drive crashed leaving this file unreadable - after reading what i could form the broken drive file was good but data inside was messed out.
InnoDB: Error: log file ./ib_logfile0 is of different size 0 268435456 bytes
InnoDB: than specified in the. Cnf file 0 536870912 bytes!
There it is -> You file is 256 mb size yet your configuration tells it should be 512mb size. Fix the error in the configuration.
Then - backup all yours data (innodb tables with data).
Shutdown database (clean shutdown - all transactions must be finished - the log shouldn't contain them in order to make the file bigger).
Then backup the ib* file and change configuration.
Restart DB.
For more detailed information go there http://dev.mysql.com/doc/refman/5.5/en/innodb-data-log-reconfiguration.html