nodejs mysql docker ECONNRESET if idle for 16 minutes - mysql

Question

Whenever my nodejs mysql connection in a docker container is idle for 16 minutes or longer, I get the following error message:
2022-04-05T11:25:53.802Z Success: [ RowDataPacket { '1': 1 } ]
2022-04-05T11:41:58.512Z
/app/index.js:12
if(err) throw err;
^
Error: read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:209:20)
--------------------
at Protocol._enqueue (/app/node_modules/mysql/lib/protocol/Protocol.js:144:48)
at Connection.query (/app/node_modules/mysql/lib/Connection.js:198:25)
at results (/app/index.js:11:64)
at new Promise (<anonymous>)
at checkConnection (/app/index.js:11:27)
at Timeout._onTimeout (/app/index.js:16:20)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7) {
errno: 'ECONNRESET',
code: 'ECONNRESET',
syscall: 'read',
fatal: true
}
The above error does not occur if I host my nodejs app and mysql directly against my host machine without docker.
Does anyone know how to fix this problem with my nodejs/mysql/docker-swarm set up? Here's all my code to reproduce the problem, I put all the files in the same directory:
// docker-compose.yml
version: "3.8"
services:
mysql:
image: mysql:5.7.34
environment:
- MYSQL_ROOT_PASSWORD=rootpass
- MYSQL_USER=myuser
- MYSQL_PASSWORD=mypass
- MYSQL_DATABASE=mydatabase
volumes:
- ./mysqld.cnf:/etc/mysql/mysql.conf.d/mysqld.cnf
command: "--wait_timeout=28800"
networks:
- app-network
nodejs:
image: node:12-alpine
working_dir: /app
volumes:
- ./:/app
depends_on:
- mysql
entrypoint: ["node", "/app/index.js"]
networks:
- app-network
networks:
app-network:
external: true
// mysqld.cnf
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 8
myisam-recover-options = BACKUP
query_cache_limit = 1M
query_cache_size = 16M
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
// package.json
{
"name": "mysql-docker",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC",
"dependencies": {
"mysql": "^2.18.1"
}
}
// index.js
const mysql = require('mysql');
const pingTime = 1000 * (60 * 16);
const connectionParameters = {
host: 'testconnection_mysql',
user: 'myuser',
password: 'mypass',
database: 'mydatabase',
port: '3306',
};
const checkConnection = async (mysqlClient) => {
const results = await new Promise((resolve) => mysqlClient.query("SELECT 1", (err, results) => {
if(err) throw err;
resolve(results);
}));
console.log(new Date(), "Success:", results);
setTimeout(()=>checkConnection(mysqlClient), pingTime);
};
const run = async () => {
// Give time for MySQL Service to warm up
await new Promise(resolve => setTimeout(()=>resolve(), 10000));
const mysqlClient = mysql.createConnection(connectionParameters);
await new Promise((resolve, reject) => mysqlClient.connect((e) => {
if (e) {
reject(e);
return;
}
resolve();
}));
try {
await checkConnection(mysqlClient);
} catch (e) {
console.log(new Date(), "Error:", e);
}
};
run();
Once I have all these files in the same directory, I simply run these commands from the directory:
npm install;
docker swarm init;
docker network create --driver overlay app-network;
docker stack deploy -c docker-compose.yml testconnection;
After 16 minutes, doing a docker logs <container id for the index.js file> gives the error shown at the top of this message.
If I change the pingTime in the index.js to 14 minutes likes this const pingTime = 1000 * (60 * 14);, restart the swarm and containers, then I don't get any disconnects. Even after many hours, doing a docker logs will show successful queries every 14 minutes.
How do I stop mysql and nodejs from losing connections when idle for 16 minutes or longer?
OTHER NOTES
If I change the network of my docker-compose.yml to this:
networks:
app-network:
driver: bridge
I get this error:
failed to create service testconnection_mysql: Error response from daemon: The network testconnection_app-network cannot be used with services. Only networks scoped to the swarm can be used, such as those created with the overlay driver.
If I change the network of my docker-compose.yml to this:
networks:
app-network:
driver: overlay
Then my nodejs app is unable to connect to mysql and I get this error:
Error: ER_DBACCESS_DENIED_ERROR: Access denied for user 'myuser'#'%' to database 'mydatabase'
So only the external network seems to let my nodejs app connect to mysql while in swarm mode.

Answer 1

In services.mysql.command you should use "--wait-timeout=28800" instead of "--wait_timeout=28800" (see MySQL Man). To verify that the system variable is set correctly, execute SHOW VARIABLES LIKE 'wait_timeout'; in your Node.js app. You can even set it dynamically with SET SESSION wait_timeout = 28800;.
If it still doesn't work, try to diagnose. In checkConnection() delete the SQL query to make the script not crash and wait 16 minutes. Connect into the nodejs container with docker exec -it <container_id> bash and try ping testconnection_mysql. If it doesn't work, the problem is in the app-network. If it does work, connect into the mysql container and try to connect to MySQL server with mysql CLI client if available. If it doesn't work, the problem is in the database server. If it does work, the problem is probably still with the timeouted connection in Node.js app.
Why you use app-network, which is set to external? Is it classical bridge network? If you use this network only with containers declared in this single Composefile, you should not declare it external. You can get more information about the network with docker network inspect app-network.