MySQL connection refused from Node on first run only (docker-compose) - mysql

Question

I'm trying to build out a NodeJS/MySQL stack, but am experiencing an intermittent issue.
Here is my docker compose file:
version: '3'
services:
db:
image: mysql:5.7
command: --default-authentication-plugin=mysql_native_password
ports:
- "3306:3306"
environment:
MYSQL_ROOT_PASSWORD: root
container_name: 'db'
networks:
- db-network
volumes:
- db-data:/var/lib/mysql
- ./db:/docker-entrypoint-initdb.d
forum:
image: "node:12"
user: "node"
working_dir: /home/node/app
depends_on:
- db
ports:
- 7000:3000
environment:
- NODE_ENV=development
- PORT=3000
volumes:
- ./server:/home/node/app
command: "npm start"
networks:
- db-network
adminer:
image: adminer
restart: always
networks:
- db-network
depends_on:
- db
ports:
- 7777:8080
environment:
ADMINER_DEFAULT_DB_DRIVER: mysql
ADMINER_DEFAULT_DB_HOST: db
volumes:
db-data:
networks:
db-network:
You can see that I've got an SQL script mapped into the initdb entrypoint. This is really simple SQL, it simply creates the db: CREATE DATABASE IF NOT EXISTS forum;
Inside the Node container I'm using knex to talk to the database. When I instantiate knex, I also run any outstanding migrations:
const environment = process.env.ENVIRONMENT || 'development'
const Knex = require('knex');
const knexConfig = require('../knexfile');
const knex = Knex(knexConfig[environment]);
async function init() {
try {
return await knex.migrate.latest();
} catch (error) {
console.log(error);
} finally {
return;
}
}
init();
module.exports = knex;
The problem i'm getting only appears on 'first up' o docker-compose (or more specifically when it needs to create the db-data volume). If I drop the volume docker-compose down -V I get the error when 'up-ing' again. The problem is your classic ECONNREFUSED:
Creating network "forum_db-network" with the default driver
Creating volume "forum_db-data" with default driver
Creating db ... done
Creating forum_adminer_1 ... done
Creating forum_forum_1 ... done
Attaching to db, forum_adminer_1, forum_forum_1
db | Initializing database
db | 2020-02-20T09:50:46.701629Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
db | 2020-02-20T09:50:47.004757Z 0 [Warning] InnoDB: New log files created, LSN=45790
db | 2020-02-20T09:50:47.062521Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
db | 2020-02-20T09:50:47.122917Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 7810f3c2-53c6-11ea-ba00-0242c0a81002.
db | 2020-02-20T09:50:47.126689Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
adminer_1 | [Thu Feb 20 09:50:47 2020] PHP 7.4.2 Development Server (http://[::]:8080) started
db | 2020-02-20T09:50:47.127637Z 1 [Warning] root#localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
db | 2020-02-20T09:50:47.955961Z 1 [Warning] 'user' entry 'root#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:47.956454Z 1 [Warning] 'user' entry 'mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:47.957026Z 1 [Warning] 'user' entry 'mysql.sys#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:47.957747Z 1 [Warning] 'db' entry 'performance_schema mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:47.958212Z 1 [Warning] 'db' entry 'sys mysql.sys#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:47.958797Z 1 [Warning] 'proxies_priv' entry '# root#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:47.959520Z 1 [Warning] 'tables_priv' entry 'user mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:47.960113Z 1 [Warning] 'tables_priv' entry 'sys_config mysql.sys#localhost' ignored in --skip-name-resolve mode.
forum_1 |
forum_1 | > express-oauth-vuex-starter#1.0.0 start /home/node/app
forum_1 | > nodemon index.js
forum_1 |
forum_1 | [nodemon] 2.0.2
forum_1 | [nodemon] to restart at any time, enter `rs`
forum_1 | [nodemon] watching dir(s): *.*
forum_1 | [nodemon] watching extensions: js,mjs,json
forum_1 | [nodemon] starting `node index.js`
db | Database initialized
db | Initializing certificates
db | Generating a RSA private key
db | ........................................................................................................................+++++
db | ..........+++++
db | unable to write 'random state'
db | writing new private key to 'ca-key.pem'
db | -----
db | Generating a RSA private key
db | .........................................................................................+++++
db | .............+++++
db | unable to write 'random state'
db | writing new private key to 'server-key.pem'
db | -----
db | Generating a RSA private key
db | .....................................................+++++
db | .+++++
db | unable to write 'random state'
db | writing new private key to 'client-key.pem'
db | -----
db | Certificates initialized
db | MySQL init process in progress...
db | 2020-02-20T09:50:51.037907Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
db | 2020-02-20T09:50:51.039423Z 0 [Note] mysqld (mysqld 5.7.24) starting as process 89 ...
db | 2020-02-20T09:50:51.044545Z 0 [Note] InnoDB: PUNCH HOLE support available
db | 2020-02-20T09:50:51.044653Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
db | 2020-02-20T09:50:51.044701Z 0 [Note] InnoDB: Uses event mutexes
db | 2020-02-20T09:50:51.044745Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
db | 2020-02-20T09:50:51.044786Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
db | 2020-02-20T09:50:51.044825Z 0 [Note] InnoDB: Using Linux native AIO
db | 2020-02-20T09:50:51.045705Z 0 [Note] InnoDB: Number of pools: 1
db | 2020-02-20T09:50:51.046113Z 0 [Note] InnoDB: Using CPU crc32 instructions
db | 2020-02-20T09:50:51.048701Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
db | 2020-02-20T09:50:51.060691Z 0 [Note] InnoDB: Completed initialization of buffer pool
db | 2020-02-20T09:50:51.064263Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
db | 2020-02-20T09:50:51.076676Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
db | 2020-02-20T09:50:51.097380Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
db | 2020-02-20T09:50:51.097533Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
db | 2020-02-20T09:50:51.131620Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
db | 2020-02-20T09:50:51.134052Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
db | 2020-02-20T09:50:51.134142Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
db | 2020-02-20T09:50:51.135177Z 0 [Note] InnoDB: Waiting for purge to start
db | 2020-02-20T09:50:51.185530Z 0 [Note] InnoDB: 5.7.24 started; log sequence number 2591440
db | 2020-02-20T09:50:51.186329Z 0 [Note] Plugin 'FEDERATED' is disabled.
db | 2020-02-20T09:50:51.192075Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
db | 2020-02-20T09:50:51.192591Z 0 [Warning] CA certificate ca.pem is self signed.
db | 2020-02-20T09:50:51.194602Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
db | 2020-02-20T09:50:51.197793Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200220 9:50:51
db | 2020-02-20T09:50:51.198483Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
db | 2020-02-20T09:50:51.200316Z 0 [Warning] 'user' entry 'root#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:51.200460Z 0 [Warning] 'user' entry 'mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:51.200532Z 0 [Warning] 'user' entry 'mysql.sys#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:51.200606Z 0 [Warning] 'db' entry 'performance_schema mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:51.200656Z 0 [Warning] 'db' entry 'sys mysql.sys#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:51.200717Z 0 [Warning] 'proxies_priv' entry '# root#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:51.202731Z 0 [Warning] 'tables_priv' entry 'user mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:51.202816Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:51.219832Z 0 [Note] Event Scheduler: Loaded 0 events
db | 2020-02-20T09:50:51.221248Z 0 [Note] mysqld: ready for connections.
db | Version: '5.7.24' socket: '/var/run/mysqld/mysqld.sock' port: 0 MySQL Community Server (GPL)
forum_1 | Listening...
forum_1 | Error: connect ECONNREFUSED 192.168.16.2:3306
forum_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1137:16)
forum_1 | at Protocol._enqueue (/home/node/app/node_modules/mysql/lib/protocol/Protocol.js:144:48)
forum_1 | at Protocol.handshake (/home/node/app/node_modules/mysql/lib/protocol/Protocol.js:51:23)
forum_1 | at Connection.connect (/home/node/app/node_modules/mysql/lib/Connection.js:116:18)
forum_1 | at /home/node/app/node_modules/knex/lib/dialects/mysql/index.js:69:18
forum_1 | From previous event:
forum_1 | at Client_MySQL.acquireRawConnection (/home/node/app/node_modules/knex/lib/dialects/mysql/index.js:64:12)
forum_1 | at create (/home/node/app/node_modules/knex/lib/client.js:291:39)
forum_1 | at processTicksAndRejections (internal/process/task_queues.js:97:5) {
forum_1 | errno: 'ECONNREFUSED',
forum_1 | code: 'ECONNREFUSED',
forum_1 | syscall: 'connect',
forum_1 | address: '192.168.16.2',
forum_1 | port: 3306,
forum_1 | fatal: true
forum_1 | }
db | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
db | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
db | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
db | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
db | 2020-02-20T09:50:54.889060Z 4 [Warning] 'user' entry 'root#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:54.889116Z 4 [Warning] 'user' entry 'mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:54.889137Z 4 [Warning] 'user' entry 'mysql.sys#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:54.889164Z 4 [Warning] 'db' entry 'performance_schema mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:54.889172Z 4 [Warning] 'db' entry 'sys mysql.sys#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:54.889188Z 4 [Warning] 'proxies_priv' entry '# root#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:54.890011Z 4 [Warning] 'tables_priv' entry 'user mysql.session#localhost' ignored in --skip-name-resolve mode.
db | 2020-02-20T09:50:54.890057Z 4 [Warning] 'tables_priv' entry 'sys_config mysql.sys#localhost' ignored in --skip-name-resolve mode.
db |
db | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
db | mysql: [Warning] Using a password on the command line interface can be insecure.
db |
db |
db | 2020-02-20T09:50:54.915362Z 0 [Note] Giving 0 client threads a chance to die gracefully
db | 2020-02-20T09:50:54.915705Z 0 [Note] Shutting down slave threads
db | 2020-02-20T09:50:54.916456Z 0 [Note] Forcefully disconnecting 0 remaining clients
db | 2020-02-20T09:50:54.917013Z 0 [Note] Event Scheduler: Purging the queue. 0 events
db | 2020-02-20T09:50:54.917696Z 0 [Note] Binlog end
db | 2020-02-20T09:50:54.918665Z 0 [Note] Shutting down plugin 'ngram'
db | 2020-02-20T09:50:54.919138Z 0 [Note] Shutting down plugin 'partition'
db | 2020-02-20T09:50:54.919357Z 0 [Note] Shutting down plugin 'BLACKHOLE'
db | 2020-02-20T09:50:54.920107Z 0 [Note] Shutting down plugin 'ARCHIVE'
db | 2020-02-20T09:50:54.920322Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
db | 2020-02-20T09:50:54.921325Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
db | 2020-02-20T09:50:54.921840Z 0 [Note] Shutting down plugin 'MyISAM'
db | 2020-02-20T09:50:54.922751Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
db | 2020-02-20T09:50:54.923300Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
db | 2020-02-20T09:50:54.923360Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
db | 2020-02-20T09:50:54.923589Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
db | 2020-02-20T09:50:54.923959Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
db | 2020-02-20T09:50:54.924112Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
db | 2020-02-20T09:50:54.924269Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
db | 2020-02-20T09:50:54.924754Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
db | 2020-02-20T09:50:54.925454Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
db | 2020-02-20T09:50:54.925535Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
db | 2020-02-20T09:50:54.925688Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
db | 2020-02-20T09:50:54.925728Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
db | 2020-02-20T09:50:54.926261Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
db | 2020-02-20T09:50:54.926443Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
db | 2020-02-20T09:50:54.926836Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
db | 2020-02-20T09:50:54.927513Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
db | 2020-02-20T09:50:54.927560Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
db | 2020-02-20T09:50:54.927718Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
db | 2020-02-20T09:50:54.927806Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
db | 2020-02-20T09:50:54.927847Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
db | 2020-02-20T09:50:54.928201Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
db | 2020-02-20T09:50:54.928360Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
db | 2020-02-20T09:50:54.928817Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
db | 2020-02-20T09:50:54.929031Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
db | 2020-02-20T09:50:54.929717Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
db | 2020-02-20T09:50:54.929740Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
db | 2020-02-20T09:50:54.929898Z 0 [Note] Shutting down plugin 'INNODB_CMP'
db | 2020-02-20T09:50:54.929912Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
db | 2020-02-20T09:50:54.930509Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
db | 2020-02-20T09:50:54.930535Z 0 [Note] Shutting down plugin 'INNODB_TRX'
db | 2020-02-20T09:50:54.930549Z 0 [Note] Shutting down plugin 'InnoDB'
db | 2020-02-20T09:50:54.931131Z 0 [Note] InnoDB: FTS optimize thread exiting.
db | 2020-02-20T09:50:54.931746Z 0 [Note] InnoDB: Starting shutdown...
db | 2020-02-20T09:50:55.032188Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
db | 2020-02-20T09:50:55.032448Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 200220 9:50:55
If the volume already exists, I get no errors, knex connects and the migrations are run.
In the error log you can see that the Node container starts up directly after mysqld: ready for connections. However, you can see that after the Node error, the db container then runs init.sql. Presumably this means that knex is trying to connect to a specific database before it exists, which I guess could be the cause of the refused connection.
So I guess I'm looking for a way to defer the instantiation of the node container until after the MySQL container has run init.sql. I'm already using the depends_on docker compose property - are there any other docker options available to me?

Answer 1

There's no native solution that I'm aware of, I'm afraid.
For waiting for a service to become available in general, I've used wait-for-it, but in your case you'd need something to verify MySQL also contains the database your app requires.
Maybe a simple Node script that loops a connection and attempts to select the database before running your app proper.

Answer 2

In the end I solved this by setting the MYSQL_DATABASE environment variable for the MySQL container. From the MySQL image docs:
MYSQL_DATABASE
This variable is optional and allows you to specify the
name of a database to be created on image startup. If a user/password
was supplied (see below) then that user will be granted superuser
access (corresponding to GRANT ALL) to this database
db:
image: mysql:5.7
command: --default-authentication-plugin=mysql_native_password
ports:
- "3306:3306"
environment:
MYSQL_DATABASE: forum
MYSQL_ROOT_PASSWORD: root
container_name: 'db'
networks:
- db-network
volumes:
- db-data:/var/lib/mysql
In addition to this, I used is-port-reachable to hold initial migrations until the MySQL container is available.
const isPortReachable = require('is-port-reachable');
const environment = process.env.ENVIRONMENT || 'development'
const Knex = require('knex');
const knexConfig = require('../knexfile');
const knex = Knex(knexConfig[environment]);
async function init() {
let dbStatus = false;
while (!dbStatus) {
dbStatus = await isPortReachable(knexConfig[environment].connection.port, {host: knexConfig[environment].connection.host});
}
try {
return await knex.migrate.latest();
} catch (error) {
console.log(error);
}
}
init();
module.exports = knex;
Update
While the above did solve this specific problem, there was a raft of other issues that kept popping up - all to do with MySQL configuration. In the end I got tired of playing whack-a-mole and switched to Postgres (incredibly simple thanks to Knex). Span the image up and everything just worked...
If you are having issues with docker-compose, knex and mysql, my advice would be to just use Postgres!