I have a Node.js application using MySQL on an AWS RDS with Bookshelf & Knex libraries. The RDS Instance have a max_connections value 90.
I am using the following as the connection object.
knex: {
client: 'mysql',
connection: {
host: 'xxxxxxx.rds.amazonaws.com',
user: 'xxx',
password: 'xxxxx',
database: 'xxxx',
charset: 'utf8'
},
debug: true,
pool: {
min: 2,
max: 20
},
acquireConnectionTimeout: 10000
},
const config = require('./environment');
const knex = require('knex')(config.knex);
module.exports = require('bookshelf')(knex).plugin('registry');
'use strict';
const bookshelf = require('../config/bookshelf');
const config = require('../config/environment');
module.exports = bookshelf.model(`TableA`, {
tableName: 'TableA'
}, {});
I have many requests coming along to the application and sometimes crashes with the following errors.
Unhandled rejection TimeoutError: Knex: Timeout acquiring a
connection. The pool is probably full. Are you missing a
.transacting(trx) call?
and
Error: ER_CON_COUNT_ERROR: Too many connections
Also I see a number of connections (40 to 50 on an average) in the server PROCESSLIST with Command as sleep.
I suspect these error happen when all the 90 connections on the server used fully / knex cannot acquire a new connection from he pool when it tries to. What could be a permanent solution for this, and best practices for handling these kind of applications.
I dont think it is the RDS max_connections that is causing the issue, assuming you only have one instance of the above application code running at any time.
Your application uses a DB connection pool, which can hold up to 20 connections. If all those connections are in use, then the application waits for up to acquireConnectionTimeout ms in your case that is 10000 before connection timeout.
So I suspect that your application either has a lot of db queries to be processed due to load or there are some slow queries hogging connections. This causes a backlog of queries waiting for connections that eventually times out. Investigate which might be the case and do update us.
Things you can try in the mean time.
Increase acquireConnectionTimeout.
Increase connection pool size.
If caused by slow queries then optimize them before trying the above.
Possible methods for logging slow queries:
Enable slow query log on RDS.
Knex query event to log transaction duration assuming you are using transactions.
When a client is finished with MySQL, have it disconnect.
Also, check the value of wait_timeout. Lowering it will force disconnections rather than "Sleeping" until you come back.
Related
I built a multi-tenant microservice architecture in Nest.js, multi-tenant connection are made using TypeOrm latest Datasource API. After upgrading to latest TypeORM version we encountering the MySQL "Too many connections" error.
During searching about this I found that in their latest version they added option "PoolSize" to control number of active connections. I've added that too but the issue is still there.
Ideally, TypeOrm should close the connection once the DB operation finished or use opened connection (if any) on new request, but the connection to MySQL is keeping active but in sleep state, and on a new request it create new connection. see below:
By running show processlist; cmd on MySQL
I've created the multi-tenant connection using nest.js provider for incoming request in microservice:
databaseSource is used for initial database connection to default database and then on each tenant request we create the new DB connection.
export const databaseSource = new DataSource({
type: process.env.DB_CONNECTION,
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT, 10),
username: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: process.env.DB_DATABASE,
entities: ["src/models/entities/**/*.ts"],
synchronize: true,
poolSize: 10,
});
const connectionFactory = {
provide: CONNECTION,
scope: Scope.REQUEST,
useFactory: async (request: RequestContext) => {
const tenantHost = request.data["tenantId"] || request.data;
if (tenantHost) {
const tenantConnection: DataSource = await getTenantConnection(
tenantHost
);
return tenantConnection;
}
return null;
},
inject: [REQUEST],
};
#Global()
#Module({
providers: [connectionFactory],
exports: [CONNECTION],
})
export class TenancyModule {}
export async function getTenantConnection(
tenantHost: string
): Promise<DataSource> {
const tenantId = getTenantId(tenantHost);
const connectionName = `${tenantId}`;
const DBConfig: DataSourceOptions = {
type: process.env.DB_CONNECTION,
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT, 10),
username: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: connectionName,
entities: ["src/models/entities/**/*.ts"],
synchronize: true,
poolSize: 10,
};
const dataSource = new DataSource(DBConfig);
if (!dataSource.isInitialized) {
await dataSource.initialize();
}
return dataSource;
}
Then once datasource initalized,I inject it into Service and used it to getRepository and performed DB operation.
I researched a lot about this some saying increase the MySQL "max_connections" limit, some saying passed the "connectionLimit" options in TypeOrm config (poolSize in latest version) but nothing works for me.
Am I doing anything wrong to create the tenant connection?
Is there any way to closed the connection manually after DB operation?
Your error doesn't have to do anything with typeorm version. It's most likely that the number of tenants have increased and the way you're creating the connection, you're going to run out of connections if not now then later.
There are a number of things that you can do to make it work. First of them all would be to limit the number of connections per tenant. The correct parameter to limit the number of connections in a pool is to use connectionLimit parameter inside extra object for typeorm versions < 0.3.10 and poolSize for typeorm versions >= 0.3.10.
TypeORM is just an ORM, it delegates underlying communication to the database to the corresponding driver. In case of mysql, it uses this mysql npm module. Whatever option that you specify in dataSourceOptions is passed onto the driver. You can see the set of available pool options. You might wanna set this number to a very small value as you're going to have multiple tenants. Perhaps keep this value configurable for every tenant. A smaller number for a not so big tenant and a larger value for a very busy one. This way you'll be able to reduce overall connections pressure on your database server.
Talking about the screenshot you've pasted with high number of connections in sleep command, this is mostly due to the pool of connections made by your application. This doesn't pose any harm unless it surpasses the max_connections variable on your mysql database server. In your case, it has happened indeed that's why the error: Too many connections.
Other options you might explore is to increase the value of the variable max_connections so that you're able to accommodate all your tenants. You might also wanna increase the server size as increasing this variable will increase RAM usage, unless of course mysql is already running on a very big machine.
Edit 1: After reading your comment, I see a probable issue at this line:
if (databaseSource.options.database === connectionName) {
//
}
databaseSource.options.database will always be equal to process.env.DB_DATABASE when the databaseSource is first initialised. Upon any subsequent request for connection for any tenantId, this check will fail and every time a new connection pool will be created.
Edit 2: Again the issue lies within your code. You're always creating a new DataSource object without checking if there is already a connection pool for that tenant. isInitialized flag will always be false for a new object and your code will do dataSource.initialize() which will create new pool. Hint: Try to keep connection pools created in a map:
const tenantPools = {
tenantId: dataSource
}
and before creating a new DataSource object, check if that already exists in this map.
I am using mysql package of npm in my NodeJS project. I am using connection pool as below -
var pool = mysql.createPool({
connectionLimit: 50,
host: host,
user: user,
password: password,
database: database
});
And then I am using the pool as -
pool.query("Select ....", function (err, data) {
});
But sometimes our database server is stuck due to large queries and I think the connection limit of this connection gets crossed. Then after the stuck queries have executed successfully, the mysql library cannot acquire new connections. I cannot even see the queries in SHOW PROCESSLIST of MySQL. So there is issue in acquiring new connections. There is nothing in the logs too. I sort the issue by restarting the Node Server, but it isnt the ideal solution. Please help me in identifying the cause of the issue. Similar issue occurs with MSSQL connections in NodeJS and I just cannot identify the reason for this.
After you're done processing your query, you should send pool.end() to close the current connection.
I built a program with NodeJS where multiple users access it in the same time and do a lot of operations that queries the MySQL database.
My approach is very simple. I only open one connection when the app is started and leave it that way.
const dbConfig = require('./db-config');
const mysql = require('mysql');
// Create mySQL Connection
const db = mysql.createConnection({
host: dbConfig.host,
user: dbConfig.user,
password: dbConfig.password,
database: dbConfig.database,
multipleStatements: true
});
// Connect MySQL
db.connect((err) => {
if (err) {
throw err;
} else {
console.log('MySQL connected!');
}
});
module.exports = db;
And then, whenever the program needs to query the database, i do like this
db.query('query_in_here', (error, result) => {
*error_handling_and_doing_stuff*
}
I'm having trouble when noone access the app for a long period of time (some hours).
Because when this happens i think the connection is being closed automatically. And then, when a user try to access the app, i see in the console that the connection timed out.
My first thought was too handle the disconnection and connect again. But, it get me thinking if this is the correct approach.
Should i use pool connections instead? Because if i keep only one connection it means that two users can't query the database in the same time?
I tried to understand tutorials with pool connections but couldn't figure out when to create new connections and when should i end them.
UPDATE 1
Instead of create one connection when the app is started i changed to create a pool connection.
const dbConfig = require('./db-config');
const mysql = require('mysql');
// Create mySQL Connection
const db = mysql.createPool({
host: dbConfig.host,
user: dbConfig.user,
password: dbConfig.password,
database: dbConfig.database,
multipleStatements: true
});
module.exports = db;
It seems that when i use now "db.query(....)" the mysql connection and release of that connection is done automatically.
So, it should resolve my issue but i don't know if this is the correct approach.
Should i use pool connections instead?
Yes you should. Pooling is supported out-of-the-box with the mysql module.
var mysql = require('mysql');
var pool = mysql.createPool({
connectionLimit : 10,
host : 'example.org',
user : 'bob',
password : 'secret',
database : 'my_db'
});
pool.query('SELECT 1 + 1 AS solution', function (error, results, fields) {
// should actually use an error-first callback to propagate the error, but anyway...
if (error) return console.error(error);
console.log('The solution is: ', results[0].solution);
});
You're not supposed to know how pooling works. It's abstracted from you. All you need to do is use pool to dispatch queries. How it works internally is not something you're required to understand.
What you should pay attention to is the connectionLimit configuration option. This should match your MySQL server connection limit (minus one, in case you want to connect to it yourself while your application is running), otherwise you'll get "too many connections" errors. The default connection limit for MySQL is 100, so I'd suggest you set connectionLimit to 99.
Because if i keep only one connection it means that two users can't query the database in the same time?
Without pooling, you can't serve multiple user requests in-parallel. It's a must have for any non-hobby, data-driven application.
Now, if you really want to know how connection pooling works, this article sums it up pretty nicely.
In software engineering, a connection pool is a cache of database connections maintained so that the connections can be reused when future requests to the database are required. Connection pools are used to enhance the performance of executing commands on a database. Opening and maintaining a database connection for each user, especially requests made to a dynamic database-driven website application, is costly and wastes resources. In connection pooling, after a connection is created, it is placed in the pool and it is used again so that a new connection does not have to be established. If all the connections are being used, a new connection is made and is added to the pool. Connection pooling also cuts down on the amount of time a user must wait to establish a connection to the database.
I have connected an AWS Lambda function to Amazon RDS (MySQL). When the Lambda function is invoked 100 times simultaneously, there are almost 400 connections opened in RDS (as shown in RDS console). Why is this?
I checked the active connections using:
SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST WHERE DB = "MYDB";
All the connections are from Lambda containers. Does anyone know how Lambda containers act on simultaneous requests? Why are the containers not reused?
Current Configuration:
var sequelize = new Sequelize('DB','username', 'password' ,{
dialect: 'mysql',
port: port,
host: host,
pool: {
max: 20,
min: 0,
idle: 300000
}
});
Even if one connection is opened per request it should be 100. How 400 connections are opened?
I'm using Sequelize. Node JS 6.9.1
Note: Connection Happens only once outside Lambda Handler method
Sequelize by defaults creates a connection pool, so it's creating 4 connections because it's designed to run as a long running service. You can disable this by setting options.pool to false see the API reference
However as your application scales this is a fundamental problem with Lambda that won't go away. I recommend building a DB proxy layer in EC2 to terminate your db connections (i.e. using ProxySQL). It will then have a connection pool to the rds db.
The only way around this is to use dynamodb as your backend store.
The lambda can have many concurrent executions.
So, more than one connection can be through by the lambda.
To fix this you need to change sequelize:
var sequelize = new Sequelize('DB','username', 'password' ,{
dialect: 'mysql',
port: port,
host: host,
pool: {
max: 1,
min: 1,
idle: 15000
}
});
Keep the pool with only one connection, in this way the connection will be reused on the next execution and remember to dont close connection after the lambda execution.
Read this article about to reuse database connection on next executions.
AWS "new" solution for this problem seems to be RDS Proxy: https://aws.amazon.com/de/rds/proxy/
RDS Proxy establishes and manages the necessary connection pools to your database so that your application creates fewer database connections.
You can use RDS Proxy for any application that makes SQL calls to your database. But in the context of serverless, we focus on how this improves the Lambda experience.
https://aws.amazon.com/de/blogs/compute/using-amazon-rds-proxy-with-aws-lambda/
I'm trying to utilize Knex with the Loopback framework. Currently, Loopback does not provide a good way to create advanced queries.
I'm using Knex's query builder, however by default Knex will initialize its own connection pool ON TOP of Loopback's. Instead, I want to use the connection pool already created by Loopback.
I've tried to use Knex's .connection() method to set the connection to be the one from loopback, however when I monitor the processes on my MySql server I notice that each time I make a call that uses Knex a new connection is created. Over time this is causing my server to run out of connections to the database.
I'm doing something like this:
var knex = require('knex')({
client: 'mysql',
connection: {
host: mysql.host,
port: mysql.port,
user: mysql.username,
password: mysql.password,
database: mysql.database,
debug: false
}
});
app.datasources.mysqldb.client.getConnection(function(err, connection){
knex.connection(connection).
//continue with the query building
}
My question is, how do I utilize Loopback's existing connection with Knex so that Knex doesn't burn through all the available connections in my database? I've also tried using knex's "pool" configuration but it doesnt seem to do anything...