I have connected an AWS Lambda function to Amazon RDS (MySQL). When the Lambda function is invoked 100 times simultaneously, there are almost 400 connections opened in RDS (as shown in RDS console). Why is this?
I checked the active connections using:
SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST WHERE DB = "MYDB";
All the connections are from Lambda containers. Does anyone know how Lambda containers act on simultaneous requests? Why are the containers not reused?
Current Configuration:
var sequelize = new Sequelize('DB','username', 'password' ,{
dialect: 'mysql',
port: port,
host: host,
pool: {
max: 20,
min: 0,
idle: 300000
}
});
Even if one connection is opened per request it should be 100. How 400 connections are opened?
I'm using Sequelize. Node JS 6.9.1
Note: Connection Happens only once outside Lambda Handler method
Sequelize by defaults creates a connection pool, so it's creating 4 connections because it's designed to run as a long running service. You can disable this by setting options.pool to false see the API reference
However as your application scales this is a fundamental problem with Lambda that won't go away. I recommend building a DB proxy layer in EC2 to terminate your db connections (i.e. using ProxySQL). It will then have a connection pool to the rds db.
The only way around this is to use dynamodb as your backend store.
The lambda can have many concurrent executions.
So, more than one connection can be through by the lambda.
To fix this you need to change sequelize:
var sequelize = new Sequelize('DB','username', 'password' ,{
dialect: 'mysql',
port: port,
host: host,
pool: {
max: 1,
min: 1,
idle: 15000
}
});
Keep the pool with only one connection, in this way the connection will be reused on the next execution and remember to dont close connection after the lambda execution.
Read this article about to reuse database connection on next executions.
AWS "new" solution for this problem seems to be RDS Proxy: https://aws.amazon.com/de/rds/proxy/
RDS Proxy establishes and manages the necessary connection pools to your database so that your application creates fewer database connections.
You can use RDS Proxy for any application that makes SQL calls to your database. But in the context of serverless, we focus on how this improves the Lambda experience.
https://aws.amazon.com/de/blogs/compute/using-amazon-rds-proxy-with-aws-lambda/
Related
I built a multi-tenant microservice architecture in Nest.js, multi-tenant connection are made using TypeOrm latest Datasource API. After upgrading to latest TypeORM version we encountering the MySQL "Too many connections" error.
During searching about this I found that in their latest version they added option "PoolSize" to control number of active connections. I've added that too but the issue is still there.
Ideally, TypeOrm should close the connection once the DB operation finished or use opened connection (if any) on new request, but the connection to MySQL is keeping active but in sleep state, and on a new request it create new connection. see below:
By running show processlist; cmd on MySQL
I've created the multi-tenant connection using nest.js provider for incoming request in microservice:
databaseSource is used for initial database connection to default database and then on each tenant request we create the new DB connection.
export const databaseSource = new DataSource({
type: process.env.DB_CONNECTION,
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT, 10),
username: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: process.env.DB_DATABASE,
entities: ["src/models/entities/**/*.ts"],
synchronize: true,
poolSize: 10,
});
const connectionFactory = {
provide: CONNECTION,
scope: Scope.REQUEST,
useFactory: async (request: RequestContext) => {
const tenantHost = request.data["tenantId"] || request.data;
if (tenantHost) {
const tenantConnection: DataSource = await getTenantConnection(
tenantHost
);
return tenantConnection;
}
return null;
},
inject: [REQUEST],
};
#Global()
#Module({
providers: [connectionFactory],
exports: [CONNECTION],
})
export class TenancyModule {}
export async function getTenantConnection(
tenantHost: string
): Promise<DataSource> {
const tenantId = getTenantId(tenantHost);
const connectionName = `${tenantId}`;
const DBConfig: DataSourceOptions = {
type: process.env.DB_CONNECTION,
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT, 10),
username: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: connectionName,
entities: ["src/models/entities/**/*.ts"],
synchronize: true,
poolSize: 10,
};
const dataSource = new DataSource(DBConfig);
if (!dataSource.isInitialized) {
await dataSource.initialize();
}
return dataSource;
}
Then once datasource initalized,I inject it into Service and used it to getRepository and performed DB operation.
I researched a lot about this some saying increase the MySQL "max_connections" limit, some saying passed the "connectionLimit" options in TypeOrm config (poolSize in latest version) but nothing works for me.
Am I doing anything wrong to create the tenant connection?
Is there any way to closed the connection manually after DB operation?
Your error doesn't have to do anything with typeorm version. It's most likely that the number of tenants have increased and the way you're creating the connection, you're going to run out of connections if not now then later.
There are a number of things that you can do to make it work. First of them all would be to limit the number of connections per tenant. The correct parameter to limit the number of connections in a pool is to use connectionLimit parameter inside extra object for typeorm versions < 0.3.10 and poolSize for typeorm versions >= 0.3.10.
TypeORM is just an ORM, it delegates underlying communication to the database to the corresponding driver. In case of mysql, it uses this mysql npm module. Whatever option that you specify in dataSourceOptions is passed onto the driver. You can see the set of available pool options. You might wanna set this number to a very small value as you're going to have multiple tenants. Perhaps keep this value configurable for every tenant. A smaller number for a not so big tenant and a larger value for a very busy one. This way you'll be able to reduce overall connections pressure on your database server.
Talking about the screenshot you've pasted with high number of connections in sleep command, this is mostly due to the pool of connections made by your application. This doesn't pose any harm unless it surpasses the max_connections variable on your mysql database server. In your case, it has happened indeed that's why the error: Too many connections.
Other options you might explore is to increase the value of the variable max_connections so that you're able to accommodate all your tenants. You might also wanna increase the server size as increasing this variable will increase RAM usage, unless of course mysql is already running on a very big machine.
Edit 1: After reading your comment, I see a probable issue at this line:
if (databaseSource.options.database === connectionName) {
//
}
databaseSource.options.database will always be equal to process.env.DB_DATABASE when the databaseSource is first initialised. Upon any subsequent request for connection for any tenantId, this check will fail and every time a new connection pool will be created.
Edit 2: Again the issue lies within your code. You're always creating a new DataSource object without checking if there is already a connection pool for that tenant. isInitialized flag will always be false for a new object and your code will do dataSource.initialize() which will create new pool. Hint: Try to keep connection pools created in a map:
const tenantPools = {
tenantId: dataSource
}
and before creating a new DataSource object, check if that already exists in this map.
I am using mysql package of npm in my NodeJS project. I am using connection pool as below -
var pool = mysql.createPool({
connectionLimit: 50,
host: host,
user: user,
password: password,
database: database
});
And then I am using the pool as -
pool.query("Select ....", function (err, data) {
});
But sometimes our database server is stuck due to large queries and I think the connection limit of this connection gets crossed. Then after the stuck queries have executed successfully, the mysql library cannot acquire new connections. I cannot even see the queries in SHOW PROCESSLIST of MySQL. So there is issue in acquiring new connections. There is nothing in the logs too. I sort the issue by restarting the Node Server, but it isnt the ideal solution. Please help me in identifying the cause of the issue. Similar issue occurs with MSSQL connections in NodeJS and I just cannot identify the reason for this.
After you're done processing your query, you should send pool.end() to close the current connection.
I have a Node.js application using MySQL on an AWS RDS with Bookshelf & Knex libraries. The RDS Instance have a max_connections value 90.
I am using the following as the connection object.
knex: {
client: 'mysql',
connection: {
host: 'xxxxxxx.rds.amazonaws.com',
user: 'xxx',
password: 'xxxxx',
database: 'xxxx',
charset: 'utf8'
},
debug: true,
pool: {
min: 2,
max: 20
},
acquireConnectionTimeout: 10000
},
const config = require('./environment');
const knex = require('knex')(config.knex);
module.exports = require('bookshelf')(knex).plugin('registry');
'use strict';
const bookshelf = require('../config/bookshelf');
const config = require('../config/environment');
module.exports = bookshelf.model(`TableA`, {
tableName: 'TableA'
}, {});
I have many requests coming along to the application and sometimes crashes with the following errors.
Unhandled rejection TimeoutError: Knex: Timeout acquiring a
connection. The pool is probably full. Are you missing a
.transacting(trx) call?
and
Error: ER_CON_COUNT_ERROR: Too many connections
Also I see a number of connections (40 to 50 on an average) in the server PROCESSLIST with Command as sleep.
I suspect these error happen when all the 90 connections on the server used fully / knex cannot acquire a new connection from he pool when it tries to. What could be a permanent solution for this, and best practices for handling these kind of applications.
I dont think it is the RDS max_connections that is causing the issue, assuming you only have one instance of the above application code running at any time.
Your application uses a DB connection pool, which can hold up to 20 connections. If all those connections are in use, then the application waits for up to acquireConnectionTimeout ms in your case that is 10000 before connection timeout.
So I suspect that your application either has a lot of db queries to be processed due to load or there are some slow queries hogging connections. This causes a backlog of queries waiting for connections that eventually times out. Investigate which might be the case and do update us.
Things you can try in the mean time.
Increase acquireConnectionTimeout.
Increase connection pool size.
If caused by slow queries then optimize them before trying the above.
Possible methods for logging slow queries:
Enable slow query log on RDS.
Knex query event to log transaction duration assuming you are using transactions.
When a client is finished with MySQL, have it disconnect.
Also, check the value of wait_timeout. Lowering it will force disconnections rather than "Sleeping" until you come back.
I'm trying to connect to my Aurora Serverless MySQL DB cluster using the mysql module, but my connection always times out.
const mysql = require('mysql');
//create connection
const db = mysql.createConnection({
host : 'database endpoint',
user : 'root',
password : 'pass',
database : 'testdb'
});
//connect
db.connect((err) => {
if(err){
throw err;
console.log('connection failed');
}
console.log('mysql connected...');
})
db.end();
My cluster doesn't have a public IP address so I'm trying to use the endpoint. I've successfully connected to the db using Cloud9, but I can't connect using node. I must be missing something.
Aurora Serverless uses an internal AWS networking setup that currently only supports connections from inside a VPC, and it must be the same VPC where the serverless cluster is deployed.
Q: How do I connect to an Aurora Serverless DB cluster?
You access an Aurora Serverless DB cluster from within a client application runing in the same Amazon Virtual Private Cloud (VPC). You can't give an Aurora Serverless DB cluster a public IP address.
https://aws.amazon.com/rds/aurora/faqs/#serverless
This same limitation applies to Amazon EFS, for architecturally similar reasons. You can work around the limitation in EFS, and the same workaround could be used for Aurora Serverless, but you'd need to disable the health checks entirely since those health checking connections would keep the instance alive all the time. Exposing a database to the Internet is a practice best avoided.
You could also use some VPN solutions. They would need to be instance-based and would probably need to use NAT to masquerade the client address behind the VPN instance's internal address -- that's effectively what the proxy workaround mentioned above does, but at a different OSI layer.
I'm trying to utilize Knex with the Loopback framework. Currently, Loopback does not provide a good way to create advanced queries.
I'm using Knex's query builder, however by default Knex will initialize its own connection pool ON TOP of Loopback's. Instead, I want to use the connection pool already created by Loopback.
I've tried to use Knex's .connection() method to set the connection to be the one from loopback, however when I monitor the processes on my MySql server I notice that each time I make a call that uses Knex a new connection is created. Over time this is causing my server to run out of connections to the database.
I'm doing something like this:
var knex = require('knex')({
client: 'mysql',
connection: {
host: mysql.host,
port: mysql.port,
user: mysql.username,
password: mysql.password,
database: mysql.database,
debug: false
}
});
app.datasources.mysqldb.client.getConnection(function(err, connection){
knex.connection(connection).
//continue with the query building
}
My question is, how do I utilize Loopback's existing connection with Knex so that Knex doesn't burn through all the available connections in my database? I've also tried using knex's "pool" configuration but it doesnt seem to do anything...