I am trying to create a multi tenant node.js application (with also sub domains) where each user has their own database under a single host, (say an rds instance).
So it will be easier for me to handover the database to that particular user to access that particular database.
So my problem here is when an user request the api, is it possible to dynamically change the database name based on the request.
One thing that popped up in my mind is to create a pool connection with credentials that have access to all database and add the database name in the query, in this way i can achieve what i want, but just out of curiosity is it possible to add the database name after the connection pool has been created, without adding it to the query.
My solution :
select * from 'dbname'.users
What i am looking for:
const pool = mysql.createPool({
host: process.env.DB_HOST,
user: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
// database: process.env.DB_NAME, leaving out since db name should be dynamic
connectionLimit: 5,
supportBigNumbers: true,
});
And in middleware when i need db access, i should be able to set the db name based on the request.
The reason behind this is, I have created the application and wrote the queries without the database name, now i should add the logic to every query in my application.
Reading the official docs: https://github.com/mysqljs/mysql/blob/master/Readme.md#connection-options
It is mentioned that the database option is optional.
You can discard this option when initializing the pool and specify the right database in your SQL query. You cannot create pool without database configuration and then set a database to the connection. You must use the query for this purpose.
Related
I built a multi-tenant microservice architecture in Nest.js, multi-tenant connection are made using TypeOrm latest Datasource API. After upgrading to latest TypeORM version we encountering the MySQL "Too many connections" error.
During searching about this I found that in their latest version they added option "PoolSize" to control number of active connections. I've added that too but the issue is still there.
Ideally, TypeOrm should close the connection once the DB operation finished or use opened connection (if any) on new request, but the connection to MySQL is keeping active but in sleep state, and on a new request it create new connection. see below:
By running show processlist; cmd on MySQL
I've created the multi-tenant connection using nest.js provider for incoming request in microservice:
databaseSource is used for initial database connection to default database and then on each tenant request we create the new DB connection.
export const databaseSource = new DataSource({
type: process.env.DB_CONNECTION,
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT, 10),
username: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: process.env.DB_DATABASE,
entities: ["src/models/entities/**/*.ts"],
synchronize: true,
poolSize: 10,
});
const connectionFactory = {
provide: CONNECTION,
scope: Scope.REQUEST,
useFactory: async (request: RequestContext) => {
const tenantHost = request.data["tenantId"] || request.data;
if (tenantHost) {
const tenantConnection: DataSource = await getTenantConnection(
tenantHost
);
return tenantConnection;
}
return null;
},
inject: [REQUEST],
};
#Global()
#Module({
providers: [connectionFactory],
exports: [CONNECTION],
})
export class TenancyModule {}
export async function getTenantConnection(
tenantHost: string
): Promise<DataSource> {
const tenantId = getTenantId(tenantHost);
const connectionName = `${tenantId}`;
const DBConfig: DataSourceOptions = {
type: process.env.DB_CONNECTION,
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT, 10),
username: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: connectionName,
entities: ["src/models/entities/**/*.ts"],
synchronize: true,
poolSize: 10,
};
const dataSource = new DataSource(DBConfig);
if (!dataSource.isInitialized) {
await dataSource.initialize();
}
return dataSource;
}
Then once datasource initalized,I inject it into Service and used it to getRepository and performed DB operation.
I researched a lot about this some saying increase the MySQL "max_connections" limit, some saying passed the "connectionLimit" options in TypeOrm config (poolSize in latest version) but nothing works for me.
Am I doing anything wrong to create the tenant connection?
Is there any way to closed the connection manually after DB operation?
Your error doesn't have to do anything with typeorm version. It's most likely that the number of tenants have increased and the way you're creating the connection, you're going to run out of connections if not now then later.
There are a number of things that you can do to make it work. First of them all would be to limit the number of connections per tenant. The correct parameter to limit the number of connections in a pool is to use connectionLimit parameter inside extra object for typeorm versions < 0.3.10 and poolSize for typeorm versions >= 0.3.10.
TypeORM is just an ORM, it delegates underlying communication to the database to the corresponding driver. In case of mysql, it uses this mysql npm module. Whatever option that you specify in dataSourceOptions is passed onto the driver. You can see the set of available pool options. You might wanna set this number to a very small value as you're going to have multiple tenants. Perhaps keep this value configurable for every tenant. A smaller number for a not so big tenant and a larger value for a very busy one. This way you'll be able to reduce overall connections pressure on your database server.
Talking about the screenshot you've pasted with high number of connections in sleep command, this is mostly due to the pool of connections made by your application. This doesn't pose any harm unless it surpasses the max_connections variable on your mysql database server. In your case, it has happened indeed that's why the error: Too many connections.
Other options you might explore is to increase the value of the variable max_connections so that you're able to accommodate all your tenants. You might also wanna increase the server size as increasing this variable will increase RAM usage, unless of course mysql is already running on a very big machine.
Edit 1: After reading your comment, I see a probable issue at this line:
if (databaseSource.options.database === connectionName) {
//
}
databaseSource.options.database will always be equal to process.env.DB_DATABASE when the databaseSource is first initialised. Upon any subsequent request for connection for any tenantId, this check will fail and every time a new connection pool will be created.
Edit 2: Again the issue lies within your code. You're always creating a new DataSource object without checking if there is already a connection pool for that tenant. isInitialized flag will always be false for a new object and your code will do dataSource.initialize() which will create new pool. Hint: Try to keep connection pools created in a map:
const tenantPools = {
tenantId: dataSource
}
and before creating a new DataSource object, check if that already exists in this map.
I want to use multiple db with 1 connection to MySQL server with a user. I am making a web server but it will get connections from multiple hostnames like example.com example2.com and they will have a panel to manage their page. Then they will have these credentials to connect: username, password and db name. So they can integrate to other things they want. But while we are creating a pool we specify a database and I want to optimize db connection so I don't want to create multiple connections. I am using mysql2 with multiple db connections for now.
Thanks!
It worked when I used a query like this:
SELECT value FROM databaseName.tableName WHERE name="something"
I am working on a SaaS application(Node.js + AngularJS + mySql). All clients are using a common database and common app server. One of the clients requires a separate database.
In the present scenario, When the server starts, I create mySql pool variable with host, user and password in a file name commonRouter.js. Like below:-
var pool = mysql.createPool({
connectionLimit: '',
host: '<host IP>',
user: '',
password: '',
database: '<database_name>'
});
exports.mysql_pool = pool;
And on each API call I retrieve pool variable from commonRouter.js file like below:
commonRouter.mysql_pool.getConnection(function (err, connection) {
// do database operations.
});
The problem with me that how I will decide that which database should I hit for any client if there are multiple databases.
Any help will be appreciated. Thanks.
I'm trying to utilize Knex with the Loopback framework. Currently, Loopback does not provide a good way to create advanced queries.
I'm using Knex's query builder, however by default Knex will initialize its own connection pool ON TOP of Loopback's. Instead, I want to use the connection pool already created by Loopback.
I've tried to use Knex's .connection() method to set the connection to be the one from loopback, however when I monitor the processes on my MySql server I notice that each time I make a call that uses Knex a new connection is created. Over time this is causing my server to run out of connections to the database.
I'm doing something like this:
var knex = require('knex')({
client: 'mysql',
connection: {
host: mysql.host,
port: mysql.port,
user: mysql.username,
password: mysql.password,
database: mysql.database,
debug: false
}
});
app.datasources.mysqldb.client.getConnection(function(err, connection){
knex.connection(connection).
//continue with the query building
}
My question is, how do I utilize Loopback's existing connection with Knex so that Knex doesn't burn through all the available connections in my database? I've also tried using knex's "pool" configuration but it doesnt seem to do anything...
Specifically, I'm talking about MySQL Master-Slave(s) Replication. As I understand it, all of my reads should happen from a slave (or multiple slaves hid behind a load balancer) and all of my writes directly to the master.
How can I do this with node.js? I'm using the MySQL Active Record package for my database queries and I don't think that supports it natively. I supposed I can hack away at the package and have a if is write-query, then use db1 type of situation, but I wonder if there's an easier way.
You can just create two adapters like this:
var activeRecord = require('mysql-activerecord');
var readDB = new activeRecord.Adapter({
server: 'slave.db.com',
username: 'user',
password: 'pass',
database: 'db'
});
var writeDB = new activeRecord.Adapter({
server: 'master.db.com',
username: 'user',
password: 'pass',
database: 'db'
});
When you go to do an update or other writing query, use writeDB; when you go to do a select use readDB.
Another option would be to switch to another ORM that supports this out of the box. I know this isn't as nice of a solution though. Sequelize for example is widely used and supports this out of the box.