I am currently working on a project that utilizes Next.js and I am looking to establish a connection with a MySQL database. To accomplish this, I have chosen to use Sequelize as my ORM. However, I am uncertain about the appropriate location to authenticate access to the database.
According to the documentation, in order to connect to the database, it is necessary to define a new instance of Sequelize using the following syntax:
const sequelize = new Sequelize(database, username, password, {
host: 'localhost',
dialect: 'mysql',
});
After defining the new instance of Sequelize, it is necessary to authenticate the connection to the database using the db.authenticate() method. In an Express application, this is typically executed in the server.js file. However, in Next.js, it cannot be placed in the server.js file as it would require the creation of a custom server, which I would like to avoid in order to retain other features such as optimization. That is why I have chosen to place the db.authenticate() method immediately after the code that initializes the instance of Sequelize. In summary, the file where I initialize Sequelize and authenticate the connection looks something like this:
import { Sequelize } from "sequelize";
const sequelize = new Sequelize(database, username, password, {
host: "localhost",
dialect: "mysql",
});
db.authenticate()
.then(() => {
console.log("Connection has been established successfully.");
})
.catch((err) => {
console.error("Unable to connect to the database:", err);
});
export default db;
I am concerned that placing the db.authenticate() method in this location may result in inefficiency, as it may cause Sequelize to re-authenticate the connection each time a query is made to the database. I am worried that this could have a negative impact on performance, as it would not save the connection that was made the first time. Will this be the case? If so, do you have any recommendations on how to optimize this situation.
Are there any alternative ORMs for MySQL that are compatible with Next.js that you would recommend over Sequelize?
Related
I built a multi-tenant microservice architecture in Nest.js, multi-tenant connection are made using TypeOrm latest Datasource API. After upgrading to latest TypeORM version we encountering the MySQL "Too many connections" error.
During searching about this I found that in their latest version they added option "PoolSize" to control number of active connections. I've added that too but the issue is still there.
Ideally, TypeOrm should close the connection once the DB operation finished or use opened connection (if any) on new request, but the connection to MySQL is keeping active but in sleep state, and on a new request it create new connection. see below:
By running show processlist; cmd on MySQL
I've created the multi-tenant connection using nest.js provider for incoming request in microservice:
databaseSource is used for initial database connection to default database and then on each tenant request we create the new DB connection.
export const databaseSource = new DataSource({
type: process.env.DB_CONNECTION,
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT, 10),
username: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: process.env.DB_DATABASE,
entities: ["src/models/entities/**/*.ts"],
synchronize: true,
poolSize: 10,
});
const connectionFactory = {
provide: CONNECTION,
scope: Scope.REQUEST,
useFactory: async (request: RequestContext) => {
const tenantHost = request.data["tenantId"] || request.data;
if (tenantHost) {
const tenantConnection: DataSource = await getTenantConnection(
tenantHost
);
return tenantConnection;
}
return null;
},
inject: [REQUEST],
};
#Global()
#Module({
providers: [connectionFactory],
exports: [CONNECTION],
})
export class TenancyModule {}
export async function getTenantConnection(
tenantHost: string
): Promise<DataSource> {
const tenantId = getTenantId(tenantHost);
const connectionName = `${tenantId}`;
const DBConfig: DataSourceOptions = {
type: process.env.DB_CONNECTION,
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT, 10),
username: process.env.DB_USERNAME,
password: process.env.DB_PASSWORD,
database: connectionName,
entities: ["src/models/entities/**/*.ts"],
synchronize: true,
poolSize: 10,
};
const dataSource = new DataSource(DBConfig);
if (!dataSource.isInitialized) {
await dataSource.initialize();
}
return dataSource;
}
Then once datasource initalized,I inject it into Service and used it to getRepository and performed DB operation.
I researched a lot about this some saying increase the MySQL "max_connections" limit, some saying passed the "connectionLimit" options in TypeOrm config (poolSize in latest version) but nothing works for me.
Am I doing anything wrong to create the tenant connection?
Is there any way to closed the connection manually after DB operation?
Your error doesn't have to do anything with typeorm version. It's most likely that the number of tenants have increased and the way you're creating the connection, you're going to run out of connections if not now then later.
There are a number of things that you can do to make it work. First of them all would be to limit the number of connections per tenant. The correct parameter to limit the number of connections in a pool is to use connectionLimit parameter inside extra object for typeorm versions < 0.3.10 and poolSize for typeorm versions >= 0.3.10.
TypeORM is just an ORM, it delegates underlying communication to the database to the corresponding driver. In case of mysql, it uses this mysql npm module. Whatever option that you specify in dataSourceOptions is passed onto the driver. You can see the set of available pool options. You might wanna set this number to a very small value as you're going to have multiple tenants. Perhaps keep this value configurable for every tenant. A smaller number for a not so big tenant and a larger value for a very busy one. This way you'll be able to reduce overall connections pressure on your database server.
Talking about the screenshot you've pasted with high number of connections in sleep command, this is mostly due to the pool of connections made by your application. This doesn't pose any harm unless it surpasses the max_connections variable on your mysql database server. In your case, it has happened indeed that's why the error: Too many connections.
Other options you might explore is to increase the value of the variable max_connections so that you're able to accommodate all your tenants. You might also wanna increase the server size as increasing this variable will increase RAM usage, unless of course mysql is already running on a very big machine.
Edit 1: After reading your comment, I see a probable issue at this line:
if (databaseSource.options.database === connectionName) {
//
}
databaseSource.options.database will always be equal to process.env.DB_DATABASE when the databaseSource is first initialised. Upon any subsequent request for connection for any tenantId, this check will fail and every time a new connection pool will be created.
Edit 2: Again the issue lies within your code. You're always creating a new DataSource object without checking if there is already a connection pool for that tenant. isInitialized flag will always be false for a new object and your code will do dataSource.initialize() which will create new pool. Hint: Try to keep connection pools created in a map:
const tenantPools = {
tenantId: dataSource
}
and before creating a new DataSource object, check if that already exists in this map.
I built a program with NodeJS where multiple users access it in the same time and do a lot of operations that queries the MySQL database.
My approach is very simple. I only open one connection when the app is started and leave it that way.
const dbConfig = require('./db-config');
const mysql = require('mysql');
// Create mySQL Connection
const db = mysql.createConnection({
host: dbConfig.host,
user: dbConfig.user,
password: dbConfig.password,
database: dbConfig.database,
multipleStatements: true
});
// Connect MySQL
db.connect((err) => {
if (err) {
throw err;
} else {
console.log('MySQL connected!');
}
});
module.exports = db;
And then, whenever the program needs to query the database, i do like this
db.query('query_in_here', (error, result) => {
*error_handling_and_doing_stuff*
}
I'm having trouble when noone access the app for a long period of time (some hours).
Because when this happens i think the connection is being closed automatically. And then, when a user try to access the app, i see in the console that the connection timed out.
My first thought was too handle the disconnection and connect again. But, it get me thinking if this is the correct approach.
Should i use pool connections instead? Because if i keep only one connection it means that two users can't query the database in the same time?
I tried to understand tutorials with pool connections but couldn't figure out when to create new connections and when should i end them.
UPDATE 1
Instead of create one connection when the app is started i changed to create a pool connection.
const dbConfig = require('./db-config');
const mysql = require('mysql');
// Create mySQL Connection
const db = mysql.createPool({
host: dbConfig.host,
user: dbConfig.user,
password: dbConfig.password,
database: dbConfig.database,
multipleStatements: true
});
module.exports = db;
It seems that when i use now "db.query(....)" the mysql connection and release of that connection is done automatically.
So, it should resolve my issue but i don't know if this is the correct approach.
Should i use pool connections instead?
Yes you should. Pooling is supported out-of-the-box with the mysql module.
var mysql = require('mysql');
var pool = mysql.createPool({
connectionLimit : 10,
host : 'example.org',
user : 'bob',
password : 'secret',
database : 'my_db'
});
pool.query('SELECT 1 + 1 AS solution', function (error, results, fields) {
// should actually use an error-first callback to propagate the error, but anyway...
if (error) return console.error(error);
console.log('The solution is: ', results[0].solution);
});
You're not supposed to know how pooling works. It's abstracted from you. All you need to do is use pool to dispatch queries. How it works internally is not something you're required to understand.
What you should pay attention to is the connectionLimit configuration option. This should match your MySQL server connection limit (minus one, in case you want to connect to it yourself while your application is running), otherwise you'll get "too many connections" errors. The default connection limit for MySQL is 100, so I'd suggest you set connectionLimit to 99.
Because if i keep only one connection it means that two users can't query the database in the same time?
Without pooling, you can't serve multiple user requests in-parallel. It's a must have for any non-hobby, data-driven application.
Now, if you really want to know how connection pooling works, this article sums it up pretty nicely.
In software engineering, a connection pool is a cache of database connections maintained so that the connections can be reused when future requests to the database are required. Connection pools are used to enhance the performance of executing commands on a database. Opening and maintaining a database connection for each user, especially requests made to a dynamic database-driven website application, is costly and wastes resources. In connection pooling, after a connection is created, it is placed in the pool and it is used again so that a new connection does not have to be established. If all the connections are being used, a new connection is made and is added to the pool. Connection pooling also cuts down on the amount of time a user must wait to establish a connection to the database.
I wonder what is the optimal way to establish/maintain connection with MySQL/Redis (from nodejs): store in one single object (conn) or create a new connection on every request? Namely:
1, Should we use a single connection for every nodejs http request? Use connection pool? Or a single connection on every new request (so reconnection should be important because the connection could be randomly lost)? How is the performance?
2, What is the difference between MySQL and Redis in term of maintaining such connection?
I will tell you how I used to manage to do this.
1, Should we use a single connection for every nodejs http request? Use connection pool? Or a single connection on every new request (so reconnection should be important because the connection could be randomly lost)? How is the performance?
You don't want to create connections manually for every nodejs http request. Always use connection pooling if you are using nodejs mysqlijs/mysql module. I use it.
Pools take care of server reconnections automatically.
I don't have a benchmarks, but performance should be better because within pools connections can be reused once released. In other factors, believe me, creating and managing connections manually is cumbersome and error-prone.
Eg:
Declare your mysql connection pool in a Db.js file like below and export it.
var mysql = require("mysql");
var pool = mysql.createPool({
connectionLimit : 5,
host : process.env.DB_HOST || 'localhost',
user : process.env.DB_USER,
password : process.env.DB_PASSWORD,
database : 'mydb'
});
module.exports = db;
And use it in your inside an end-point in another file.
var pool = require('./Db.js');
app.get('/endpoint', function (req, res) {
// ...
pool.query('<your-query-here>', function (err, res, fields) {
if (err) throw err;
// do something with result (res)
});
});
I prefer using both pool.query and pool.getConnection based on the scenario. Using query is safer because you don't need to consider releasing connection. It will be automatically handled by the library. getConnection is used only where several queries has to be run inside an end-point, so I can reuse the same connection to do that.
2, What is the difference between MySQL and Redis in term of maintaining such connection?
In short, You don't need pooling for redis. We don't think about pooling redis connections according to this.
I am trying to achieve Oauth2 authentication and I came across a GitHub Repo which fits my description but the database used is MongoDB and I want to use MySql.
I want to convert this code of MongoDB Authentication to MySQL
const mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/MyDatabase');
const Schema = mongoose.Schema;
const UserDetail = new Schema({
username: String,
password: String
});
const UserDetails = mongoose.model('userInfo', UserDetail, 'userInfo');
I have written this much code till now
var mysql = require('mysql');
var connection = mysql.createConnection({
supportBigNumbers: true,
bigNumberStrings: true,
host : "localhost",
user : "root",
password : "root",
database : "db_users"
});
Please refer the following GitHub repo for full code
https://github.com/sitepoint-editors/LocalPassportAuth
What should I add in this code to make it work?
I had great success answering this question using the following NPM package
https://github.com/aponica/mysqlgoose-js
It saved me allot of time migrating my NodeJS authentication API's to using MySQL as my back-end database instead of using MongoDB after my free mLabs account was deactivated and MongoDB was forcing me to a paid plan.
The steps for me were
Create all MySql tables using my MongoDb models as a template
Create a mysql.json connection file allowing a connection from NodeJS to MySql
Use the NPX Import tool to create a definitions JSON file of my MySQL schema that I load at runtime in NodeJS
Build a promise-based initialization function in my routes that connects with MySql and loads the defs tile to perform mongoose-like operations on my MySql tables
Make sure to close the MySQL connection upon sending a response from my API
I recommend you to add the connection in a separate file.
After this what are your main issue ? Because the mysql code is correct for a connection.
With connection.connect() it's now working.
I'm using node-mysql as the orm. (https://github.com/mysqljs/mysql)
I'm creating a REST API using MySQL and Node.JS. Earlier I did the same using MongoDB. And it's working fine. github.com/chanakaDe/Mint-REST. Now I want to do the same using MySQL. I did most of the part and having a little issue. There are many classes and I need to use mysql connection in a centralized way. As singleton design pattern.
This is the my new repo. https://github.com/chanakaDe/MintRestSQL. And I will show where I want to use those patterns. Here I have the database file. And I created a new connection pool. github.com/chanakaDe/MintRestSQL/blob/master/app/util/database.js.
Now I want to use this connection/pool inside my controller classes. Because I cannot create a connection in each and every controller class. No ? These are my two controllers for now.
github.com/chanakaDe/MintRestSQL/blob/master/app/routes/report.js
github.com/chanakaDe/MintRestSQL/blob/master/app/routes/api.js
Please show me a better way to do this. I am new to node-mysql. But it's a great way to use MySQL inside Node.JS environments even for production grade systems. So I want to make a good API using those standards. Is there any way to use singleton pattern or something like that and centralized the connection and use it in all the controllers ????
IT WILL TAKE SOME TIME TO CHECK MY FILES AND UNDERSTAND THE CODE. BUT PLEASE CHECK IT AND GIVE ME A SOLUTION. I TRIED MANY THINGS, BUT DIDN'T WORK :(
You are welcome to pull the repo and make any update
Try change database.js to
var mysql = require('mysql');
mysql.createPool({
host : 'localhost',
user : 'root',
password : 'chanaka',
database : 'shared_adult'
}).connect();
module.exports = mysql;
To offer an alternative to anyone arriving here (like me) that needed to inject a dependency (i.e. the credentials). Below is the approach that will yield a runtime confiurable singleton:
var mysql = require('mysql'),
db;
module.exports = {
init: function(conf){
if(!db){
db = mysql.createPool({
host: conf.host,
user: conf.root,
password: conf.password,
database: conf.database
});
}
},
get: function() {
if(!db) {
throw new Error('The db pool has not been initialized, call init({}) prior to get().');
}
return db;
}
};
//initialize where you access your config (which should theoretically only be one place)
var mysqlDb = require('db'); //path to above db module
mysqlDb.init({
host: 'host',
user: 'user',
password: 'password',
database: 'database'
});
//use in your modules
var mysqlDb = require('db'), //path to above db module
db = mysqlDb.get();