I am having trouble with pooling mysql connections with nodeJS. As I understand it, when creating a pool, I should be able to use pool.query() to:
Get a connection
Run my query
Release the connection
However, the issue I am running into is the connections remain open and "sleeps" until the server closes the connection itself (60 seconds). Is there something I am doing wrong here? For context, I am connecting to Heroku clearDB and am viewing the connection on its dashboard.
Example: Upon login, I make a query to check login credentials to authenticate, another query to fetch one set of data, and another query to fetch another set. After logging in, the server is left with 2 connection in "sleep" mode. They do not disconnect until 60 seconds expire.
Here is my db.js file:
const mysql = require("mysql");
const dbConfig = require("./db.config.js");
const connection = mysql.createPool({
host: dbConfig.HOST,
user: dbConfig.USER,
password: dbConfig.PASSWORD,
database: dbConfig.DB,
port: dbConfig.PORT,
multipleStatements: true,
connectionLimit: 10
})
module.exports = connection;
Here is how I am making queries:
const sql = require("./db.js"); //Our db connection
---OMITTED---
return new Promise((resolve, reject) => {
const select_user = "SELECT user_id, first_name, last_name, username, password, is_admin, is_active FROM users WHERE username = ? LIMIT 1"
const params = [username]
const user_pass = password
sql.query(select_user, params, (err, res) => {
if (res) {
if (res.length) {
const user = res[0]
const hash = user.password
const is_active = user.is_active
if (is_active) {
bcrypt.compare(user_pass, hash).then(res => {
if (res) {
resolve({ result: true, info: { fname: user.first_name, lname: user.last_name, username: user.username, is_admin: user.is_admin, user_id: user.user_id } })
} else {
reject({ result: false, msg: "wrong_creds" })
}
}).catch(err => {
reject({ result: false, msg: err })
})
} else {
reject({ result: false, msg: "inactive" })
}
} else {
reject({ result: false, msg: "user_not_exist" })
}
} else {
console.log("MYSQL QUERY, COULD NOT SELECT USER FOR AUTHENTICATION")
reject({ result: false, msg: err })
}
})
})
I'm under the impression that pool.query() will close the connection for me after each query. Is this not the case? Am I missing something once the query has been completed?
I have also tried the manual way:
const select_string = "SELECT * FROM user_backlogs WHERE movie_id = ? AND user_id = ?"
const params = [movie_id, user_id]
sql.getConnection((err, connection) => {
if (err) reject(err)
else {
connection.query(select_string, params, (err, res) => {
connection.release()
if (err) reject(err)
else {
if (res.length) {
resolve([movie_id, "exists", res[0]["added_on"]])
} else {
resolve([movie_id, "not_exists"])
}
}
})
}
})
But still, the connections remain up until the server kicks them off. Any help is greatly appreciated.
The pool will not close the connection. It will release the connection, which me means it can be reused by another query. The pool does this to optimize performance, since it takes time to open a new connection.
If you want to close the connection after use, you can explicitly destroy it (connection.destroy()) and the pool will create a new one the next time you issue a query. You can find more information on this in the documentation under pooling connections.
Related
I'm having a little trouble with my site and I can't understand what is happening.
First of all I have to say that I was NOT having this behavior when developing on localhost, but now that my site is close to be completed I think that uploading my code to a hosting service and make some tests there would be a good idea.
The issue is that when I make a request to the database, most of the times the site keeps in an eternal loading state, until the error code 500: Internal Server Error appears (I said "most of the times" because it works nice sometime, but normally it remains in a pending state).
Given the fact that SOME TIMES the request work nice, makes me think that the issue is not on the server.js file (where I defined the endpoints), and also is not on my controllers files (where I have some logic and the requests itself).
I'll leave here some pics as example of what is happening but if you need some extra info just tell me:
A simple login example, I just fill the fields and send the request
And here you can see how the request remain as pending
Until it fails
EDIT: I'm using package Mysql2 to connect to the DB, and I was reading that this behavior may be because a bad use of connections (and I'm reading about "pools", but I'm kinda lost tbh)
Here is the connection file:
require("dotenv").config();
const mysql = require("mysql2");
const db = mysql.createConnection({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
waitForConnections: true,
});
const connection = async () => {
db.connect((err) => {
if (err) throw err;
console.log("Successfully connected");
})
}
exports.db = db;
exports.connection = connection;
The first call to the DB (just to check the connection)
connection().then(() => {
app.listen(port, () => {
console.log(`Server running at ...`);
});
});
And the login logic
app.post("/dev-end/api/login", async (req, res) => {
await singleAccount(db, req.body.email)
.then(async (response) => {
if (response.code) {
res.render("templateLogin");
}
try {
if (await bcrypt.compare(req.body.password, response.password)) {
const user = { id: response._id, name: response.name };
await deleteTokenById(db, user.id.toString());
const accessToken = generateAccessToken(user);
const refreshToken = jwt.sign(
user,
process.env.REFRESH_TOKEN_SECRET,
{ expiresIn: "604800s" }
);
createToken(db, {
_id: user.id,
accessToken: accessToken,
refreshToken: refreshToken,
createdAt: new Date().toISOString().slice(0, 19).replace("T", " "),
}).then(
res
.cookie("access_token", accessToken, {
httpOnly: true,
maxAge: 60000 * 60 * 24 * 7,
})
.redirect("/dev-end/dashboard")
);
} else {
res.render("templateLogin");
}
} catch {
res.status(500).send();
}
})
.catch(console.log);
});
=>=>=>=>=>=>=>=>=>=>=>=>=>=>=>=>
const singleAccount = async (conn, email) => {
return await read(conn).then((res) => {
if (!res.code) {
const result = res.find((e) => e.email.toString() === email);
if (!result) {
return {
code: 404,
msg: "No account was found with the provided id",
};
}
return result;
}
return res;
});
};
=>=>=>=>=>=>=>=>=>=>=>=>=>=>=>=>
const read = async (conn) => {
const sql = `SELECT * FROM accounts`;
return await conn.promise().query(sql)
.then(([res, fields]) => res);
};
I'm having trouble getting my cookies in the browser using Express, MySQL and Express-MySQL-Session. I made sure that my browsers are allowing cookies, I made sure that my store is connected properly. When I or anyone logs into the app I attach some user data to the session object. I can console log it and see it. It also shows in my remote MySQL DB.
Here is a sample of my code
// sessions
const session = require('express-session');
// mysql and its store
const mysql = require('mysql');
const MySQLStore = require('express-mysql-session')(session);
// create session store
const connection = mysql.createPool({
host: process.env.HOST,
port: 3306,
user: process.env.USER,
password: process.env.PASSWORD,
database: process.env.DATABASE,
createDatabaseTable: true,
});
const sessionStore = new MySQLStore({connectionLimit: 10}, connection);
// session
app.use(session({
secret: process.env.SECRET,
resave: false,
saveUninitialized: false,
cookie: {
maxAge: 1000 * 60 * 60 * 24
},
store: sessionStore
}));
// login
app.post('/api/v1/login', (req, res) => {
// get input from front
const { username, password } = req.body;
// find user
db.query(`SELECT * from users WHERE username = ?`, username, async (err, user) => {
if(err) throw err;
// user not found
if(!user) {
res.status(404).json({message: "User Not Found"})
} else {
// compare passwords
console.log(user)
const matchedPassword = await bcrypt.compare(password, user[0].hashPassword);
// password doesn't match
if(matchedPassword === false) {
res.json({message: "Bad Credentials", user, matchedPassword})
} else {
// user found
req.session.user = user[0].username;
console.log(req.session);
res.status(200).send({message: 'Success' , user})
}
}
})
});
// register
app.post('/api/v1/register', async (req, res, next) => {
// get input from user
const { username, password } = req.body;
try {
// if user already exists
db.query(`SELECT username FROM users WHERE username = ?`, username, (err, user) => {
if (err) res.status(400).json({message: 'Error Occurred'})
if (!user) res.status(400).json({message: 'User Not Found'});
});
// create user
const salt = await bcrypt.genSalt(2);
const hashedPassword = await bcrypt.hash(password, salt);
db.query(`INSERT INTO users (username, hashPassword) VALUES (?, ?)`, [username, hashedPassword], (err, user) => {
if (err) next(err);
res.status(200).json({message: 'User Created!', user});
})
} catch (err) {
next(err);
}
});
Every time I visit the page a new session is created in the DB despite the configuration of the session object.
UPDATE
I think my problem may have to do with my backend being hosted on Heroku, but the frontend being hosted on Netlify. When I make a request through postman I see that it returning a cookie with the Session ID, but thats it.
SOLVED
So, I looked into setting the domain attribute of my cookies to other domain where the frontend is hosted but apparently this isn't allowed. Makes sense. So I have to choose between hosting the whole site in one place, or forgetting about sessions and cookies for this small project.
i'm learning node and have hit an issue with running a mysql query. I am certain i am getting connectivity to the mysql db (as when i run the query from the same module as i connect to the DB i am able to print the rows to the console log). However i get a 500 error when i require the DB connection into my another file and use in the query. I've spent last few hours on stack overflow and still stuck. Its definitely not the function as if i just return text it works fine so its definitely something around the way i am utilising the db connection. The DB connection file is below:
// initialize database connection
const mysql = require('mysql')
var db;
function connectDatabase() {
if (!db) {
db = mysql.createConnection({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASS,
database: process.env.DB_DATABASE
})
db.connect(function(err){
if(!err) {
console.log('Database is connected!')
} else {
console.log('Error connecting database!')
}
})
}
return db
}
module.exports = connectDatabase()
The module where the DB connection us used in query below:
const db = require('../db').connectDatabase
exports.getNightlyRatesData = () => {
//const { rows } = db.query('SELECT * FROM rate', function (err, rows) {
db.query('SELECT * FROM rate', function (err, rows) {
if (err) {
console.log("ERROR");
console.log(err);
return;
}
console.log("good");
})
return rows
}
I have an API server which runs different API calls to a mysql database: this is the file which entirely handles the connections and the queries:
const mysql = require('mysql')
const pool = mysql.createPool({
host: sqlhost,
port: sqlport,
user: sqluser,
password: sqlpsw,
database: sqldb,
charset: 'utf8mb4',
connectionLimit: 10
})
module.exports = {
get: function (sqlQuery, type) {
return new Promise((resolve, reject) => {
pool.getConnection((err, connection) => {
if (err) reject(new Error(505))
else {
connection.query(sqlQuery, function (err, results) {
connection.release()
if (err !== null) reject(new Error(503))
else resolve(results)
})
}
})
})
}
}
After some time I get the Error: ER_CON_COUNT_ERROR: Too many connections.
I know i may have used pool.query instead of the flow pool.getConnection --> connection.query --> connection.release but the result is (I guess) the same.
I have no idea what is causing the "connections leak". Any idea? Thanks
My very simple Node.js code doesn't seem like its connection pool work as it's supposed to do. _connectionQueue of Pool object just gets longer and longer infinitely, and app dies. I mean it does make a pool and there are pre-made connections already, but they are not reusable or insert requests are too many and fast? I'm not sure..
I've tried to put some more connectionLimit like following :
let state = { pool: null }
export const connect = () => {
state.pool = mysql.createPool({
connectionLimit: 200,
host: process.env.DATABASE_HOST || 'localhost',
user: process.env.DATABASE_USER || 'root',
password: process.env.DATABASE_PASSWORD || 'password',
database: process.env.DATABASE_NAME || 'database'
})
}
export const get = () => state.pool
Mostly given job of this server is subscription and insertion. It subscribes several MQTT topics and just tries to insert messages into RDB. About 100 messages arrives every second, and that code looks like this.
mqttClient.on('message', function (topic, message) {
if(topic.includes('sensor')){
try {
const data = JSON.parse(message.toString())
if(validate(data.uuid)){
const params = [data.a, data.b, data.c, ...]
sensor.setStatus(params)
}
} catch(err){
console.error(err)
}
}
}
export const setStatus = (params) => {
const SQL = `INSERT INTO ...`
db.get().query(SQL, params, (err, result) => {
if (err) console.error(err)
})
}
Then, I see this through chrome-devtools
Object
pool: Pool
config: PoolConfig {acquireTimeout: 10000, connectionConfig: ConnectionConfig, waitForConnections: true, connectionLimit: 200, queueLimit: 0}
domain: null
_acquiringConnections: []
_allConnections: (200) [PoolConnection, PoolConnection, …]
_closed: false
_connectionQueue: (11561) [ƒ, ƒ, ƒ, ƒ, …]
_events: {}
_eventsCount: 0
_freeConnections: []
_maxListeners: undefined
__proto__: EventEmitter
__proto__: Object
I've put console.log into setStatus like following :
export const setStatus = (params) => {
const SQL = `INSERT INTO ...`
console.log(`allConnections=${db.get()._allConnections.length}, connectionQueue=${db.get()._connectionQueue.length}`)
db.get().query(SQL, params, (err, result) => {
if (err) console.error(err)
})
}
, and got these.
allConnections=200, connectionQueue=29
allConnections=200, connectionQueue=30
allConnections=200, connectionQueue=31
allConnections=200, connectionQueue=32
allConnections=200, connectionQueue=33
allConnections=200, connectionQueue=34
...
It seems like server created a connection pool very well, but not using those connections. Instead, trying to create a new connection more and more all the time and those requests just get stuck in _connectionQueue.
It appears you are creating a new pool every time you'd like to make a query. The common model is to create a pool once when the application starts, then use connections from that pool as needed (one pool, many connections).
Also if you're using a simple DB model you can simplify access to the pool by making it global. Below is an alternate to your code you might try:
app.js
const mysql = require('mysql');
const connection = mysql.createPool({
host: process.env.DB_HOST || '127.0.0.1',
user: process.env.DB_USER || 'local_user',
password: process.env.DB_PASSWORD || 'local_password',
database: process.env.DB_NAME || 'local_database'
});
global.db = connection;
modules.js
export const setStatus = (params) => {
let SQL = `INSERT INTO ...`
db.query(SQL, params, (err, result) => {
if (err) console.error(err)
console.log(result)
})
}
Documentation for further reference :: https://github.com/mysqljs/mysql#pooling-connections
Edit 1 - Log pool events
db.on('acquire', function (connection) {
console.log('Connection %d acquired', connection.threadId);
});
db.on('connection', function (connection) {
console.log('Pool id %d connected', connection.threadId);
});
db.on('enqueue', function () {
console.log('Waiting for available connection slot');
});
db.on('release', function (connection) {
console.log('Connection %d released', connection.threadId);
});