I do understand that one should be closing mysql connections on existing node process. But with long live connections like Http servers things are little different and i seems to be confusing myself.
With node-mysql (npm package) i will be doing connection.end(); in order to end connection with mysql.
Now the questions is should i be doing it every time before sending the final response to http client or how it should work ?
For Example
// assuming express app
app.get("/users",function(req,res){
connection.query("SELECT * FROM users",function(err,rows){
connection.end();
if(error){
res.send(error);
}else{
res.send(rows);
}
})
})
Personally in a huge project it is hard to maintain connection.end(); statements for every Http request. So i am interested in knowing what are the best ways to manage mysql connection in NodeJs HTTP Servers.
Related
I have a websocket node.js app (game server) that runs a multiplayer html5 game.
The game has a website also. The game server and the website are on the same Apache VPS.
The game server uses mysql to store and retrieve data using mysql pooling from the node.js mysql package.
It works fine 99% of the time, but intermittently, at a random point, it will all of a sudden stop being able to get a mysql connection.
When this happens the website stops working and shows a 500 http error. I believe that this is whats causing the problem in the game server. Because of the 500 http error, mysql can no longer be connected to and thus pool.getConnection no longer works in the game server.
I find it strange that even though Apache is throwing up a 500 error, the game server can still be accessed successfully through a websocket as usual. The only thing that appears to have stopped working inside the game server is mysql. The game client connects to the game server via websocket and the functions work correctly, except for being able to connect to mysql.
If I ctrl+c the game server to stop the node.js app (game server) then the 500 error goes away. The website instantly serves up again, and then if I restarting the game server, mysql is now working again.
Obviously something in the game server is causing this to happen. So far i cannot find what it is. I am stuck now, i've spent a full week trying everything i could think of to debug this.
After running debug mode on mysql, im seeing this;
<-- ErrorPacket
ErrorPacket {
fieldCount: 255,
errno: 1203,
sqlStateMarker: '#',
sqlState: '42000',
message: 'User (i've hidden this) already has more than
\'max_user_connections\'
active connections' }
But I have it set to 100000 connections. No way is there that many being used. Every time I finish with a connection I use connection.release() to put it back into the pool. What do I need to do to fix this?
Please, any suggestion you have to debug this is greatly appreciated!
Thanks in advance.
here is the way i'm using mysql in the game server
const mysql = require('mysql');
const pool = mysql.createPool({
connectionLimit : 100000,
host : '***********',
user : '***********',
password : "'***********",
database : '***********',
debug : true
});
pool.getConnection(function(err,connection){
if (err) {
console.log(err);
return false;
}
connection.query("select * from aTable ",function(err,rows){
if(err) {
console.log(err);
connection.release();
return false;
}
// dos stuff here
connection.release();
})
})
1 thing i am wondering, if there is an error in the top error catch block
here ->
pool.getConnection(function(err,connection){
if (err) {
console.log(err);
return false;
}
Then the connection is not released right? So that would keep a connection alive and this is whats causing the problem? over time an error happens here and there, after a random amount of time, enough of these happen and that's what is causing it? it's like a build up of open connections???
This was the mistake i was making;
pool.getConnection(function(err,connection){
if (err) {
console.log(err);
return false;
}
connection.query("select * from aTable ",function(err,rows){
if(err) {
console.log(err);
connection.release();
return false;
}
if(somethingrelevant){
// do stuff
connection.release();
}
})
})
And that meant that if somethingrelevant didn't happen, then the connection would stay open.
My pool would continue to open new connections but they weren't always being put back.
I wonder what is the optimal way to establish/maintain connection with MySQL/Redis (from nodejs): store in one single object (conn) or create a new connection on every request? Namely:
1, Should we use a single connection for every nodejs http request? Use connection pool? Or a single connection on every new request (so reconnection should be important because the connection could be randomly lost)? How is the performance?
2, What is the difference between MySQL and Redis in term of maintaining such connection?
I will tell you how I used to manage to do this.
1, Should we use a single connection for every nodejs http request? Use connection pool? Or a single connection on every new request (so reconnection should be important because the connection could be randomly lost)? How is the performance?
You don't want to create connections manually for every nodejs http request. Always use connection pooling if you are using nodejs mysqlijs/mysql module. I use it.
Pools take care of server reconnections automatically.
I don't have a benchmarks, but performance should be better because within pools connections can be reused once released. In other factors, believe me, creating and managing connections manually is cumbersome and error-prone.
Eg:
Declare your mysql connection pool in a Db.js file like below and export it.
var mysql = require("mysql");
var pool = mysql.createPool({
connectionLimit : 5,
host : process.env.DB_HOST || 'localhost',
user : process.env.DB_USER,
password : process.env.DB_PASSWORD,
database : 'mydb'
});
module.exports = db;
And use it in your inside an end-point in another file.
var pool = require('./Db.js');
app.get('/endpoint', function (req, res) {
// ...
pool.query('<your-query-here>', function (err, res, fields) {
if (err) throw err;
// do something with result (res)
});
});
I prefer using both pool.query and pool.getConnection based on the scenario. Using query is safer because you don't need to consider releasing connection. It will be automatically handled by the library. getConnection is used only where several queries has to be run inside an end-point, so I can reuse the same connection to do that.
2, What is the difference between MySQL and Redis in term of maintaining such connection?
In short, You don't need pooling for redis. We don't think about pooling redis connections according to this.
I'm trying to learn mysql npm on my Node js server.
I care about my server need to close connection from mysql when it done a task or not ?
This is my code
let insertUserByEmail = function (data, callback){
db.query("INSERT INTO user (username, password) VALUES (?,?)",[data.email, data.password], callback);
// then close it
}
and the next task, i will connect again.
Is this my solution is good?
According to standard practice and security measures you should have to close any open database connection if made by your script. If you are leaving connection open then it will be auto time out by database server.
Specifically for node applications you can close connection after pool by
db.end(function (err) {
// all open connections in the db cluster have ended
});
I was developing a CRUD web app through the use of express js, node js and a local SQL database (MySQL) locally, but now it's time to move to the web and I chose an apache web server that supports MySQL.
I already have the database loaded on the Apache web server.
I have already modified the credentials for the connection to the web server (I know that they are correct because I am connected via MySQL local server to it via a tunnel), through something like that:
app.use(
connection(mysql,{
host: '127.127.127.127', //'web hostname'
user: 'root',
port: '80',
password : 'password',
database:'db_name'
},'pool') //or single
);
But when I'm going to query on it I get an error on the connection.query() command:
... throw err; // Rethrow non-MySQL errors
^
TypeError: Cannot read property 'query' of undefined ...
for example in this part of the code:
exports.view1 = function(req, res){
req.getConnection(function(err,connection){
var query = connection.query('SELECT * FROM table',function(err,rows)
{
if(err){
console.log("Error Selecting : %s ",err );
}else{
res.render('view',{page_title:"Table view",data:rows});
}
});
});
};
1) I'm sure that this error is due to the fact that the connection variable is empty, so it is not possible to perform the query, every help is welcome.
2) I have not found much on the net, but is it possible to continue to use express js even on an Apache web server? Do I have to change something in the formulation of database queries?
[SOLVED] Maybe it can help someone in my same situation. I had to change servers online, going from a Mysql server to a node js, in order to already have the basic functionality of node for the management of my app. However I had to install the mysql-server package, but the connection to my app express js, once moved from local to the server, with the right credential changes, worked properly.
I found a very good module (node-mysql) to connect to Mysql database.
The module is very good, I Only have a question about "WHEN" to open the connection to Mysql.
I always have used php-mysql before starting with node, for each request i opened a connection...then query....then close.
Is the same with node? for each request do I have to open a connection and then close it? or can i use persistent connection?
Thank you
The open-query-close pattern generally relies on connection pooling to perform well. Node-mysql doesn't have any built in connection pooling, so if you use this pattern you'll be paying the cost of establishing a new connection each time you run a query (which may or may not be fine in your case).
Because node is single threaded, you can get away with a single persistent connection (especially since node-mysql will attempt to reconnect if the connection dies), but there are possible problems with that approach if you intend to use transactions (since all users of the node client are sharing the same connection and so same transaction state). Also, a single connection can be a limit in throughput since only one sql command can be executed at a time.
So, for transactional safety and for performance, the best case is really to use some sort of pooling. You could build a simple pool yourself in your app or investigate what other packages are out there to provide that capability. But either open-query-close, or persistent connection approaches may work in your case also.
felixge/node-mysql now has connection pooling (at the time of this writing.)
https://github.com/felixge/node-mysql#pooling-connections
Here's a sample code from the above link:
var mysql = require('mysql');
var pool = mysql.createPool(...);
pool.getConnection(function(err, connection) {
// Use the connection
connection.query( 'SELECT something FROM sometable', function(err, rows) {
// And done with the connection.
connection.end();
// Don't use the connection here, it has been returned to the pool.
});
});
So to answer your question (and same as #Geoff Chappell's answer): best case would be to utilize pooling to manage connections.