I found a very good module (node-mysql) to connect to Mysql database.
The module is very good, I Only have a question about "WHEN" to open the connection to Mysql.
I always have used php-mysql before starting with node, for each request i opened a connection...then query....then close.
Is the same with node? for each request do I have to open a connection and then close it? or can i use persistent connection?
Thank you
The open-query-close pattern generally relies on connection pooling to perform well. Node-mysql doesn't have any built in connection pooling, so if you use this pattern you'll be paying the cost of establishing a new connection each time you run a query (which may or may not be fine in your case).
Because node is single threaded, you can get away with a single persistent connection (especially since node-mysql will attempt to reconnect if the connection dies), but there are possible problems with that approach if you intend to use transactions (since all users of the node client are sharing the same connection and so same transaction state). Also, a single connection can be a limit in throughput since only one sql command can be executed at a time.
So, for transactional safety and for performance, the best case is really to use some sort of pooling. You could build a simple pool yourself in your app or investigate what other packages are out there to provide that capability. But either open-query-close, or persistent connection approaches may work in your case also.
felixge/node-mysql now has connection pooling (at the time of this writing.)
https://github.com/felixge/node-mysql#pooling-connections
Here's a sample code from the above link:
var mysql = require('mysql');
var pool = mysql.createPool(...);
pool.getConnection(function(err, connection) {
// Use the connection
connection.query( 'SELECT something FROM sometable', function(err, rows) {
// And done with the connection.
connection.end();
// Don't use the connection here, it has been returned to the pool.
});
});
So to answer your question (and same as #Geoff Chappell's answer): best case would be to utilize pooling to manage connections.
Related
I wonder what is the optimal way to establish/maintain connection with MySQL/Redis (from nodejs): store in one single object (conn) or create a new connection on every request? Namely:
1, Should we use a single connection for every nodejs http request? Use connection pool? Or a single connection on every new request (so reconnection should be important because the connection could be randomly lost)? How is the performance?
2, What is the difference between MySQL and Redis in term of maintaining such connection?
I will tell you how I used to manage to do this.
1, Should we use a single connection for every nodejs http request? Use connection pool? Or a single connection on every new request (so reconnection should be important because the connection could be randomly lost)? How is the performance?
You don't want to create connections manually for every nodejs http request. Always use connection pooling if you are using nodejs mysqlijs/mysql module. I use it.
Pools take care of server reconnections automatically.
I don't have a benchmarks, but performance should be better because within pools connections can be reused once released. In other factors, believe me, creating and managing connections manually is cumbersome and error-prone.
Eg:
Declare your mysql connection pool in a Db.js file like below and export it.
var mysql = require("mysql");
var pool = mysql.createPool({
connectionLimit : 5,
host : process.env.DB_HOST || 'localhost',
user : process.env.DB_USER,
password : process.env.DB_PASSWORD,
database : 'mydb'
});
module.exports = db;
And use it in your inside an end-point in another file.
var pool = require('./Db.js');
app.get('/endpoint', function (req, res) {
// ...
pool.query('<your-query-here>', function (err, res, fields) {
if (err) throw err;
// do something with result (res)
});
});
I prefer using both pool.query and pool.getConnection based on the scenario. Using query is safer because you don't need to consider releasing connection. It will be automatically handled by the library. getConnection is used only where several queries has to be run inside an end-point, so I can reuse the same connection to do that.
2, What is the difference between MySQL and Redis in term of maintaining such connection?
In short, You don't need pooling for redis. We don't think about pooling redis connections according to this.
Is it possible to cache database connections when using PHP like you would in a J2EE container? If so, how?
There is no connection pooling in php.
mysql_pconnect and connection pooling are two different things.
There are many problems connected with mysql_pconnect and first you should read the manual and carefully use it, but this is not connection pooling.
Connection pooling is a technique where the application server manages the connections. When the application needs a connection it asks the application server for it and the application server returns one of the pooled connections if there is one free.
We can do connection scaling in php for that please go through following link: http://www.oracle.com/technetwork/articles/dsl/white-php-part1-355135.html
So no connection pooling in php.
As Julio said apache releases all resources when the request ends for the current reques. You can use mysql_pconnect but you are limited with that function and you must be very careful. Other choice is to use singleton pattern, but none of this is pooling.
This is a good article: https://blogs.oracle.com/opal/highly-scalable-connection-pooling-in-php
Also read this one http://www.apache2.es/2.2.2/mod/mod_dbd.html
Persistent connections are nothing like connection pooling. A persistent connection in php will only be reused if you make multiple db connects within the same request/script execution context. In most typical web dev scenarios you'll max out your connections way faster if you use mysql_pconnect because your script will have no way to get a reference to any open connections on your next request. The best way to use db connections in php is to make a singleton instance of a db object so that the connection is reused within the context of your script execution. This still incurs at least 1 db connect per request, but it's better than making multiple db connects per reqeust.
There is no real db connection pooling in php due to the nature of php. Php is not an application server that can sit there in between requests and manage references to a pool of open connections, at least not without some kind of major hack. I think in theory you could write an app server in php and run it as a commandline script that would just sit there in the background and keep a bunch of db connections open and pass references to them to your other scripts, but I don't know if that would be possible in practice, how you'd pass the references from your commandline script to other scripts, and I sort of doubt it would perform well even if you could pull it off. Anyway that's mostly speculation. I did just notice the link someone else posted to an apache module to allow connection pooling for prefork servers such as php. Looks interesting:
https://github.com/junamai2000/mod_namy_pool#readme
I suppose you're using mod_php, right?
When a PHP file finishes executing all it's state is killed so there's no way (in PHP code) to do connection pooling. Instead you have to rely on extensions.
You can mysql_pconnect so that your connections won't get closed after the page finishes, that way they get reused in the next request.
This might be all that you need but this isn't the same as connection pooling as there's no way to specify the number of connections to maintain opened.
You can use MySQLi.
For more info, scroll down to Connection pooling section # http://www.php.net/manual/en/mysqli.quickstart.connections.php#example-1622
Note that Connection pooling is also dependent on your server (i.e. Apache httpd) and its configuration.
If an unused persistent connection for a given combination of "host, username, password, socket, port and default database can not be found" in the open connection pool, then only mysqli opens a new connection otherwise it would reuse already open available persistent connections, which is in a way similar to the concept of connection pooling. The use of persistent connections can be enabled and disabled using the PHP directive mysqli.allow_persistent. The total number of connections opened by a script can be limited with mysqli.max_links (this may be interesting to you to address max_user_connections issue hitting hosting server's limit). The maximum number of persistent connections per PHP process can be restricted with mysqli.max_persistent.
In wider programming context, it's a task of web/app server however in this context, it's being handled by mysqli directive of PHP itself in a way supporting connection re-usability. You may also implement a singleton class to get a static instance of connection to reuse just like in Java. Just want to remind that java also doesn't support connection pooling as part of its standard JDBC, they're being different module/layers on top of JDBC drivers.
Coming to PHP, the good thing is that for the common databases in the PHP echosystem it does support Persistent Database Connections which persists the connection for 500 requests (config of max_requests in php.ini) and this avoids creating a new connection in each request. So check it out in docs in detail, it solves most of your challenges. Please note that PHP is not so much sophisticated in terms of extensive multi-threading mechanism and concurrent processing together with powerful asynchronous event handling, when compared to strictly object oriented Java. So in a way it is very less effective for PHP to have such in-built mechanism like pooling.
You cannot instantiate connection pools manually.
But you can use the "built in" connection pooling with the mysql_pconnect function.
I would like to suggest PDO::ATTR_PERSISTENT
Persistent connections are links that do not close when the execution of your script ends. When a persistent connection is requested, PHP checks if there's already an identical persistent connection (that remained open from earlier) - and if it exists, it uses it. If it does not exist, it creates the link.
Connection pooling works at MySQL server side like this.
If persistence connection is enabled into MySQL server config then MySQL keep a connection open and in sleep state after requested client (php script) finises its work and die.
When a 2nd request comes with same credential data (Same User Name, Same Password, Same Connection Parameter, Same Database name, Maybe from same IP, I am not sure about the IP) Then MySQL pool the previous connection from sleep state to active state and let the client use the connection. This helps MySQL to save time for initial resource for connection and reduce the total number of connection.
So the connection pooling option is actually available at MySQL server side. At PHP code end there is no option. mysql_pconnect() is just a wrapper that inform PHP to not send connection close request signal at the end of script run.
For features such as connection pooling - you need to install swoole extension first: https://openswoole.com/
It adds async features to php.
After that its trivial to add mysql and redis connection pooling:
https://github.com/open-smf/connection-pool
Some PHP frameworks come with pooling built-in: https://hyperf.wiki/2.2/#/en/pool
With the node-mysql module, there are two connection options - a single connection and a connection pool. What is the best way to set up a connection to a MySQL database, using a single global connection for all requests, or creating a pool of connections and taking one from the pool for each request? Or is there a better way to do this? Will I run in to problems using just a single shared connection for all requests?
Maintaining a single connection for the whole app might be a little bit tricky.
Normally, You want to open a connection to your mysql instance, and wait for it to be established.
From this point you can start using the database (maybe start a HTTP(S) server, process the requests and query the database as needed.)
The problem is when the connection gets destroyed (ex. due to a network error).
Since you're using one connection for the whole application, you must reconnect to MySQL and somehow queue all queries while the connection is being established. It's relatively hard to implement such functionality properly.
node-mysql has a built-in pooler. A pooler, creates a few connections and keeps them in a pool. Whenever you want to close a connection obtained from the pool, the pooler returns it to the pool instead of actually closing it. Connections on the pool can be reused on next open calls.
IMO using a connection pool, is obviously simpler and shouldn't affect the performance much.
If we don't close or end mysql connection in node js, is it effects in feature or not.
I am doing like this
var mysql = require('mysql');
var connection = mysql.createConnection(...);
connection.query('SELECT 1', function(err, rows) {
// connected! (unless `err` is set)
});
Am not ending mysql connection any where in my code. My question is
Is it necessary to close the mysql connection.
If we don't close the mysql connection will i face any other problems in future.
Please help me am new to nodejs
EDIT1
I will not face any problems like unable to connect, too many connections to open etc, means any resource related issues? right.
EDIT2
At which instant mysql connection will be close if we don't end it manually or by using end function?
You should close the connection.
The point at which you do it depends on what your program does with that connection.
If the program is long lived and only needs a single connection but uses that connection continuously then you can just leave that one connection open, but you should be be prepared to re-open the connection if required - e.g. if the server gets restarted.
If the program just opens one connection, does some stuff, and then doesn't use the connection for some time you should close the connection while you're not using it.
If the program is short lived, i.e. it makes a connection, does some stuff, and then exits you can get away without closing the connection because it'll get closed automagically when your program exits.
Actually, you will run into problems such as this one from MySQL:
"[host] is blocked because of many connection errors; unblock with ' mysqladmin flush-hosts"
As MySQL will start counting those failed disconnects as "connection errors". And then, your database remains unusable until you execute FLUSH HOSTS on the MySQL server. I know this for a fact, it happened with our NodeJS project.
Write some code to explicitly close the db connection before any output or render statements using connection.end() or connection.destroy()
No you don't need to close it.
If you question is "is there any specific cleanup on mysql side when I tell server that I'm closing connection" then the answer is "no, if you just exit your client process or close socket its the same as calling connection.end()"
I have developed a application which send large number of emails to different users in batches. The most common problem I faced in this application is the problem of mysql connection timeout. In between the batches when there is no queries executed in the previously opened connection and connection remained idle for long time, mysql itself close the connection. After sending the current batch when I again try to execute any sql query it gives me mysql connection error.
Right now I am using mysql_ping($conn) function to check whether the connection id timedout or not. If the connect is timed out I connect again with mysql_connect() function. Now I am moving to doctrine rather than native PHP function. Is there a recconnect() function in Doctorine as well ?
Never faced this problem with my batch actions but I think you could probably do something like this in those places in your code where you think there's a risk the connection will time out:
// Fetch current connection
$conn = Doctrine_Manager::connection();
if(!$conn) {
// Open a new connection
$conn = Doctrine_Manager::connection('mysql://username:password#localhost/whatever', 'connection 1');
}
http://docs.doctrine-project.org/projects/doctrine1/en/latest/en/manual/connections.html
There's a custom bundle that allow connection to reopen:
https://github.com/facile-it/doctrine-mysql-come-back
You can set a maximum amount of retry, or you can manually call method retry on Connection!