I'm using knex version 3.10.10, in my node app, connecting to MySQL DB.
My configuration of knex in the app is using the pool option configuration.
1) Is there a need to EXPLICITLY return a connection to the pool after I fired a query? If yes - how
2) Is there a need to EXPLICITLY perform a check on a pool's connection, before firing the query?
Thanks in advance
No. There is no need to do either.
Knex handles a connection pool for you. You can adjust the pool size if you need to by using the setting: pool: { min: 0, max: 7 } within your connection setup, and the documentation also includes a link to the library that Knex uses for pool handling if you care about the gory details.
The knex documentation has a little info on this here: link
Each connection will be used by Knex for the duration of a query or a transaction, then released back to the pool.
BUT, if you implement transactions (i.e. multiple SQL statements to be saved or cancelled as a unit) without using Promises, then you will need to explicitly commit/rollback the transaction to properly complete the transaction, which will also release the connection back to the pool when the transaction is complete. (see more on Knex Transactions: here).
There is no such info in the documentation but based on the source code you can access knex pool like this
const knex = require('knex')(config);
const pool = knex.client.pool;
console.log(pool);
knex uses tarn pool under the hood, so you can check out it's methods there.
P.S. I don't know where did you get that knex version (3 point something) but the current version of it on this answer moment is 0.14.4
Related
there!
I use PgBouncer with Sqlalchemy for getting connections.
For my understanding, I would like to clarify the following points:
I use the sqlalchemy default pool (QueuePool) and take connections from my pgbouncer. After the transaction is completed, is the connection returned to the "lazy" connection storage on my side (sqlalchemy) or directly to the pgbouncer pool?
If I use QueuePool with pool_size=5 parameters and create an engine for pgbouncer pool, does pgbouncer allocate these 5 connections at once, or are the connections given on demand?
If I remove the connection pooling (using NullPool) and also create an engine for pgbouncer, does this mean that after exiting the transaction context, the connection is closed and a new one will be created already inside pgbouncer at the next request?
Which of these approaches is more correct in the context of using sqlalchemy + pgbouncer?
With connections I work like this:
async with async_session() as connect:
yield connect
await connect.commit()
I think you don't need in-app connection pool if you use pgbouncer, you can either use NullPool+pgbouncer of QueuePool + use_lifo=True flag to use LIFO Queue which should reuse old connections first. I am not entirely sure if using both pgbouncer + in-app connection pool would be beneficial or harmful though.
Here's a relevant documentation link: https://docs.sqlalchemy.org/en/20/core/pooling.html
Also for automatic transaction commit you could use sessionmaker.begin():
async with async_sessinomaker.begin() as session:
...
I would like to know if I am to use createConnection instead of using createPool, does it mean that the createConnection pool will be automatically created and closed when done with the query?
Many thanks in advance.
What is the different createConnection vs createPool of MySQL NodeJS and in fact, using createPool the right way to go. Please refer here for the answer.
https://codeforgeek.com/nodejs-mysql-tutorial/
Connection Pooling (createPool) is a mechanism to maintain a cache of database connection so that the connection can be reused after releasing it.
So connections get stored, and do not need to get closed.
While using a connection (createConnection), you only have one connection and it lasts until you close it.
my node.js file is interacting with a MySQL database through the MySQL module. For now I basically just create a pool
var connection = mysql.createPool(dataConnection);
then I make simple queries using connection.query().
Sometimes I get an error saying Error: Connection lost: The server closed the connection. From what I've understood, whenever I call the query method a brand new connection is created from the pool, then it is closed immediately after it is done. Hence I am quite puzzled: how can the connection be closed since the server should explicitly create one for this query? And, most importantly, is there a way I can actually avoid this issue (which doesn't happen too often, fortunately, but still).
Thanks for your help!
Noël.
Since you haven't shared a lot of details and code related to connection pool creation, I'll try to ans as best as I can.
var pool = mysql.createPool(dataConnection);
// check if you have configured any timeouts etc for the pool
I'm not sure about the exact code, but it should look something like this:
connection = pool.GetConnection();
result = connection.query();
// now instead of closing the connection, it should be returned/released back to the pool
connection.release();
How do I create a MySQL connection pool while working with NodeJS and Express?
If Node is single-threaded, what is the advantage of using a pool to connect with MySQL?
If it is, when should I release a connection?
Sharing the same, persistent, connection with the whole application isn't enough?
Nodejs is single threaded, right. But it is also async, meaning that the single thread fires multiple sql queries without waiting for the result. The result is only processed via callbacks. Therefore it makes sense to use a connection pool with more than one connection. The database is likely multi-threaded, which makes it possible to parallelize the queries, although they were fired consecutively. There is no guarantee however in which order the results are processed if you don't take extra care for that.
Addendum about connection release
If you use a connection pool, than you should aquire/release each connection from the pool for each query. There is no big overhead here, since the pool manages the underlying connections.
Get connection from pool
Query
In the callback release connection back to the pool.
I have written a web server using Delphi and the Indy TIdHttpServer component. I am managing a pool of TAdoConnection connections to a MySql database. When a request comes in I query my pool for available database connections. If one is not available then a new TAdoConnection is created and added to the pool.
Problems occur when a connection becomes "stale" (i.e. it has not been used in quite some time). I think in this instance the query results in the "MySql has gone away" error.
Does anyone have a method for getting around this? Or would I have manage it myself by one of the following:
Writing a thread that will periodically "refresh" all connections.
Keeping track of the last active query, and if too old pass up using the connection and instead free it.
Two suggestions:
store a 'last used' time stamp with every pooled connection, and if a connection is requested check if the connection is too old - in this case, create a new one
add a validateObject() method which issues a no-op SQL query to detect if the connection is still healthy
a background thread which cleans up the pool in regular intervals: removing idle connections allows to reduce the pool size back to a minimum after peak usage
For some suggestions, see this article about the Apache Commons Pool Framework: http://www.javaworld.com/article/2071834/build-ci-sdlc/pool-resources-using-apache-s-commons-pool-framework.html