NodeJS + mysql - automatically closing pool connections? - mysql

I wish to use connection pooling using NodeJS with MySQL database. According to docs, there are two ways to do that: either I explicitly get connection from the pool, use it and release it:
var pool = require('mysql').createPool(opts);
pool.getConnection(function(err, conn) {
conn.query('select 1+1', function(err, res) {
conn.release();
});
});
Or I can use it like this:
var mysql = require('mysql');
var pool = mysql.createPool({opts});
pool.query('select 1+1', function(err, rows, fields) {
if (err) throw err;
console.log('The solution is: ', rows[0].solution);
});
If I use the second options, does that mean, that connections are automatically pulled from the pool, used and released? And if so, is there reason to use the first approach?

Yes, the second one means that the pool is responsible to get the next free connection do a query on that and then release it again. You use this for "one shot" queries that have no dependencies.
You use the first one if you want to do multiple queries that depend on each other. A connection holds certain states, like locks, transaction, encoding, timezone, variables, ... .
Here an example that changes the used timezone:
pool.getConnection(function(err, conn) {
function setTimezone() {
// set the timezone for the this connection
conn.query("SET time_zone='+02:00'", queryData);
}
function queryData() {
conn.query( /* some query */, queryData);
}
function restoreTimezoneToUTC() {
// restore the timezone to UTC (or what ever you use as default)
// otherwise this one connection would use +02 for future request
// if it is reused in a future `getConnection`
conn.query("SET time_zone='+00:00'", releseQuery);
}
function releaseQuery() {
// return the query back to the pool
conn.release()
}
setTimezone();
});

In case anyone else stumbles upon this:
When you use pool.query you are in fact calling a shortcut which does what the first example does.
From the readme:
This is a shortcut for the pool.getConnection() -> connection.query() -> connection.release() code flow. Using pool.getConnection() is useful to share connection state for subsequent queries. This is because two calls to pool.query() may use two different connections and run in parallel.
So yes, the second one is also calling connection.release() you just don't need to type it.

Related

Handling of rabbit messages via NESTJs microservice issue

I'm currently having a problem that i'm unable to solve. This is only in extreme cases, when the server, for some reason, goes offline, messages accumulate (100k or more) and then they need to be processed all at the same time. Even though i'm planning for this never to happen, i would like to have a backup plan for it and more control on this issue.
I'm running an NestJS microservice against a RabbitMQ broker to get messages that arrive from IOT devices and insert them into a MySQL database.
Every message has a little conversion/translation operation that needs to be done before the insert. This conversion is based on a single row query done against a table on the same SQL Server.
The order is the following:
read message;
select 1 row from database (table has few thousand rows);
insert 1 row into the database;
Now, i'm facing this error:
(node:1129233) UnhandledPromiseRejectionWarning: SequelizeConnectionAcquireTimeoutError: Operation timeout
at ConnectionManager.getConnection (/home/nunovivas/NestJSProjects/integrador/node_modules/sequelize/lib/dialects/abstract/connection-manager.js:288:48)
at runNextTicks (internal/process/task_queues.js:60:5)
at listOnTimeout (internal/timers.js:526:9)
at processTimers (internal/timers.js:500:7)
at /home/nunovivas/NestJSProjects/integrador/node_modules/sequelize/lib/sequelize.js:613:26
at MySQLQueryInterface.select (/home/nunovivas/NestJSProjects/integrador/node_modules/sequelize/lib/dialects/abstract/query-interface.js:953:12)
at Function.findAll (/home/nunovivas/NestJSProjects/integrador/node_modules/sequelize/lib/model.js:1752:21)
at Function.findOne (/home/nunovivas/NestJSProjects/integrador/node_modules/sequelize/lib/model.js:1916:12)
node_modules/source-map-support/source-map-support.js:516
(node:1129233) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1349)
I think that the promise rejection is inside the sequelize module.
This is my sequelize configuration:
useFactory: async (ConfigService: ConfigService) => ({
dialect: 'mysql',
host: 'someserver',
port: 3306,
username: 'dede',
password: 'dudu!',
database: 'dada',
autoLoadModels: true,
pool: { max: 5, min: 0, adquire: 1800000, idle: 5000 },
synchronize: true,
logQueryParameters: true,
This is part of my message service:
#RabbitRPC({
exchange: 'BACKEND_MAINEXCHANGE',
routingKey: 'Facility_DeviceReadings',
queue: 'Facility_DeviceReadings',
})
public async rpcHandlerDeviceReadings(mensagem: ReadingDevicesPerFacility) {
const schemavalid = mensagem;
this.mylogger.log( 'Received message from BACKEND_MAINEXCHANGE - listening to the queue Facility_DeviceReadings : ' +
' was registered locally on ' +
schemavalid.DateTimeRegistered,
MessagingService.name,
'rpcHandlerDeviceReadings',
);
if (schemavalid) {
try {
let finalschema = new CreateReadingDevicesDto();
if (element.Slot > 0) {
const result = this.readingTransService
.findOneByPlcId(element.deviceId, element.Slot)
.then((message) => {
if (!message) {
throw new NotFoundException('Message with ID not found');
} else {
finalschema.deviceId = message.deviceId;
finalschema.Slot = message.Slot2;
if (this.isNumeric(element.ReadingValue)) {
finalschema.ReadingValue = element.ReadingValue;
finalschema.DateTimeRegistered =
schemavalid.DateTimeRegistered;
this.readingDeviceService
.create(finalschema)
.then((message) => {
this.mylogger.debug(
'Saved',
MessagingService.name,
'rpcHandlerDeviceReadings',
);
return 42;
});
} else {
this.mylogger.error(
'error',
MessagingService.name,
'rpcHandlerDeviceReadings',
);
}
return message;
}
});
The problem seems that this RPC keeps going against rabbit and reading/consuming messages (8 per millisecond) before SQL as a chance to replay back, forcing sequelize into a state that it can't handle anymore and thus throwing the above error.
I have tried tweaking the sequelize config but to no good outcome.
Is there any way to force the RPC to just handle the next message after the previous one is processed?
Would love if someone could steer me in the right direction since this could eventually become a breaking issue.
Thanks in advance for any input you can give me.
It looks to me like your Sequelize connection pool options need some tweaking.
You have
pool: { max: 5, min: 0, adquire: 1800000, idle: 5000 }
adquire isn't a thing. Maybe acquire? Half an hour (1.8 million milliseconds) is a really long time to wait for a connection. Shorten it? acquire: 300000 will give you five minutes. A big production app such as yours probably should always keep one or two connections open. Increase min to 1 or 2.
A modest maximum number of connections is good as long as each operation grabs a connection from the pool, uses it, and releases it. If your operation grabs a connection and then awaits something external, you'll need more connections.
If it's possible to get your program to read in a whole bunch of messages (at least 10) at a time, then put them into your database in one go with bulkCreate(), you'll speed things up. A lot. That's because inserts are cheap, but the commit operations after those inserts aren't so cheap. So, doing multiple inserts within single transactions, then commiting them all at once, can make things dramatically faster. Read about autocommit for more information on this.
Writing your service to chow down on a big message backlog quickly will make errors like the one you showed us less likely.
Edit To use .bulkCreate() you need to accumulate multiple incoming messages. Try something like this.
Create an array of your received CreateReadingDevicesDto messages. let incomingMessages = []
Instead of using .create() to put each new message into your database as you finish receiving and validating it, instead put it into your array. incomingMessages.push(finalschema).
Set up a Javascript interval to take the data from the array and put it into your database with .bulkCreate(). This will do that every 500ms.
setInterval(
function (this) {
if (incomingMessages.length > 0) {
/* create all the items in the array */
this.readingDeviceService
.bulkCreate(incomingMessages)
/* empty out the array */
incomingMessages = []
}, 500, this);
At the cost of somewhere between 0 and 500ms extra latency, this batches up your messages and will let you process your backlog faster.
I haven't debugged this, and it's probably a little more crude than you want in production code. But I have used similar techniques to good effect.

How to optimize a sequential insert on mysql

I'm trying to implement an HTTP event streaming server using MySQL where Users are able to append an event to a stream (a MySQL table) and also define the expected sequence number of the event.
The logic is somewhat simple:
Open transaction
get the next sequence number in the table to insert
verify if the next sequence number matches the expected(if supplied)
insert in database
Here's my code:
public async append(
data: any = {},
expectedSeq?: number
): Promise<void> {
let published_at = $date.create();
try {
await $mysql.transaction(async trx => {
let max = await trx(this.table)
.max({
seq: "seq",
})
.first();
if (!max) {
throw $error.InternalError(`unexpected mysql response`);
}
let next = (max.seq || 0) + 1;
if (expectedSeq && expectedSeq !== next) {
throw $error.ExpectationFailed(
`expected seq does not match current seq`
);
}
await trx(this.table).insert({
published_at,
seq: next,
data: $json.stringify(data),
});
});
} catch (err) {
if (err.code === "ER_DUP_ENTRY") {
return this.append(data, expectedSeq);
}
throw err;
}
}
My problem is this is extremely slow since there are race conditions between parallel requests to append to the same stream.. my laptop inserts/second on one stream went from ~1k to ~75.
Any pointers/suggestions on how to optimize this logic?
CONCLUSION
After consideration from comments, I decided to go with auto increment and reset the auto_increment only if there's an error. It yields around the same writes/sec with expectedSeq but much higher rate if ordering is not required.
Here's the solution:
public async append(data: any = {}, expectedSeq?: number): Promise<Event> {
if (!$validator.validate(data, this.schema)) {
throw $error.ValidationFailed("validation failed for event data");
}
let published_at = $date.create();
try {
let seq = await $mysql.transaction(async _trx => {
let result = (await _trx(this.table).insert({
published_at,
data: $json.stringify(data),
})).shift();
if (!result) {
throw $error.InternalError(`unexpected mysql response`);
}
if (expectedSeq && expectedSeq !== result) {
throw $error.ExpectationFailed(
`expected seq ${expectedSeq} but got ${result}`
);
}
return result;
});
return eventFactory(this.topic, seq, published_at, data);
} catch (err) {
await $mysql.raw(`ALTER TABLE ${this.table} auto_increment = ${this.seqStart}`);
throw err;
}
}
Why does the web page need to provide the sequence number? That is just a recipe for messy code, perhaps even messier than what you sketched out. Simply let the auto_increment value be returned to the User.
INSERT ...;
SELECT LAST_INSERT_ID(); -- session-specific, so no need for transaction.
Return that value to user.
Why not use Apache Kafka, It does all of this natively. With the easy answer out of the way, optimization is always tricky with partial information, however I think you've given us one hint that might enable a suggestion. You said without the order clause it performs much faster, which means that getting the max value is what is taking so long. That tells me a few things, first this value is not the clustered index (which is good news), second that you probably do not have sufficient index support (also good news since it's fixable by creating an index on this column, and sorting the index desc). This sounds like a table with millions or billions of rows in it, and this particular column has no guaranteed order, without the right indexing you could be doing a table scan between inserts to get the max value.
Why not use a GUID for your primary key instead of an auto-incremented integer? Then your client could generate the key and would also be able to insert it every time for sure.
Batch inserts versus singleton inserts
Your latency/performance problem is due to a batch size of 1 - as each send to the the database requires multiple round trips to the rdbms. Rather than inserting one row at a time, with a commit and verification after each row, you should rewrite your code to issue batch sizes of 100 or 1000 at a time, inserting n rows and verifying per batch rather than per row. If the batch insert fails, you can retry one row at a time.

Node js mysql stored procedure call in a for loop

So, I'm currently using mysql npm package https://www.npmjs.com/package/mysql. I have a requirement where I'll have to call a stored procedure multiple times with the caveat that subsequent stored procedure calls depend on the previous call. Pseudo code will be as follows:
let mysql = require("mysql");
let mysqlPoolConnection = mysql.createPool({
connectionLimit: 20,
host: '0.0.0.0',
port: '3306',
user: 'user1',
password: 'pwd',
database: 'mysql_db'
});
for (let index = 0; index < params.length; index++) {
let sp_ProcedureCall = "CALL sp_StoredProcedure(?, ?, ?, ?)";
let sp_ProcedureParams = [params[index].firstParam, params[index].secondParam, params[index].thirdParam, params[index].fourthParam];
// This is where the issue is. I'd like to call the stored procedure once and then once I get a response from it then make subsequent calls. Basically, depending on the previous stored procedure result, I decide whether I continue with subsequent calls or not.
mysqlPoolConnection.query(sp_ProcedureCall, sp_ProcedureParams, (errorObj, responseObj, fieldsObj) => {
}
}
NodeJS is asynchronous, meaning your for-loop iterates without waiting for the result of the call. You need to control the flow of your program if you want to "wait" for previous results.
Look at a library like async to enabled that type of control flow. From your description, eachSeries() might be a good fit - to run a function for each value in an array, and only run one at a time. (Or reduce, depending what you need - there are other options too)

nodejs blocking further mysql queries

I have a MySQL query that returns ~11,000 results that then need further queries run on each result. While the code is running it doesn't allow other users to log in to the website.
Potential problems I've seen are the use of callback functions or foreach loops but that doesn't seen to help.
code looks like this:
query(sql, params, function(err, result) {
result.foreach(
query(generate_sql(result), params, function(err, result) {
//do stuff
});
)
});
I recommend you using Promises
Then your code would be something like that:
var Promise = require('bluebird');
_query = Promise.promisify(query);
_query(sql, params)
.map(function(singleRow) {
// you get here every single row of your query
})
.then(function() {
// you are finished
});
Your problem is having that much queries. 11,001 query is a very large number, and I don't think that MySQL can handle that number of queries in parallel.
So another way is to use subqueries to handle your problem.

mysql - node - Row inserted and queryable via server connection lost on DB restart

I ran into an issue testing today that occurred during or after an insert via a connection on my node server. The code where the insert is performed looks something like this:
// ...
username = esc(username);
firstname = esc(firstname);
lastname = esc(lastname);
var values = [username, firstname, lastname].join(',');
var statement = 'INSERT INTO User(Username,FirstName,LastName) VALUES({0});\n'+
'SELECT FirstName, LastName, Username, Id, IsActive FROM User WHERE Username={1};'
statement = merge( statement, [ values, username ] );
conn.query(statement, function(e, rows, fields){
e ? function() {
res.status(400);
var err = new Error;
err.name = 'Bad request';
err.message = 'A problem occurred during sign-up.';
err.details = e;
res.json(err);
}() : function(){
res.json( rows[1] );
}();
}
A quick note on esc() and merge(), these are simply util functions that help prepare the database statement.
The above code completed successfully, ie. the response was a 200 with the newly inserted user row in the body. The row inserted was queryable via the same connection throughout the day. I only noticed this afternoon when running the following generic query as root via shell, the row was missing.
SELECT Id, FirstName, LastName FROM User;
So at that point I restarted the database and the node server. Unfortunately. Now it would appear the row is gone entirely, as well as any reliable path to troubleshoot.
Here are some details of interest to my server setup. As of yet, no idea how (if at all) any of these could be suspect.
Uses only single connection as opposed to conn pool (for now)
multipleStatements=true in the connection config (obviously above snippet makes use of this)
SET autocommit = 0; START TRANSACTION; COMMIT; used elsewhere in the codebase to control rollback
Using poor man's keep-alive every 30 seconds to avoid connection timing out: SELECT 1;
I've been reading up all evening and am running out of ideas. Any suggestions? Is this likely an issue of uncommitted data? If so, is there a reliable way to debug this? Better yet, is there any way to prevent it? What could be the cause? And finally, if in debugging my server I find data in this state, is there a way to force commit at least so that I don't lose my changes?