NodeJS + mysql: using connection pool leads to deadlock tables - mysql

I am using NodeJS and mysql library to access MySQL database.
When I establish single connection and repeatedly use it, it works fine:
global.mysql = mysql_module.createConnection({
host: config.mysql.host,
user: config.mysql.user,
password: config.mysql.password
});
When I use connection pool instead, I get ER_LOCK_WAIT_TIMEOUT errors in transactions.
global.mysql = mysql_module.createPool({
host: config.mysql.host,
user: config.mysql.user,
password: config.mysql.password,
database : config.mysql.database,
connectionLimit : 50
});
Strangely enough, the errors do occur on exactly the same data at exactly the same times.
I.e. I have transaction in which I insert in three tables in a row, each time using last inserted ID from previous INSERT statement. With some data this works fine, with some data, the third INSERT produces ER_LOCK_WAIT_TIMEOUT error. When I use single connection in NodeJS, this works fine, so this must be problem related to connection pool.
Any help would be appreciated.

Not a big fan of answering my own questions, but this problem is solved by explicitly creating connection from a pool and using it for every sql statement during transaction
pool.getConnection(function(err, connection) {
connection.query( 'START TRANSACTION', function(err, rows) {
// do all sql statements with connection and then
connection.query( 'COMMIT', function(err, rows) {
connection.release();
}
});
});

Related

Executing Multiple Statements Using Knex & MySQL (Variables & Migration)

Problem:
I'm trying to add a new column to a table with tens of millions of rows, without blocking it.
I'm aware there are a few options here, and I've given ALTER TABLE a try with NULL values & also with a default value, but it takes a very long time (not workable) and locks the table.
So I'm trying to write a migration script that will duplicate the structure of the original table to a new table, add my new column to that new table, and then slowly migrate the old table data in to the new table.
My issue is with Knex though on the data copy.
I'm testing locally and the below query runs absolutely fine in MySQL version 5.6.34 in Sequel Pro, but I get a You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ' in Knex.
knex.schema.raw(`
SELECT #prevMaxIdMovedRecord;
SELECT #newMaxIdMovedRecord;
SELECT maxIdMoved FROM migration_status
ORDER BY maxIdMoved DESC
INTO #prevMaxIdMovedRecord;
INSERT IGNORE INTO table_copy (field1, newField1)
SELECT t.field1, 1 FROM table t
WHERE t.id BETWEEN #prevMaxIdMovedRecord AND #prevMaxIdMovedRecord + 50000;
SELECT id FROM table_copy
ORDER BY id DESC
LIMIT 1
INTO #newMaxIdMovedRecord;
INSERT INTO migration_status (maxIdMoved)
VALUES (#newMaxIdMovedRecord);
DELIMITER ;
`)
Any experts here that can assist? I can't see that Knex allows you to specify MySQL version in the Knexfile.js, and I was wondering if maybe there was a mismatch between versions on syntax. All my other commands in terms of new tables etc work fine...
Thanks
The mysql driver won't accept multiple statements in a single query by default. You can demonstrate this with a simple Node program:
const mysql = require('mysql')
connection = mysql.createConnection({
host: 'localhost',
user: 'youruser',
password: 'yourpassword',
database: 'yourdb',
multipleStatements: true
})
connection.connect()
connection.query('SELECT 1+1; SELECT 2+2;', function(err, results) {
if (err) {
throw err
}
console.log(results[0], results[1])
})
connection.end()
If you set multipleStatements to false, you should see an error. While multipleStatements could be passed to the driver via your knexfile.js,
{
client: "mysql",
connection: {
// ... other options ...
multipleStatements: true
}
}
and you might try this, I recommend you use a transaction as executing multiple statements in one query like this on a large table seems fragile at best. See docs for an introduction.

Connecting to an existing database - node.js

I want to develop an API in Node.JS with only one endpoint taking 2 parameters : a number and a datetime.
This endpoint will return the result of a request in a MySql database, in json.
But my problem is : I don't know if I need to define the models in my code. Indeed, my database is already created, I am connected to it and I only need to return the result of one SQL request with the 2 parameters.
According to me, I think there would be a solution to just call the database and directly return the result.
Is it possible ?
Thank you in advance !!
Not sure if I understand correctly the issue, but from the looks of it, maybe you are bound to some kind of ORM. In any case, most ORMs or the underlying database drivers allow you to send raw SQL queries to the MySQL server without the need for any kind of models or schemas.
For instance, using the mysql package from npm (sample taken from the official repo):
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'localhost',
user : 'me',
password : 'secret',
database : 'my_db'
});
connection.connect();
connection.query('SELECT 1 + 1 AS solution', function (error, results, fields) {
if (error) throw error;
console.log('The solution is: ', results[0].solution);
});
connection.end();

MySQL pooling within nodejs

Hi i've just read the docs of mysql package for nodejs. Lil bit not sure of how is the best practice to work with pooling.
var mysql = require('mysql');
var pool = mysql.createPool(...);
pool.getConnection(function(err, connection) {
// Use the connection
connection.query('SELECT something FROM sometable', function (error, results, fields) {
// And done with the connection.
connection.release();
// Handle error after the release.
if (error) throw error;
// Don't use the connection here, it has been returned to the pool.
});
});
Do we have to call release() method everytime we have performed query?
And one more.
What is the difference between using pool directly to perform the query vs. using getConnection method then perform the query?
Code using pool direcly:
var pool = mysql.createPool(...);
pool.query(...)
Using getConnection method then perform the query:
pool.getConnection(function(err, connection) {
connection.query(....);
});
If you ask for getting a connection, you basically reserve that connection for a little while. This is important for 2 reasons:
Only 1 query can be done on a connection at a time, never in parallel. So this prevents 2 things from using the same connection.
Transactions are connection-based and all queries within the transaction must happen on that connection object.
The mysql library would have no way to predict that you are 'done' your transaction, this is why you need to release it.
Aside: You should consider looking into mysql2 for a similar library that's more powerful, and use promises instead of this callback pattern.
Update based on comment
When you do query directly on the pool, the pool will automatically get the connection, run the query and release it for you.
This is useful if you just need to do a single query and don't care about transactions.

How to kill a MySQL query with Node.js without disconnecting?

Context:
I'm building a web application that calls data from a large db (several millions of rows for table); sometimes a user can change his mind and call for new data before the query on the db has been completed.
Technical question:
I tried to kill the query in this cases using:
app.get("/data", function(req, res) {
if (req.query.killQuery == "true") {
con.query("KILL \"" + threadId + "\"", function(err) {
if (err) throw err;
console.log("I have interrupted the executing query for a new request");
giveData(req, res); //The function that will execute a new query
});
return;
}
giveData(req, res); //The function that will execute a new query
});
Now I have several doubts about this code:
I had to use a second connection to kill the thread of the first, since the first was unable to perform new queries before the first was completed. Is this a Node.js behaviour or is it the right way to do this kind of things?
The KILL thread_id statement closes the whole connection instead of stopping the single query. Again, is it Node.js behaviour, or is it MySQL itself? Should I really disconnect and reconnect to stop a query and start with an other?
If you have a modern version of MySQL, you can use KILL QUERY <threadId> instead which will only kill the currently executing query on that connection but leave the connection intact.

Socket.io and MySQL Connections

I'm working on my first node.js-socket.io project. Until now i coded only in PHP. In PHP it is common to close the mysql connection, when it is not needed any more.
My Question: Does it make sense to keep just one mysql-connection during server is running open, or should i handle this like PHP.
Info: In the happy hours i will have about 5 requests/seconds from socket clients and for almost all of them i have to make a mysql_crud.
Which one would you prefer?
io = require('socket.io').listen(3000); var mysql = require('mysql');
var connection = mysql.createConnection({
host:'localhost',user:'root',password :'pass',database :'myDB'
});
connection.connect(); // and never 'end' or 'destroy'
// ...
or
var app = {};
app.set_geolocation = function(driver_id, driver_location) {
connection.connect();
connection.query('UPDATE drivers set ....', function (err) {
/* do something */
})
connection.end();
}
...
The whole idea of Node.js is async io (that includes db queries).
And the rule with a mysql connection is that you can only have one query per connection at a time. So you either make a queue and have a single connection, as in the first option or create a connection each time as with option 2.
I personally would go with option 2, as opening and closing connections are not such a big overhead.
Here are some code samples to help you out:
https://codeforgeek.com/2015/01/nodejs-mysql-tutorial/