Use CLS with Sequelize Unmanaged transactions - mysql

I am writing unit tests for my code and wish to use transactions to prevent any stray data between tests.
The code uses Sequelize ORM for all interactions with the database. Since changing the actual code is not an option, I would be using cls-hooked to maintain transaction context instead of passing transaction to all the queries. There is a problem, however. On reading the official documentation and trying to go about it, the above approach seems to only work for managed transactions.
So far, the test code looks somewhat like:
test("Test decription", async () => {
try {
await sequelize.transaction(async (t) => {
//Actual test code
});
} catch (error) {
//Do nothing if query rolled back
}
});
What I intend to achieve (for obvious reasons):
let t;
beforeEach(async () => {
t = await sequelize.transaction();
});
test("Test decription", async () => {
//Actual test code
});
afterEach(async () => {
await t.rollback();
});
Is this possible? If yes, any help in implementing this would be appreciated.

I'm having the same problem -- after much Googling, I found this closed issue where they indicate that unmanaged transactions aren't supported by design 🥲

It's true that Sequelize doesn't automatically pass transactions to queries when you're using unmanaged transactions. But you can manually set the transaction property on the CLS namespace, just like Sequelize does on a managed transaction:
https://github.com/sequelize/sequelize/blob/v6.9.0/lib/transaction.js#L135
namespace.run(() => {
namespace.set('transaction', transaction);
/* run code that does DB operations */
});
This is tricky for tests because describe() calls can be nested and each can have their own beforeAll()/afterAll() and beforeEach()/afterEach() hooks. To do this right, each before hook needs to set up a nested transaction and the corresponding after hook should roll it back. In addition, the test case itself needs to run in a nested transaction so that its DB operations don't leak into other tests.

For anyone from the future:
I was facing the above problem and found a way to fix it with a helper function and cls-hooked.
const transaction = async (namespace: string, fn: (transaction: Transaction) => unknown) => {
const nameSpace = createNamespace(namespace);
db.Sequelize.useCLS(nameSpace);
const sequelize = db.sequelize;
const promise = sequelize.transaction(async (transaction: Transaction) => {
try {
await fn(transaction);
} catch (error) {
console.error(error);
throw error;
}
throw new TransactionError();
});
await expect(promise).rejects.toThrow(TransactionError);
destroyNamespace(namespace);
};
What the above code does is creating a cls namespace and transaction that will be discarded after the test run, the TransactionError is thrown to ensure the entire transaction is always rolled back on each test run.
Usage on tests would be:
describe('Transaction', () => {
test('Test', async () => {
await transaction('test', async () => {
// test logic here
});
});
});

Related

Unnesting Node database calls

I have an ordinary
var express = require('express')
Node express www page, using session, pug, etc as usual. My db calls
var db = require('./scripts/myHappyMysqlScript')
I'm naturally using mysql, so in the db script
var mysql = require('mysql')
So for example
app.get('/catPhotos', (req, response) => {
response.render('catPhotos.pug');
})
Say a page has a table showing something from the petNames database,
app.get('/pets', function(req, res, next) {
db.allPetNames(function(err, petsList) {
res.render('pets.pug',
{
'petsList': petsList,
'pretty' : true
})
})
all good so far.
But here's a case with three tables on the pug page, and three different database calls:
db.cats(function(err, c) {
db.dogs(function(err, d) {
db.budgies(function(err, b) {
res.render('bigScreen.pug',
{
'cats' : c,
'k9s': d,
'budgies': b,
'pretty' : true
})
})
})
})
I just nest them like that.
This does seem to work perfectly.
It correctly waits sequentially. Errors fall through and are handled properly, and so on.
But is there a better syntax, better way?
What's the Node Way for realâ„¢ Node, not-Swift, programmers?!
Perhaps given that I'm using the mysql library, if that's relevant.
Note, one better way overall is to use something like Ajax to just stream in each "part" of the web page. Indeed I do that all the time. What I'm asking here, assuming at res.render I indeed want to return all that info at once, is there something better than nesting like that? Cheers
You can get rid of nested database calls by using promises.
Since you mentioned that you are using mysql library for interacting with the database, unfortunately, this library doesn't provide a promise-based API. So to get rid of nested database calls in your code, you need to create a promise-based wrapper around the callback version of database calls.
For a general overview of what promises are and how they work, see the following links:
MDN - Promise.
MDN - Using Promises
Following is an example of how you can create a promise-based wrapper and then use that wrapper to get rid of nested database calls.
This promise-based wrapper is just a function that returns a promise. It creates a promise instance, wraps the underlying database call, and eventually when the database call returns the data, it notifies your code.
function getCats() {
return new Promise((resolve, reject) => {
// make the database call
db.cats((error, cats) => {
// in case of an error, reject the promise by
// calling "reject" function
// Also pass the "error" object to the "reject" function
// as an argument to get access to the error message
// in the code that calls this "getCats" function
if (error) {
reject(error);
return;
}
// if there was no error, call the "resolve" function
// to resolve the promise. Promise will be resolved
// in case of a successful database call
// Also pass the data to the "resolve" function
// to access this data in the code that calls this
// "getCats" function
resolve(cats);
});
});
}
Now in your route handler function, instead of calling db.cats(...), call this getCats wrapper function.
There are two ways you can call the function that returns a promise:
Promise-chaining (For details, visit the links mentioned above)
async-await syntax (Recommended)
The following code example uses async-await syntax. For this, first, mark the route handler function as async by using the async keyword before the function keyword. By doing this, we can use the await keyword inside this route handler function.
app.get('/pets', async function(req, res, next) {
try {
const cats = await getCats();
// similar wrappers for other database calls
const dogs = await getDogs();
const budgies = await getBudgies();
// render the pub template, passing in the data
// fetched from the database
...
catch (error) {
// catch block will be invoked if the promise returned by
// the promise-based wrapper function is rejected
// handle the error appropriately
}
});
The above code example only shows how to wrap the db.cats(...) database call in a promise-based wrapper and use that wrapper to get the data from the database. Similarly, you can create wrappers for db.dogs(...) and db.budgies(...) calls.
Instead of creating a separate promise-based wrapper for each database call, ideally, you should create a re-usable promise-based wrapper function that takes in a function to call and wraps that function call in a promise just like shown in the above code example, i.e. getCats function.
Parallel Database calls
One important thing to note in the above code is the route handler function
const cats = await getCats();
const dogs = await getDogs();
const budgies = await getBudgies();
is that this will lead to sequential database calls which may or may not be what you want.
If these database calls do not depend on each other, then you can call the promise-based wrappers in parallel using Promise.all() method.
The following code example shows how you can call your promise-based wrapper functions in parallel using Promise.all().
app.get('/pets', async function(req, res, next) {
try {
// "petsData" will be an array that will contain all the data from
// three database calls.
const petsData = await Promise.all([getCats(), getDogs(), getBudgies()]);
// render the pub template, passing in the data
// fetched from the database
...
catch (error) {
...
}
});
I hope this is enough to help you get rid of the nested database calls in your current code and start using promises in your code.
If you're trying to use MySQL with Nodejs, the module you should be looking for is mysql2 rather than mysql.
mysql2 provides a promise based approach and is a much refined version of mysql module for nodejs.
For example, for executing a query,
in mysql
con.query(sql_query, (err, rows, field)=>{ //some code here }
in mysql2, you can use the async approach as well as promise approach. Also, prepared statements in mysql2 are more easier than mysql.
//async approach
class A {
static async fn(sql, params){
const [data] = await con.execute(sql, [params]);
return data;
}
}
//promise approach remains same as **mysql** itself.
Here's the documentation for
mysql2 & more docs
If your database calls returned promises instead of using callbacks, you could:
const cats = await db.cats();
const dogs = await db.dogs();
const budgies = await db.budgies();
res.render('bigScreen.pug', {
cats : cats,
k9s: dogs,
budgies: budgies,
pretty : true
});
// Or request them all in parallel instead of waiting for each to finish
const [
cats,
dogs,
budgies
] = Promise.all([
dg.cats(),
dg.dogs(),
db.budgies()
]);
Simply convert the mysql functions into promises using the nodejs standard lib util.promisify
example:
const { promisify } = require('util');
const catsPromise = promisify(db.cats);
const dogsPromise = promisify(db.dogs);
const budgiesPromise = promisify(db.budgies);
async function routeHandler() {
let err = null;
try {
const cats = await catsPromise();
const dogs = await dogsPromise();
const budgies = await budgiesPromise();
} catch(error) {
err = error;
}
if (err) {
console.log(err);
// you should res.end() or res.render(someErrorPage) here
// failure to do so will leave the request open
} else {
res.render('bigScreen.pug', {
'cats' : cats,
'k9s': dogs,
'budgies': budgies,
'pretty' : true
});
}
}
Promise.all() method seems a more famous and cleaner way to make multiple calls in parallel like your use case.
But there is one more alternate way. : Multiple statement queries
To use this feature you have to enable it for your connection:
var connection = mysql.createConnection({multipleStatements: true});
Once enabled, you can execute multiple statement queries like any other query:
db.query('SELECT cats; SELECT dogs', function (error, results, fields) {
if (error) throw error;
// `results` is an array with one element for every statement in the query:
console.log(results[0]); // [{cat1,cat2}]
console.log(results[1]); // [{dog1,dog2}]
});
It is technically more efficient as requires less back and forth with MySQL connection.
(However, this feature is disabled by default as it allows for SQL injection attacks if values are not properly escaped). To use this feature you have to enable it for your connection.)

Next.js MySQL INSERT/UPDATE query never seems to execute

Quite the odd issue here.. I think this may be more of a problem of debugging, however I'm going to post in-case it is truly an issue and I'm quite frankly at my wits end anyway. I am doing a basic React.js/next.js form that takes a few inputs and adds them to state, then using axios sends the update to the api, which then makes a query insert or update to MySQL. The problem is, this Insert/Update doesn't work and I can't get any error output besides generic ETIMEDOUT from time to time, which I'm not even sure are related. I had this fixed before but am still unsure what I did. ALL other queries on the site work fine, the connection to the MySQL (AWS RDS) database is just fine.
My theories are A) the final query syntax has a silly issue causing this to just get lost in the abyss, or B) there's some server side code trying be run client side that I don't quite understand. (have also gotten the module 'fs' not found), or C) an async issue that I am not weathered enough in next.js to fix. And before you say it, yes there is data to be updated in the table, it is not trying to update the same data and thus bypassing the update. It is new data, every time I test.
NOTE-- I should also say, this code works PERFECT on my local osx environment. This ONLY happens when I try to run this on my Vercel deployment environment. This is important to know. The Database and Code are the EXACT same between both environments.
Without further ado, some code:
To save code display, lets assume our values are in state and ready to go to the API, as I know for a fact they are, and they make it to the actual query.
handleSubmit - gets run when the form is submitted.
const handleSubmit = (e) => {
e.preventDefault();
// Loop data, create a list of IDs for the Delete and an
// array of array of arrays for the insert.
let segmentItemIDList = [];
const segmentItemArray = [];
originalSegmentItemList = originalSegmentItemList.join(',')
segmentItemState.map((val, idx) => (
segmentItemArray[idx] = [
segmentItemState[idx].segmentID,
Number(segmentItemState[idx].chronologicalOrder),
Number(segmentItemState[idx].releaseOrder),
segmentItemState[idx].name,
segmentItemState[idx].typeID
]
))
let action = 'updatesegmentitem'
axios.post('/api/list', { action, segmentItemArray })
.then((result) => {
action = 'deletesegmentitem'
axios.post('/api/list', { action, originalSegmentItemList })
.then((result) => {
alert("Updated!!");
})
.catch(error => console.error('Error:', error));
})
.catch(error => console.error('Error:', error));
}
api/list (assume it gets into this block, because it does)
else if(req.body.action == 'updatesegmentitem') {
console.log("2. API updatesegmentitem req.body: ", req.body);
const segmentItemArray = req.body.segmentItemArray;
console.log("SegmentItemArray: ", segmentItemArray);
try {
if(Array.isArray(segmentItemArray) && segmentItemArray.length > 0) {
console.log("Inside IsArray: ", segmentItemArray);
const segmentItemInsertResults = await insertBatchSegmentItems(segmentItemArray);
res.send(segmentItemInsertResults);
} else {
res.send(true);
}
} catch (e) {
res.send('error');
}
insertBatchSegmentItems (mysql query) .. Sometimes I get the console logs in here, sometimes not..
export async function insertBatchSegmentItems(segmentItemData) {
let mysqlConnection = mysql.createConnection({
host: process.env.MYSQL_HOST,
database: process.env.MYSQL_DATABASE,
user: process.env.MYSQL_USER,
password: process.env.MYSQL_PASSWORD,
debug: false,
});
mysqlConnection.connect();
const insertSQL = 'INSERT INTO segmentItem (segmentID, chronologicalOrder, releaseOrder, name, typeID) VALUES ?'
try {
await mysqlConnection.query(insertSQL, [segmentItemData], function(err, result) {
console.log("Connex Query Inside Result: ", result);
if (err) throw err;
//mysqlConnection.destroy();
return result;
});
} catch (e) {
console.log("ERROR: ", e);
//mysqlConnection.destroy();
return e;
}
return true;
}
Please excuse my mess, I have been trying so many different things to try and get this to work but it will be cleaned up after a solution has been found.
Whenever I run into similar situations, I usually drop out exception handling and let it fail hard. It might give you a better insight of where it's happening. Good luck!

Knex Transaction - Using Await - Not executing 2nd SQL Statement

I am using knex 0.19.4 in node js 10.x. I have 2 SQL statements - Insert and Update which has to happen as a transaction.
// Using Var, so that below code has access to this variable
var sqlStageInsert = kx('stage').insert({
officeid: 'OFF000',
taskid: 'T002',
});
var sqlTaskPcUpdate = kx('task')
.update({ pc: 100})
.where('task.taskno', taskno)
.limit(1);
1st Try - 2nd SQL sqlTaskPcUpdate Not getting Executed
const sqlUpdateInsert = kx.transaction(function (trx) {
sqlStageInsert.transacting(trx)
.then(function () {
console.log(sqlTaskPcUpdate.toString()); // This is outputing correct SQL
return sqlTaskPcUpdate.transacting(trx);
})
.then(trx.commit)
.catch(trx.rollback);
});
await sqlUpdateInsert;
2nd Try - Getting error Transaction query already complete. This is based on Commit/rollback a knex transaction using async/await
await kx.transaction(async (trx) => {
try {
await sqlStageInsert.transacting(trx);
await sqlTaskPcUpdate.transacting(trx);
trx.commit();
} catch (error) {
trx.rollback();
throw error;
}
});
I would suggest you to try inserting the data in stage table first and then retrieve a common value which belongs to both the table for applying in where class of updating task table(Assuming both the table contain any one common column having same data).
Please note that as per knexjs.org website, knex.transaction() uses returning statement with respect to PostgreSQL to insert/ update more than one table to maintain consistency, however MySQL won't support transaction due to which I'm using return values in below code.
FYR, http://knexjs.org/#Transactions
Please refer below code snippet for your reference :
db.transaction(trx => {
trx.insert({
officeid: 'OFF000',
taskid: 'T002'
})
.into('stage')
.then(() => {
return trx('stage').where({taskid}).then(resp => resp[0].taskno)
})
.then(stageTaskno => {
return trx('task')
.update({pc: 100})
.where({stageTaskno})
.limit(1)
.then(resp => {
resp.json(resp)
})
})
.then(trx.commit)
.catch(trx.rollback)
});
hope this is helpful, cheers!

How can i convert typescript callback to promise

i want to make a method in my class. This method should connect to a MySQL database. I created my SQL code. And now I don't want to do a callback because this is old, I want to start using promises.
My function with callback (old school):
public does_player_exist(username: string, callback: any) {
this.mysql.connect();
this.mysql.query('USE devdb');
this.mysql.query('SELECT p_name FROM players WHERE p_name = "'+username+'"', (err: Error, result: any[]) {
if (result.length === 1) {
callback(true)
} else {
callback(false);
}
});
}
And here follows the method I tried to make a promise, but I failed:
public does_player_exist(username: string): Promise<boolean> {
this.mysql.connect();
this.mysql.query('USE devdb');
return this.mysql.query('SELECT p_name FROM players WHERE p_name = "'+username+'").toPromise().then((result) => {
return result.length === 1;
})
}
When I call this method:
service.does_player_exist('test').then((result) => { console.log(result) })
I hope someone can help me. Because I really don't want to be old school forever xD
Thanks in advance.
create a new Promise and resolve it / reject it in the callback functions of query. then return the promise. now does_player_exist returns a Promise object which contains for example then function
public does_player_exist(username: string, callback: any): Promise<boolean> {
this.mysql.connect();
this.mysql.query('USE devdb');
var promise = new Promise<boolean>();
this.mysql.query('SELECT p_name FROM players WHERE p_name = "'+username+'"', (err: Error, result: any[]) {
if (!err) promise.resolve(!!result.length);
else promise.reject();
});
return promise;
}
you will have to make sure that you have a Promise class available. that depends on your environment.
please be aware that without sanitizing your input (username) your application will be vulnerable and attackers could hijack your app.
Tip #1. Please use some kind of factory function to build your connection and reuse it later on.
Tip #2. Use prepared statement to prevent SQL injection.
Tip #3. Please use some promise library for that like Bluebird or Q. Most of the 3rd party promise libraries have a lot of useful utility methods to work with promises on of them is promisify. Promisify can wrap any nodejs type callback function into function which returns a promise.
Your example would look like:
// it's not a factory function*
public query() {
this.mysql.connect();
this.mysql.query('USE devdb');
return Promise.promisify(this.mysql.query.bind(this.mysql))
}
public does_player_exist(username: string): Promise<boolean> {
return this
.query('SELECT p_name FROM players WHERE p_name = ?', [username])
.then(result => result.length === 1);
}
I wrote a simple npm package to do this in a way that retains type-safety:
ts-promisify-callback
It requires you use typescript#^4.0 (upcoming release) and currently only works for callbacks in the popular format of (err, result) => {}.

Yielding streams in generator functions

I have recently started using Q.spawn function to run generators that yield promises. This works well in browsers where the support for streams is yet to land but in case of node we have streams. If you're using streams inside a generator function and would like to yield once writer stream is done then your code becomes not so clean.
Q.spawn(function* () {
yield new Promise(resolve => {
let fromStream = fs.createReadStream('x.txt');
let toStream = fs.createWriteStream('y.txt');
toStream.on('end', () => resolve());
fromStream.pipe(toStream);
});
});
It works but as soon as I start dealing with a lot streams the code becomes really ugly. Can this be made as simple as following snippet?.
someGeneratorFuncRunner(function* () {
yield fs.createReadStream('x.txt')
.pipe(fs.createWriteStream('y.txt'));
});
You don't need to put most of that code inside the Promise constructor:
Q.spawn(function* () {
let fromStream = fs.createReadStream('x.txt');
let toStream = fs.createWriteStream('y.txt');
fromStream.pipe(toStream);
yield new Promise(resolve => {
toStream.on('end', resolve);
});
});
And of course if you're waiting for lots of streams, it would make sense to factor out the promise constructor call into a helper function:
function endOf(stream) {
return new Promise(resolve => {
stream.on('end', resolve);
});
}