Async/Await in NodeJs and MySql connection - mysql

I've two problems.
Async/Await in queries
Env variables in pool config
I use TypeScript with Node.js and Express and I install promise-mysql but I would not refuse the use of myslq or mysql2. Currently this is my code:
database.ts
import mysql, { PoolConfig } from 'promise-mysql';
import keys from './keys';
const pool = startConnection();
export default pool;
// Functions
function startConnection() {
const pool = mysql.createPool(dbOptions(true));
pool.get('getConnection').then(async () => {
await (await pool).releaseConnection;
console.log('DB is connected.');
});
return pool;
}
function dbOptions(dev: boolean): PoolConfig {
var dbInfo: PoolConfig;
if (dev) {
dbInfo = {
database: process.env.DB || keys.DB.DB,
host: process.env.DB_HOST || keys.DB.DB_HOST,
user: process.env.DB_USER || keys.DB.DB_USER,
password: process.env.DB_U_PASS || keys.DB.DB_U_PASS
};
} else {
dbInfo = {
database: process.env.DB,
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_U_PASS
};
}
return dbInfo;
}
And a query looks like this:
import pool from 'database';
const books: Book[] = await (await pool).query('SELECT * FROM book WHERE pages => ?', [minPages]);
So, i don't want the second await when i use the pool (if it's possible), and the dbOptions() don't works because the startConnection() function starts before the dotenv.config() in index.ts then the procces.env.something returns undefined each time, so how can i fixed that?.
Thanks for help! 😁

So, I cannot view all your code to help with dotenv issue, but you can pass await pool as a paramter for a callback function that starts your app. Anyway, I would implement your code by using mysql2:
database.ts:
import { createPool, Pool, PoolOptions } from "mysql2";
import keys from "./keys";
function startConnection(): Pool {
const options: PoolOptions = dbOptions({ isDev: true });
const pool: Pool = createPool(options);
return pool;
}
function dbOptions({ isDev }: {
isDev: boolean;
}): PoolOptions {
let dbConfig: PoolOptions = {
database: process.env.DB ?? (isDev &&= keys.DB.DB),
host: process.env.DB_HOST ?? (isDev &&= keys.DB.DB_HOST),
user: process.env.DB_USER ?? (isDev &&= keys.DB.DB_USER),
password: process.env.DB_U_PASS ?? (isDev &&= keys.DB.DB_U_PASS),
};
return dbConfig;
}
export const connectionPool = startConnection();
query.ts
import { connectionPool } from "database";
const books: Book[] = await connectionPool.query("SELECT * FROM book WHERE pages => ?", [minPages]);

Related

Connecting Cypress V10++ into sql database [duplicate]

I get an error when running the integration tests:
0 passing (17s)
1 failure
1) Registration page
register new users allowed and update status in the database:
TypeError: Net.connect is not a function
at new Connection (webpack:///./node_modules/mysql2/lib/connection.js:50:0)
at ./node_modules/mysql2/index.js.exports.createConnection (webpack:///./node_modules/mysql2/index.js:10:0)
at Context.eval (webpack:///./cypress/integration/registration.spec.js:23:34)
Here is my environment:
MySQL Workbench
MySQL Server 8.0.29
I raised local backendless, I have access to the database. Here is my code:
const mysql2 = require('mysql2');
describe('Registration page', () => {
beforeEach(() => {
// visit the registration page
cy.visit('http://localhost:3000/registration');
});
it('register new users allowed and update status in the database', () => {
// fill out the registration form
cy.get('input[name="fullName"]').type("Nazar Dmytryshyn")
cy.get('input[type="email"]').type('testuser#example.com');
cy.get('input[name="pwd"]').type('testpassword');
cy.get('input[name="confirmPassword"]').type('testpassword');
// submit the form
cy.get('button[class="btn btn-success"]').click();
// check that the user is redirected to the login page
cy.url().should('include', '/login');
// create a connection to the test database
const connection = mysql2.createConnection({
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'local1'
});
// open the connection
connection.connect();
// update the developer status in the database
connection.query(
'UPDATE `main_backendless`.`Developer` SET `developerStatusId` = "1" WHERE (`email` = "testuser#example.com")',
(error, results) => {
if (error) throw error;
expect(results.affectedRows).to.equal(1);
}
);
// close the connection
connection.end();
});
});
I checked this data 10 times, it is correct and I can connect to the database through MySQL WorkBench
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'main_backendless'
I will be grateful for any ideas that can be achieved!
I recommend using the cypress-mysql, which hides a lot of the implementation details for you.
If you try to roll your own task, you may end up with an undefined return value.
Install
npm install cypress-mysql
//or
yarn add cypress-mysql
Configure
The release notes are out of date, here is the configuration for Cypress 10+
// cypress.config.js
const { defineConfig } = require("cypress");
const mysql = require('cypress-mysql');
module.exports = defineConfig({
// ...
e2e: {
setupNodeEvents(on, config) {
mysql.configurePlugin(on);
},
"env": {
"db": {
"host": "localhost",
"user": "user",
"password": "password",
"database": "database"
}
}
})
// cypress/support/e2e.js
const mysql = require('cypress-mysql');
mysql.addCommands();
Test
const sql = 'UPDATE "main_backendless.Developer" SET "developerStatusId" = "1" WHERE ("email" = "testuser#example.com")'
cy.query(sql).then(res => {
expect(res.affectedRows).to.equal(1)
});
If you want to use a task to call the mySql library, you must return a Promise from the task.
This is because the mysql calls are asynchronous, and the only way Cypress knows to wait for them is to get a promise returned from your code.
cypress.config.js
const { defineConfig } = require("cypress")
const mysql2 = require('mysql2')
const connection = mysql2.createConnection({
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'local1'
})
module.exports = defineConfig({
e2e: {
setupNodeEvents(on, config) {
on('task', {
mySql: (sql) => {
return new Promise((resolve, reject) => {
connection.query(sql, (error, results) => {
if (error) {
reject(error)
} else {
resolve(results.affectedRows)
})
})
})
}
})
},
})
it('tests with mysql', () => {
cy.task('mySql', 'sql staement here')
.then(result => {
expect(result).to.equal(1);
})
})
With Promise-wrapper
Alternatively, mysql2 provides a promise-wrapper that can simplify your code:
const { defineConfig } = require("cypress")
const mysql = require('mysql2/promise') // different import here
const connection = mysql2.createConnection({
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'local1'
})
module.exports = defineConfig({
e2e: {
setupNodeEvents(on, config) {
on('task', {
mySql: async (sql) => { // async here
const result = await connection.execute(sql) // await here
return result;
}
})
},
})
The issue is that you're using a nodejs library with Cypress. Cypress tests execute inside a browser and cannot directly utilize nodejs libraries within a test.
To do what you want to need to create a cy.task() to be able to execute code in nodejs.
Assuming you're using js, create a file with a function to use the sql connector
// runSql.js
const mysql2 = require('mysql2');
function runSql(sql) {
const connection = mysql2.createConnection({
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'local1'
});
connection.connect();
let rows;
connection.query(sql, (error, results) => {
if (error) throw error;
rows = results.affectedRows
});
connection.end();
return rows;
}
module.exports = runSql;
Then in the cypress.config.js file
const runSql = require('./runSql.js');
module.exports = defineConfig({
// ...
e2e: {
setupNodeEvents(on, config) {
on('task', {
runSql
});
},
});
Now to call the task in a test
describe('Registration page', () => {
it('test', () => {
const sql = 'UPDATE `main_backendless`.`Developer` SET `developerStatusId` = "1" WHERE (`email` = "testuser#example.com")';
cy.task('runSql', sql).then((rows) => {
expect(rows).to.equal(1);
});
});
});

Can't connect to MySQL database when testing in Cypress (mysql2)

I get an error when running the integration tests:
0 passing (17s)
1 failure
1) Registration page
register new users allowed and update status in the database:
TypeError: Net.connect is not a function
at new Connection (webpack:///./node_modules/mysql2/lib/connection.js:50:0)
at ./node_modules/mysql2/index.js.exports.createConnection (webpack:///./node_modules/mysql2/index.js:10:0)
at Context.eval (webpack:///./cypress/integration/registration.spec.js:23:34)
Here is my environment:
MySQL Workbench
MySQL Server 8.0.29
I raised local backendless, I have access to the database. Here is my code:
const mysql2 = require('mysql2');
describe('Registration page', () => {
beforeEach(() => {
// visit the registration page
cy.visit('http://localhost:3000/registration');
});
it('register new users allowed and update status in the database', () => {
// fill out the registration form
cy.get('input[name="fullName"]').type("Nazar Dmytryshyn")
cy.get('input[type="email"]').type('testuser#example.com');
cy.get('input[name="pwd"]').type('testpassword');
cy.get('input[name="confirmPassword"]').type('testpassword');
// submit the form
cy.get('button[class="btn btn-success"]').click();
// check that the user is redirected to the login page
cy.url().should('include', '/login');
// create a connection to the test database
const connection = mysql2.createConnection({
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'local1'
});
// open the connection
connection.connect();
// update the developer status in the database
connection.query(
'UPDATE `main_backendless`.`Developer` SET `developerStatusId` = "1" WHERE (`email` = "testuser#example.com")',
(error, results) => {
if (error) throw error;
expect(results.affectedRows).to.equal(1);
}
);
// close the connection
connection.end();
});
});
I checked this data 10 times, it is correct and I can connect to the database through MySQL WorkBench
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'main_backendless'
I will be grateful for any ideas that can be achieved!
I recommend using the cypress-mysql, which hides a lot of the implementation details for you.
If you try to roll your own task, you may end up with an undefined return value.
Install
npm install cypress-mysql
//or
yarn add cypress-mysql
Configure
The release notes are out of date, here is the configuration for Cypress 10+
// cypress.config.js
const { defineConfig } = require("cypress");
const mysql = require('cypress-mysql');
module.exports = defineConfig({
// ...
e2e: {
setupNodeEvents(on, config) {
mysql.configurePlugin(on);
},
"env": {
"db": {
"host": "localhost",
"user": "user",
"password": "password",
"database": "database"
}
}
})
// cypress/support/e2e.js
const mysql = require('cypress-mysql');
mysql.addCommands();
Test
const sql = 'UPDATE "main_backendless.Developer" SET "developerStatusId" = "1" WHERE ("email" = "testuser#example.com")'
cy.query(sql).then(res => {
expect(res.affectedRows).to.equal(1)
});
If you want to use a task to call the mySql library, you must return a Promise from the task.
This is because the mysql calls are asynchronous, and the only way Cypress knows to wait for them is to get a promise returned from your code.
cypress.config.js
const { defineConfig } = require("cypress")
const mysql2 = require('mysql2')
const connection = mysql2.createConnection({
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'local1'
})
module.exports = defineConfig({
e2e: {
setupNodeEvents(on, config) {
on('task', {
mySql: (sql) => {
return new Promise((resolve, reject) => {
connection.query(sql, (error, results) => {
if (error) {
reject(error)
} else {
resolve(results.affectedRows)
})
})
})
}
})
},
})
it('tests with mysql', () => {
cy.task('mySql', 'sql staement here')
.then(result => {
expect(result).to.equal(1);
})
})
With Promise-wrapper
Alternatively, mysql2 provides a promise-wrapper that can simplify your code:
const { defineConfig } = require("cypress")
const mysql = require('mysql2/promise') // different import here
const connection = mysql2.createConnection({
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'local1'
})
module.exports = defineConfig({
e2e: {
setupNodeEvents(on, config) {
on('task', {
mySql: async (sql) => { // async here
const result = await connection.execute(sql) // await here
return result;
}
})
},
})
The issue is that you're using a nodejs library with Cypress. Cypress tests execute inside a browser and cannot directly utilize nodejs libraries within a test.
To do what you want to need to create a cy.task() to be able to execute code in nodejs.
Assuming you're using js, create a file with a function to use the sql connector
// runSql.js
const mysql2 = require('mysql2');
function runSql(sql) {
const connection = mysql2.createConnection({
host: '127.0.0.1:3306',
user: 'root',
password: 'rootpassword',
database: 'local1'
});
connection.connect();
let rows;
connection.query(sql, (error, results) => {
if (error) throw error;
rows = results.affectedRows
});
connection.end();
return rows;
}
module.exports = runSql;
Then in the cypress.config.js file
const runSql = require('./runSql.js');
module.exports = defineConfig({
// ...
e2e: {
setupNodeEvents(on, config) {
on('task', {
runSql
});
},
});
Now to call the task in a test
describe('Registration page', () => {
it('test', () => {
const sql = 'UPDATE `main_backendless`.`Developer` SET `developerStatusId` = "1" WHERE (`email` = "testuser#example.com")';
cy.task('runSql', sql).then((rows) => {
expect(rows).to.equal(1);
});
});
});

How to mock promisify call on mysql query nodeJS using sinon and Mocha?

This is my code using mysql -
import * as mysql from 'mysql';
import {promisify} from 'util';
const connectionParams:any = {
/* set as environment variables */
host: host,
user: user,
password: password,
port: parseInt(port)
};
var connection:any;
const getRecords = async (inputValue: string) => {
//validate inputValue
const userIds: string[] = [];
logger.info("Creating mysql connection");
try {
connection = mysql.createConnection(connectionParams);
const query = promisify(connection.query).bind(connection);
const queryResult = await query({ sql: sqlQuery, timeout: 1000, values: value1, inputValue] });
if (queryResult) {
queryResult.forEach((row) => {
userIds.push(row.userid);
});
}
} catch (error) {
logger.info(error);
// console.log(error);
throw new Error('Could not retrieve user IDs');
} finally {
connection.end();
}
return userIds;
};
And this is my test -
it('should return a list of records when right inputs are given', async() => {
sinon.stub(process, 'env').value({
'database': 'TESTDB'
});
let dummyArray = [{ userid: 'xyz' }];
let createConnection = {
connect: function(connectionParams: any) {
return Promise.resolve()
},
query : sinon.stub().withArgs({}).callsFake(function (...args): Promise<Object>{
const dummyArray = [{ userid: 'xyz' }];
return new Promise(function(resolve){resolve(dummyArray)});
}),
end: function() {}
};
let mySqlStub = {
createConnection: sinon.stub().returns(createConnection)
};
const dbops = proxyquire('../../lib/dbops', {'mysql': mySqlStub}).default;
expect(await dbops.getUserIds('Delete')).to.deep.equal(['xyz']);
});
How do I write the fake function for the query?
query : sinon.stub().withArgs({}).callsFake(function (...args):
Promise{
const dummyArray = [{ userid: 'xyz' }];
return new Promise(function(resolve){resolve(dummyArray)});
})
This does not work for me. How can I get this to work? I cannot get the stub function to resolve and return the intended value in the main function. The query just hangs and throws an error after the timeout. The error is happening in "matchingfakes" method within the stub.
proxyquire is used for stubbing the standalone function exports from a module or package. Since mysql is an object, you can stub its methods by sinon.stub(obj, 'method'). You don't need to use use proxyquire package.
Even if you use util.promisify to generate promise versions for the NodeJS error-First callback method(mysql.query(sql, callback), the callback signature is function (error, results, ...args): void). You need to use .callsFake() to create a mock implementation for this method, and trigger the promise version by calling its callback.
And, you should import the function after stubbing the environment variables. Because when you import the ./dbops module, the code in the module scope will be executed immediately, at this time, the environment variables are not stubbed.
E.g.
dbops.ts:
import mysql from 'mysql';
import { promisify } from 'util';
const connectionParams: any = {
host: process.env.HOST,
user: process.env.USER,
password: process.env.PASSWORD,
port: parseInt(process.env.PORT || '3306'),
};
var connection: any;
const getRecords = async (inputValue: string) => {
const sqlQuery = 'SELECT * FROM tests';
const value1 = '';
const userIds: string[] = [];
console.info('Creating mysql connection');
try {
connection = mysql.createConnection(connectionParams);
const query = promisify(connection.query).bind(connection);
const queryResult = await query({ sql: sqlQuery, timeout: 1000, values: value1, inputValue });
if (queryResult) {
queryResult.forEach((row) => {
userIds.push(row.userid);
});
}
} catch (error) {
console.info(error);
throw new Error('Could not retrieve user IDs');
} finally {
connection.end();
}
return userIds;
};
export { getRecords };
dbops.test.ts:
import sinon from 'sinon';
import mysql from 'mysql';
describe('69702002', () => {
it('should return a list of records when right inputs are given', async () => {
sinon.stub(process, 'env').value({
HOST: '127.0.0.1',
USER: 'testuser',
PASSWORD: 'testpwd',
PORT: '3306',
});
const { getRecords } = await import('./dbops');
const dummyArray = [{ userid: 'xyz' }];
let connectionStub = {
query: sinon.stub().callsFake((sql, callback) => {
callback(null, dummyArray);
}),
end: sinon.stub(),
};
sinon.stub(mysql, 'createConnection').returns(connectionStub);
const actual = await getRecords('test input');
sinon.assert.match(actual, ['xyz']);
sinon.assert.calledWithExactly(mysql.createConnection, {
host: '127.0.0.1',
user: 'testuser',
password: 'testpwd',
port: 3306,
});
sinon.assert.calledOnce(connectionStub.end);
});
});
test result:
69702002
Creating mysql connection
✓ should return a list of records when right inputs are given (945ms)
1 passing (952ms)
----------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|-------------------
All files | 90.48 | 50 | 100 | 90 |
dbops.ts | 90.48 | 50 | 100 | 90 | 27-28
----------|---------|----------|---------|---------|-------------------

How to fix too many in _connectionQueue of Pool?

My very simple Node.js code doesn't seem like its connection pool work as it's supposed to do. _connectionQueue of Pool object just gets longer and longer infinitely, and app dies. I mean it does make a pool and there are pre-made connections already, but they are not reusable or insert requests are too many and fast? I'm not sure..
I've tried to put some more connectionLimit like following :
let state = { pool: null }
export const connect = () => {
state.pool = mysql.createPool({
connectionLimit: 200,
host: process.env.DATABASE_HOST || 'localhost',
user: process.env.DATABASE_USER || 'root',
password: process.env.DATABASE_PASSWORD || 'password',
database: process.env.DATABASE_NAME || 'database'
})
}
export const get = () => state.pool
Mostly given job of this server is subscription and insertion. It subscribes several MQTT topics and just tries to insert messages into RDB. About 100 messages arrives every second, and that code looks like this.
mqttClient.on('message', function (topic, message) {
if(topic.includes('sensor')){
try {
const data = JSON.parse(message.toString())
if(validate(data.uuid)){
const params = [data.a, data.b, data.c, ...]
sensor.setStatus(params)
}
} catch(err){
console.error(err)
}
}
}
export const setStatus = (params) => {
const SQL = `INSERT INTO ...`
db.get().query(SQL, params, (err, result) => {
if (err) console.error(err)
})
}
Then, I see this through chrome-devtools
Object
pool: Pool
config: PoolConfig {acquireTimeout: 10000, connectionConfig: ConnectionConfig, waitForConnections: true, connectionLimit: 200, queueLimit: 0}
domain: null
_acquiringConnections: []
_allConnections: (200) [PoolConnection, PoolConnection, …]
_closed: false
_connectionQueue: (11561) [ƒ, ƒ, ƒ, ƒ, …]
_events: {}
_eventsCount: 0
_freeConnections: []
_maxListeners: undefined
__proto__: EventEmitter
__proto__: Object
I've put console.log into setStatus like following :
export const setStatus = (params) => {
const SQL = `INSERT INTO ...`
console.log(`allConnections=${db.get()._allConnections.length}, connectionQueue=${db.get()._connectionQueue.length}`)
db.get().query(SQL, params, (err, result) => {
if (err) console.error(err)
})
}
, and got these.
allConnections=200, connectionQueue=29
allConnections=200, connectionQueue=30
allConnections=200, connectionQueue=31
allConnections=200, connectionQueue=32
allConnections=200, connectionQueue=33
allConnections=200, connectionQueue=34
...
It seems like server created a connection pool very well, but not using those connections. Instead, trying to create a new connection more and more all the time and those requests just get stuck in _connectionQueue.
It appears you are creating a new pool every time you'd like to make a query. The common model is to create a pool once when the application starts, then use connections from that pool as needed (one pool, many connections).
Also if you're using a simple DB model you can simplify access to the pool by making it global. Below is an alternate to your code you might try:
app.js
const mysql = require('mysql');
const connection = mysql.createPool({
host: process.env.DB_HOST || '127.0.0.1',
user: process.env.DB_USER || 'local_user',
password: process.env.DB_PASSWORD || 'local_password',
database: process.env.DB_NAME || 'local_database'
});
global.db = connection;
modules.js
export const setStatus = (params) => {
let SQL = `INSERT INTO ...`
db.query(SQL, params, (err, result) => {
if (err) console.error(err)
console.log(result)
})
}
Documentation for further reference :: https://github.com/mysqljs/mysql#pooling-connections
Edit 1 - Log pool events
db.on('acquire', function (connection) {
console.log('Connection %d acquired', connection.threadId);
});
db.on('connection', function (connection) {
console.log('Pool id %d connected', connection.threadId);
});
db.on('enqueue', function () {
console.log('Waiting for available connection slot');
});
db.on('release', function (connection) {
console.log('Connection %d released', connection.threadId);
});

Node unhandel promises and imports oop

My code have an obj name SQLFeeter that need to do the sql interaction which get the data post it and pass it along I have some problem which is one imports. The babel doesn't work second while I try to get the data and pass it
const express = require('express');
const router = express.Router();
const mysql = require('mysql')
/*
--------------------------------------
This will handel all get requests
--------------------------------------
*/
/*
//sqlInteractuin test
const SqlDataGetter = require('../../sqlInteraction/GetData');
//import SqlDataGetter from "./sqlInteraction/GetData";
let SqlGetter = new SqlDataGetter
*/
class SqlDataGetter {
constructor()
{
this.con = mysql.createConnection({
host: "localhost",
user: "XXX",
password: "XXX",
database: "APP"
});
}
GetClients()
{
let con = mysql.createConnection({
host: "localhost",
user: "XXX",
password: "AAA",
database: "APP"
});
let resultFromSql = null;
con.connect(function(err) {
if (err) throw err;
let sql_query = "SELECT * FROM contacts"
con.query(sql_query , function (err, result, fields) {
if (err) throw err;
//console.log(fields);
console.log(result);
resultFromSql = result;
});
return resultFromSql;
});
}
Tester()
{
//return this.con
//console.log(this.con)
return 'hello world'
}
}
router.get('/' , async (req , res) =>
{
//Need to make an obj that take the data and do all the querys
res.status(200).send("DataBack");
});
router.get('/Clients' , async (req , res) =>
{
let sql_getter = new SqlDataGetter();
const Clients = sql_getter.GetClients();
console.log(Clients);
SqlDataGetter.GetClients()
res.status(200);
res.send({ respond : Clients});
});
While I am trying to run this at first it works on stand alone but when I create the ajax request it saying GetClients is not a function. And when I try to make the connection to be a property of this object as this.con when I activate this.con.query undifend property query of undifend.
If you use promise-mysql instead of mysql then you'll get promises from the method calls, which will make it easier to work with:
const mysql = require('promise-mysql');
Then your class would look like this:
class SqlDataGetter {
constructor() {
this.conPromise = mysql.createConnection({
host: "localhost",
user: "XXX",
password: "XXX",
database: "APP"
});
}
async GetClients() {
const con = await this.conPromise;
const result = await con.query("SELECT * FROM contacts");
console.log(result);
return result;
}
}
Finally, you'd use that class as follows:
router.get('/Clients' , async (req , res) => {
let sql_getter = new SqlDataGetter();
const clients = await sql_getter.GetClients();
console.log(clients);
res.status(200);
res.send({ respond : clients});
});