I'm using Sequelize ORM for my nodejs project and it seems there is a mistake in configuration or I'm doing something really awry with the queries... The problem is that our client will make more than 60 connections sometimes when the user is requesting a lot of information at once (graphql request with deep structure and data loaders on top for a lot of records fetched from MySQL). The pool settings should not let do that since it is configured for 10 max connections at once. The main question is how those connections appear? The app is being hosted at AWS using ELB, for MySQL we are using RDS and we can have up to 2 app instances at once, so no more than 20 connections at all. Sequelize config:
production: {
username: process.env.DB_USER,
password: process.env.DB_PASS,
database: process.env.DB_NAME,
host: process.env.DB_HOST,
dialect: 'mysql',
define: {
charset: 'utf8mb4',
collate: 'utf8mb4_general_ci',
},
pool: {
max: parseInt(process.env.DB_POOL_SIZE, 10), // 10 from ENV variables
min: 0,
acquire: 60000,
idle: 60000,
},
operatorsAliases: 0,
logging: false,
},
once there are >50 connections on our RDS client we can see that our ELB can't perform health check since it can't make new connections and return positive status from MySQL. so our app becomes unreachable.
After bombing the backend with this particular query I've fetched processlist for MySQL. Some sleeping queries will go to more than 1000 "time".
Also I'm going to attach my Sequelize connection config. It is a singleton object which will be run only once the app is loaded. But on the other hand, I'm importing models from this file everywhere in the project. But is it a mistake?
/* eslint-disable import/no-cycle */
import { Sequelize } from 'sequelize-typescript';
import databaseConfig from '#config/database';
import models from './models';
const env = process.env.NODE_ENV || 'development';
const database = (databaseConfig as any)[env];
const sequelize = new Sequelize(database.database, database.username, database.password, {
host: database.host,
dialect: database.dialect,
define: database.define,
pool: database.pool,
operatorsAliases: database.operatorsAliases,
logging: database.logging,
retry: {
match: [
/SequelizeConnectionError/,
/SequelizeConnectionRefusedError/,
/SequelizeHostNotFoundError/,
/SequelizeHostNotReachableError/,
/SequelizeInvalidConnectionError/,
/SequelizeConnectionTimedOutError/,
],
max: Infinity,
},
});
sequelize.addModels(Object.values(models));
setTimeout(() => console.log('Waiting for mysql connection...'), 200);
sequelize.sync().then(() => console.log('Mysql connection established'));
export const {
... 16 MODELS...
} = models;
export { sequelize, Sequelize };
also I'm using sequelize suggested config in .sequelizerc file to make a connection when running migrations etc..
const path = require('path');
module.exports = {
'config': path.resolve('dist', './config/database.js'),
'models-path': path.resolve('dist', './models'),
'seeders-path': path.resolve('dist', './seeders'),
'migrations-path': path.resolve('dist', './migrations')
}
What I'm missing here?
Related
As the title says, I have a series of azure functions (function app) which call on a database (mysql) also deployed on azure using sequelize and all of them are in the same resource group (deployed to us-east-2).
When I run the api locally everything works, I am able to make calls to the api and it is able to communicate to the DB. I can deploy everything just fine using Serverless Framework but when I try to make a call to one of the deployed endpoints I get the following 500 error returned:
{
"name": "SequelizeConnectionError",
"parent": {
"errno": "EACCES",
"code": "EACCES",
"syscall": "connect",
"address": "127.0.0.1",
"port": 3306,
"fatal": true
},
"original": {
"errno": "EACCES",
"code": "EACCES",
"syscall": "connect",
"address": "127.0.0.1",
"port": 3306,
"fatal": true
}
}
Could this connection error be due to my credentials not being deployed with sequelize? If so, how do I deploy them? I have them stored as environment variables in a local.settings.json file. I checked the firewall and since this is a Proof of Concept with dummy data I am just letting everything in so it can't be that.
Here is my sequelize connection:
const {
Sequelize,
} = require("sequelize");
const DATABASE_HOST = process.env["DATABASE_HOST"];
const DATABASE_CATALOG = process.env["DATABASE_CATALOG"];
const DATABASE_PORT = process.env["DATABASE_PORT"];
const DATABASE_USER = process.env["DATABASE_USER"];
const DATABASE_PASSWORD = process.env["DATABASE_PASSWORD"];
const sequelize = new Sequelize(DATABASE_CATALOG, DATABASE_USER, DATABASE_PASSWORD, {
host: DATABASE_HOST,
port: DATABASE_PORT,
dialect: 'mysql',
ssl: true,
"dialectOptions": {
"ssl": {
"require": true
}
},
pool: {
max: 20,
min: 0,
acquire: 120000,
idle: 10000
}
});
module.exports = sequelize;
Any help would be greatly appreciated, its a side project and I have been staring at this for way too long.
you can add the required configuration setting physically inside the azure portal.
function app >> configuration >> application setting
Click on new application settings
After adding the required setting like your credentials, you have to restart the function.
Now you can call this setting in you app in a similar way like this :-
const port = process.env.PORT ;
i am facing thi error whenever i make a second call to the api i created using mysql2+expressjs+clearDB(on HERUKO)+heruko deployment server
this is my code ....
const [rows,fields]= await db.execute(MYQUERY,[myarguments])
i am calling the mysql server like this with asyn/await functionality which mysql2 provides
import mysql from 'mysql2/promise'
after that i create the the connection
let db = await mysql.createconnection({
host: 'localhost',
user: "root",
password: "asdzxcasdzxc123123",
database: "doctorapp"
})
export { db }
Any sollution to this error...???
i was facing this error in mysql2/promise
Reason for this error
The actual problem is with mysql2/promise .
when you create your connection with mysql database with mysql2 npm package
this thing is causing the error...
Sollution to this error
i solved this error by creating my connection with mysql.creatPool(yourCredentials)
creat your db config file like this
import mysql from 'mysql2'
let db = mysql.createPool({
host: 'Enter your host here',
user: "enter user name here",
password: "enter you password",
database: "enter the database name",
waitForConnections: true,
connectionLimit: 10,
queueLimit: 0
})
export { db }
Use of db in your contollers or router should be like this
const promise = db.promise()
const select_user_query=`
select *
from users
`
const [rows,fields]= await promise.execute(select_user_query)
return res.status(200).json({users:rows})
One thing to note===>
you have to create
const promise= db.promise()
in all the contollers ... so it will automatically handles all the required connnection release or other things required to make the connection working perfectly
I'm using an Amazon RDS hosted MySQL with Multi-AZ Support. Just could not find any information on how to connect Sequelize to Amazon RDS properly so that Sequelize is handling fail-overs etc. accordingly?
I'm just using the following config, but do not know if this is enough or recommended?
sequelizeConfig = {
logging: false,
pool: { maxConnections: 5, maxIdleTime: 30},
sequelizeConfig[dialectOptions] = {
ssl: 'Amazon RDS'
}
}
Using Amazon RDS with Multi-AZ I consider the following is important:
Try reconnecting if connection got lost, until it is available again
Don't cache mysql server ip address too long (Amazon suggests less than 1 min)
Amazon Docs are not writing anything about connection handling and pooling.
Here is how i got connected with my RDS:
var config = require(__dirname + '/../config/config.json')[env];
// your config file will be in your directory
var sequelize = new Sequelize(config.database, config.username, config.password, {
host: '****.****.us-west-1.rds.amazonaws.com',
port: 5432,
logging: console.log,
maxConcurrentQueries: 100,
dialect: 'postgres',
dialectOptions: {
ssl:'Amazon RDS'
},
pool: { maxConnections: 5, maxIdleTime: 30},
language: 'en'
})
The previous answer didn't work for me, after some research, this options object did:
var options = {
host: settings.database.host,
port: settings.database.port,
logging: console.log,
maxConcurrentQueries: 100,
dialect: 'mysql',
ssl: 'Amazon RDS',
pool: { maxConnections: 5, maxIdleTime: 30 },
language: 'en',
}
I'm running a RDS MySQL and a EC2 instance in the same default VPC, this options object worked when connecting a node app from that EC2 with the RDS using sequelize.
When I lift my sails project I get these warnings on terminal
debug: Deprecated: config.adapters.default debug: (see
http://links.sailsjs.org/docs/config/connections) debug: For now, I'll
pretend you set config.models.connection.
debug: Deprecated: config.adapters debug: (see
http://links.sailsjs.org/docs/config/connections) debug: For now, I'll
pretend you set config.connections.
debug: Deprecated: config.adapters.*.module debug: (see
http://links.sailsjs.org/docs/config/connections) debug: For now, I'll
pretend you set config.connections["disk"].adapter.
And my data is not stored in mysql DB.
This is my api model code
module.exports = {
schema:true,
tableName: 'BusStop',
adapters: 'mysql-adapter',
migrate: 'safe',
attributes: {
stopID:{
type: 'int'
},
stopName:{
type: 'string'
},
latitude:{
type: 'float'
},
longitude:{
type: 'float'
}
}
};
This is my local.js code
module.exports = {
port: process.env.PORT || 1337,
environment: process.env.NODE_ENV || 'development',
adapters:{
'default': 'disk',
disk:{
module:'sails-disk'
},
'mysql-adapter': {
module : 'mysql-sails',
host : '127.0.0.1',
user : 'root',
password : '',
database : 'PublicTransport',
schema : true
}
}
};
And this is my connections.js code
module.exports.connections = {
localDiskDb: {
adapter: 'sails-disk'
},
'mysql-adapter': {
module : 'sials-mysql',
host : '127.0.0.1',
port : 3306,
user : 'root',
password : '',
database : 'PublicTransport'
},
someMongodbServer: {
adapter: 'sails-mongo',
host: 'localhost',
port: 27017,
},
somePostgresqlServer: {
adapter: 'sails-postgresql',
host: 'YOUR_POSTGRES_SERVER_HOSTNAME_OR_IP_ADDRESS',
user: 'YOUR_POSTGRES_USER',
password: 'YOUR_POSTGRES_PASSWORD',
database: 'YOUR_POSTGRES_DB'
}
};
I want to know how can I remove these Deprecated warnings and why my data is not being stored in the Database.
Looks like the links in those deprecation warnings are incorrect. For information on how to configure Sails connections, see http://sailsjs.org/#/documentation/reference/sails.config/sails.config.connections.html.
You should be able to make those messages go away by following their instructions regarding what config keys to set.
In order to set a connection for a specific model, you now set the connection property:
module.exports = {
schema:true,
tableName: 'BusStop',
connection: 'mysql-adapter', // "adapter" is now "connection
migrate: 'safe',
attributes: {...}
}
In order to specify the adapter to use for a connection, use the adapter key, not module; you're already doing this in most of your connections, you just need to update mysql-adapter.
In your config/local.js you're using the deprecated adapters key to set up connections; use connections instead.
Lastly, in order to set a default connection for all of your models, you do as the deprecation message says and set sails.config.models.connection rather than sails.config.adapters.default; you can do so easily in the config/models.js file, or in your config/local.js like so:
module.exports = {
models: {
connection: 'mysql-adapter'
},
...more config...
}
This warning can also be generated if your project has an config/adapters.js file. If the file is empty it can be deleted and Zap the warning is gone.If it contains code then you will need to migrate it to the correct path config/models.js and config/connections.js
How to use mysql with Sails ? Also, if I change the database to a mySQL one, do I lose all my current model data ?
I spent some time looking for tutorials and demos but haven't found any.
In order to use mySQL with sails, you will have to:
Define an adapter in config/adapters
Example adapters file:
module.exports.adapters = {
'default': 'disk',
disk: {
module: 'sails-disk'
},
'mysql-adapter': {
module: 'sails-mysql',
host: 'HOST',
user: 'USER',
password: 'PASSWORD',
database: 'DATABASE'
}
};
Make any model use the new mysql adapter (ex. api/models/Contact.js)
module.exports = {
tableName: 'contacts',
adapter: 'mysql-adapter',
migrate: 'safe',
attributes: { ... }
}
Then it should work.