Our DB is hosted in AWS RDS.
Backend is Nodejs, Mysql, ExpressJs
We followed the same as below node-mysql-doc
var mysql = require('mysql');
var pool = mysql.createPool({
connectionLimit : 10,
host : 'example.org',
user : 'bob',
password : 'secret',
database : 'my_db'
});
pool.query('SELECT 1 + 1 AS solution', function (error, results, fields) {
if (error) throw error;
console.log('The solution is: ', results[0].solution);
});
when we hit API to get data, the DB connection touched almost max-connection exceeds
here is the graph of AWS RDS DB Connection
even though we didn't use the APIs again for 2hours, but still the connections are not released.
It got reduced when we restart the node server.
how to release the connection that showed in AWS graph
Related
function Connect() {
var connection = mysql.createConnection({
host : 'hidden',
user : 'hidden',
password : 'hidden',
database : 'hidden'
});
connection.connect(function(err) {
if (err) {
console.log('Error connecting to Database'.red);
setInterval(Connect, 5000);
connection.end();
}else{
console.log('Connected to Database'.green);
}
});
}
Connect();
I am attempting to make it so my code won't crash if the database loses connection.
It will be connected to a database which is prone to many restarts and updates, meaning that the program will crash upon losing connection.
I tried adding setInterval(Connect, 5000); so it would attempt to reconnect if it loses connection but to no luck.
Can someone help me out where I am going wrong?
Assuming you're using the mysql npm package you can use connection pooling provided by the library:
https://github.com/mysqljs/mysql#pooling-connections
When a previous connection is retrieved from the pool, a ping packet is sent to the server to check if the connection is still good.
It's rather easy to use, instead of using createConnection you can use createPool
var pool = mysql.createPool({
connectionLimit : 2,
host : 'example.org',
user : 'bob',
password : 'secret',
database : 'my_db'
});
And then use the pool to query the database.
It has many additional benefits as well e.g. load balancing connections.
I have setup a VPC and an RDS database using mysql through a nodejs lambda using serverless.
The issue I am having is that I get a server internal error when testing the lambda. The lambda is using the same VPC as the RDS.
Could this be a permission issue where the lambda needs direct permission to the db instance. If so does anyone have any suggestions on what permissions would be required.
Thankyou
Here is part of the code I am using, this is to test a query and log the result to cloudwatch. It does not show up and only shows timed out.
it seems to work locally using serverless. This is just for educational purposes.
let pool = mysql.createPool({
host: host.length > 0 ? host : body.host,
port: port.length > 0 ? port : body.port,
user: body.username,
password: body.password,
database: body.dbname
});
pool.getConnection(function(err, connection) {
connection.query('SELECT 1 + 1 AS result', function (error, results, fields) {
// And done with the connection.
//connection.release();
// Handle error after the release.
if (error) throw error;
else Logger.info(JSON.stringify(results));
});
});
return {statusCode:200, body: JSON.stringify({})};
}
I am trying to connect mysql with nodejs, but there is a problem in my code, please help!
var express = require("express");
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'localhost',
user : 'root',
password : 'admin',
database : 'dbname',
});
var app = express();
connection.connect(function(err){
if(!err) {
console.log("Database is connected ... \n\n");
} else {
console.log("Error connecting database ... \n\n");
}
});
app.listen(3000);
in terminal i am getting this output:
voldemort#Lenovo:~/Documents/cloudprint$ node server.js
Error connecting database ...
The MySQL client establishes a connection to the server using secure authentication by default.
insecureAuth: Allow connecting to MySQL instances that ask for the old (insecure) authentication method. (Default: false) 1
Looking at the error traceback in your comment, it looks like your server isn't currently configured to allow secure authentication. 2
If your MySQL server version is <= 5.6.4 then secure auth is off and you may want to enable the old auth protocol by enabling insecureAuth when making the connection in the client.
var connection = mysql.createConnection({
host : 'localhost',
user : 'root',
password : 'admin',
database : 'dbname',
insecureAuth: true
});
However, I advise updating the passwords for your users to use the more secure password hashing function if your MySQL server version is >= 5.6.5.
There is step-by-step documentation to accomplish this for user accounts.
You may also want to update your MySQL server if you find that you're running an older version that doesn't support the secure auth protocol.
We are currently experiencing what I can only describe as random intermittent timeouts between AWS Lambda and RDS. After deploying our functions and running them successfully, they can randomly switch to a state of timing out with no configuration changes. Important to note, we are also monitoring the DB connections and can confirm that we aren't running into a max connection issue.
Here are the details on our setup:
Code being executed (using Node.JS v. 6.10):
const mysql = require('mysql');
exports.dbWrite = (events, context, callback) => {
const db = mysql.createConnection({
host: <redacted>,
user: <redacted>,
password: <redacted>,
database: <redacted>
});
db.connect(function (err) {
if (err) {
console.error('error connecting: ' + err.stack);
return;
}
console.log('connected !');
});
db.end();
};
We are using the Node.JS mysql library, v. 2.14.1.
From a networking perspective:
The Lambda function is in the same VPC as our RDS instance
The Lambda function has subnets assigned, which are associated with a routing table that does not have internet access (not associated with an internet gateway)
The RDS database is not publicly accessible.
A security group has been created and associated with the Lambda function that has wide open access on all ports (for now - once DB connectivity is reliable, that will change).
The above security group has been whitelisted on port 3306 within a security group associated with the RDS instance.
CloudWatch error:
{
"errorMessage": "connect ETIMEDOUT",
"errorType": "Error",
"stackTrace": [
"Connection._handleConnectTimeout
(/var/task/node_modules/mysql/lib/Connection.js:419:13)",
"Socket.g (events.js:292:16)",
"emitNone (events.js:86:13)",
"Socket.emit (events.js:185:7)",
"Socket._onTimeout (net.js:338:8)",
"ontimeout (timers.js:386:14)",
"tryOnTimeout (timers.js:250:5)",
"Timer.listOnTimeout (timers.js:214:5)",
" --------------------",
"Protocol._enqueue
(/var/task/node_modules/mysql/lib/protocol/Protocol.js:145:48)",
"Protocol.handshake
(/var/task/node_modules/mysql/lib/protocol/Protocol.js:52:23)",
"Connection.connect
(/var/task/node_modules/mysql/lib/Connection.js:130:18)",
"Connection._implyConnect
(/var/task/node_modules/mysql/lib/Connection.js:461:10)",
"Connection.query
(/var/task/node_modules/mysql/lib/Connection.js:206:8)",
"/var/task/db-write-lambda.js:52:12",
"getOrCreateEventTypeId (/var/task/db-write-lambda.js:51:12)",
"exports.dbWrite (/var/task/db-write-lambda.js:26:9)"
]
}
Amongst the references already reviewed:
https://forums.aws.amazon.com/thread.jspa?threadID=221928
(the invocation ID in CloudWatch is different on all timeout cases)
pretty much every post in this list: https://stackoverflow.com/search?q=aws+lambda+timeouts+to+RDS
In summary, the fact that these timeouts are intermittent makes this an issue that is totally confusing. AWS support has stated that NodeJS-mysql is a third-party tool, and is technically not supported, but I know folks are using this technique.
Any help is greatly appreciated!
Considering that the RDS connections are not exhausted, there is a possibility that the lambda running into a particular subnet is always failing to connect to db. I am assuming that the RDS instances and lambdas are running in separate subnets. One way to investigate this is to check flow logs.
Go to EC2 -> Network interfaces -> search for lambda name -> copy eni ref and then go to VPC -> Subnets -> select the subnet of lambda -> Flow Logs -> search by eni ref.
If you see "REJECT OK" in your flow logs for your db port means that there is missing config in Network ACLs.
Updating this issue: It turns out that the issue was related to the fact that the database connection was being made within the handler! Due to the asynchronous nature of Lambda and Node, this was the culprit for the intermittent timeouts.
Here's the revised code:
const mysql = require('mysql');
const database = getConnection();
exports.dbWrite = (events, context, callback) => {
database.connect(function (err) {
if (err) {
console.error('error connecting: ' + err.stack);
return;
}
console.log('connected !');
});
db.end();
function getConnection() {
let db = mysql.createConnection({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASS,
database: process.env.DB_NAME
});
console.log('Host: ' + process.env.DB_HOST);
console.log('User: ' + process.env.DB_USER);
console.log('Database: ' + process.env.DB_NAME);
console.log('Connecting to ' + process.env.DB_HOST + '...');
return db;
}
ANy idea how can I connect from NodeJs to the Mysql ClearDB on Heroku?
I was able to connect to the ClearDB Mysql runing on Heroku from Navicat, I even created a table called t_users. But I'm having problems connecting from NodeJs from Heroku.
This is my code, it is very simple: everything works find until it tries to connect to MySQL
web.js:
var express = require("express");
var app = express();
app.use(express.logger());
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'us-cdbr-east-04.cleardb.com',
user : '',
password : '',
database : ''
});
connection.connect();
app.get('/', function(request, response) {
response.send('Hello World!!!! HOLA MUNDOOOOO!!!');
connection.query('SELECT 1 + 1 AS solution', function(err, rows, fields) {
if (err) throw err;
response.send('The solution is: ', rows[0].solution);
});
});
var port = process.env.PORT || 5000;
app.listen(port, function() {
console.log("Listening on " + port);
});
This is what I got when I run on the command line: Heroku config
C:\wamp\www\Nodejs_MySql_Heroku>heroku config
=== frozen-wave-8356 Config Vars
CLEARDB_DATABASE_URL: mysql://b32fa719432e40:87de815a#us-cdbr-east-04.cleardb.com/heroku_28437b49d76cc53?reconnect=true
PATH: bin:node_modules/.bin:/usr/local/bin:/usr/bin:/bin
This is the LOG:
http://d.pr/i/cExL
ANy idea how can I connect from NodeJs to the Mysql ClearDB on Heroku?
Thanks
Try this. Hope this will help you
mysql://b32fa719432e40:87de815a#us-cdbr-east-04.cleardb.com/heroku_28437b49d76cc53?reconnect=true
var connection = mysql.createConnection({
host : 'us-cdbr-east-04.cleardb.com',
user : 'b32fa719432e40',
password : '87de815a',
database : 'heroku_28437b49d76cc53'
});
Use this details and connect it in mySQL work bench, and import your localhost db
The base code was ok, I missed some NodeJS code.
I did a video explaining how to connect to MySqlusing NodeJS on a Heroku server, take a look:
http://www.youtube.com/watch?v=2OGHdii_42s
This is the code in case you want to see:
https://github.com/mescalito/MySql-NodeJS-Heroku
createConnections accepts config as well as connectionString.
export function createConnection(connectionUri: string | ConnectionConfig): Connection;
So below solution would work.
var connection = mysql.createConnection('mysql://b32fa719432e40:87de815a#us-cdbr-east-04.cleardb.com/heroku_28437b49d76cc53?reconnect=true);
or you can set the connection url in environment variable DATABASE_URL
var connection = mysql.createConnection(process.env.DATABASE_URL);
One should need to use pool connection for database connection to handle better mysql conncurrent request as follows
var pool = mysql.createPool({
connectionLimit : 100,
host : 'us-cdbr-iron-east-05.cleardb.net',
user : 'b5837b0f1d3d06',
password : '9d9ae3d5',
database : 'heroku_db89e2842543609',
debug : 'false'
});
It just because pool maintain the connection.release() on its own , so you do not have to bother where you should need to release the connection.
On the other hand you should need to provide user, password and database name as i mentioned in my code.
When u get provisioned from heroku then you get a cleardb database name.
on clicking clear db database button you will find username and password.
and to get database url you have to run the following commands in terminal as follows;
heroku login
heroku app -all (it shows the app list name )
heroku config --app appname (it will provide the database url).
Rest you would need to worry, just add all dependency into your package.json before pushing to heroku master.
After then you will have no problem on deploying nodejs application to heroku server.
Looking at the log timestamps, it seems like your connections are timing out. Either create a wrapper for 'getConnection' that checks the connection health and re-establishes if necessary, or try to use the mysql Connection Pool feature, which can do that for you.