I am investigating how to create functions, in Lambda, against MySQL. I have seen tutorials on the internet, but I can not do it. I've only been able to do the GET method.
How do I manage the variables, to make a DELETE or a PUT?
Currently, I did this function.
const mysql = require('mysql');
const con = mysql.createConnection({
host : process.env.RDS_HOSTNAME,
user : process.env.RDS_USERNAME,
password : process.env.RDS_PASSWORD,
port : process.env.RDS_PORT,
database : process.env.RDS_DATABASE
});
exports.handler = (event, context, callback) => {
// allows for using callbacks as finish/error-handlers
context.callbackWaitsForEmptyEventLoop = false;
const sql = "SELECT * FROM usuarios";
con.query(sql, function (err, result) {
if (err) throw err;
callback(null, result)
});
};
Here it's the database
My Lambda Function
And my API Gateway
I hope to create a function to delete a record. It is not a productive environment, just to learn! I appreciate, if you help me! Regards!
Related
I have a simple program that connects to an AWS MySQL DB via Lambda.
Every request from the app is basically a single query on the database. So if I have 3 users using the app, it would be impossible for more than 3 queries to be run on the database at the same time.
Yet when I look at my connection count it could be as high as 15. I've been having trouble understanding exactly how this should work.
If the code is written correctly does that mean if there's 3 users at once, I shouldn't have more than 3 connections?
I've been trying to find a basic article to explain this but obviously haven't hence I'm here.
Thank you.
EDITED TO ADD CODE
Below is a cut down version of my code
var mysql = require('mysql');
var pool =[];
exports.handler = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
pool = mysql.createPool({
host : 'host-details',
user : 'username',
password : 'password',
database : 'db'
});
return myFunc(event, context, callback, data.Username);
}
};
const myFunc = async (event, context, callback) => {
const query = 'SELECT * from db.table';
const data = await dbQueryAsyncWL(context, callback, query);
return callback(null, {data: data});
}
const dbQueryAsyncWL = async (context, callback, query) => {
return new Promise((resolve) => {
context.callbackWaitsForEmptyEventLoop = false;
pool.getConnection(function(error, connection) {
connection.query(query, function (error, results, fields) {
connection.release();
if (error) {callback(error);} else {resolve(results);}
});
});
})
}
pool = mysql.createPool({
host : 'host-details',
user : 'username',
password : 'password',
database : 'db'
});
Check the size / minimum size of the connection pool.
What is the connection pool configuration, like idleTime etc before closing a connection.
What trends are you noticing around the size of connection pool, is it stable or increase/decreasing ?
Looks like my issue was that I was creating the pool inside the exports.handler. Once I moved it outside... well a picture says it all.
https://i.ibb.co/KDvfdhp/Screen-Shot-2020-06-29-at-10-26-49-pm.png
I've created a simple mySQL database that I'm trying to access data from via an AWS Lambda function.
This is a version of the code that runs fine locally:
var mysql = require('mysql');
var config = require('./config.json');
var pool = mysql.createPool({
host : config.dbhost,
user : config.dbuser,
password : config.dbpassword,
database : config.dbname
});
pool.getConnection(function(err, connection) {
// Use the connection
connection.query("SELECT username FROM ClimbingDB.users WHERE email = 'testemail1'", function (error, results, fields) {
// And done with the connection.
connection.release();
// Handle error after the release.
if (error) throw error;
console.log(results);
process.exit();
});
});
This is that code converted to work with AWS Lambda:
var mysql = require('mysql');
var config = require('./config.json');
var pool = mysql.createPool({
host : config.dbhost,
user : config.dbuser,
password : config.dbpassword,
database : config.dbname
});
exports.handler = (event, context, callback) => {
//prevent timeout from waiting event loop
context.callbackWaitsForEmptyEventLoop = false;
pool.getConnection(function(err, connection) {
if (err) return callback(err)
// Use the connection
connection.query("SELECT username FROM ClimbingDB.users WHERE email = 'testemail1'", function (error, results, fields) {
// And done with the connection.
connection.release();
// Handle error after the release.
if (error) return callback(error);
else return callback(null,results);
});
});
};
Which times out with this error message:
{
"errorMessage": "2019-07-19T17:49:04.110Z 2f3e208c-62a6-4e90-b8ec-29398780a2a6 Task timed out after 3.00 seconds"
}
I'm not sure why it doesnt seem to be able to connect. I tried adding the function to a vpc and a security group that has access to RDB's, neither of which do anything. I'm not sure what I'm doing wrong here.
You will need:
The Amazon RDS instance in the same VPC as the AWS Lambda function
A security group on the Lambda function (Lambda-SG)
A security group on the RDS instance (DB-SG) that permits inbound connections on port 3306 from Lambda-SG
That is, DB-SG should specifically reference Lambda-SG (it will turn into a security group ID in the format sg-1234).
You might also want to increase the timeout of the Lambda function to give it a bit more time to run.
I am trying to learn how to connect MySQL using lambda functions in AWS. I have followed a couple of instructions online and basically ended up with this code:
var mysql = require('mysql');
var pool = mysql.createPool({
connectionLimit : 1000,
connectTimeout : 60 * 60 * 1000,
acquireTimeout : 60 * 60 * 1000,
timeout : 60 * 60 * 1000,
host: "foo-bar-123.us-east-2.rds.amazonaws.com",
user: "root",
password: "pass123",
database: "sample_db",
});
exports.handler = (event, context, callback) => {
// prevent timeout from waiting event loop
context.callbackWaitsForEmptyEventLoop = false;
pool.getConnection(function(err, connection) {
if (err) throw err;
// Use the connection
connection.query('SELECT id FROM customer limit 10;', function (error, results, fields) {
// And done with the connection.
connection.release();
// Handle error after the release.
if (error) callback(error);
else callback(null,results);
});
});
};
This is working on my local but when I zip this code and uploaded it as a lambda function, this returns
Response:
{
"errorMessage": "2018-11-13T02:16:10.339Z 0562b432-e6ea-11e8-81ef-bd64aa1af0a4 Task timed out after 30.03 seconds"
}
It times out no matter how many seconds I set it to.
I have pretty much set everything at default since I am new to all of these but I have added AmazonRDSFullAccess to the role of lambda function.
Does anyone have any idea what may be wrong or missing in my setup?
Thanks.
After doing some trial and errors, I was able to make it work and what I was missing is that I did not allow All TCP in the inbound of my RDS Security group. After that, I set it as my lambda function to No VPC, and it was able to query properly.
This link: https://dzone.com/articles/querying-rds-mysql-db-with-nodejs-lambda-function and the stack overflow link posted in there (which is this: AWS Lambda RDS connection timeout) helped me a lot in figuring out what was wrong with my code/setup.
Here is the final code that I ended up using.
const mysql = require('mysql');
const pool = mysql.createPool({
host: "foo-bar-123.us-east-2.rds.amazonaws.com",
user: "root",
password: "pass123",
database: "sample_db"
});
exports.handler = (event, context, callback) => {
//prevent timeout from waiting event loop
context.callbackWaitsForEmptyEventLoop = false;
pool.getConnection((err, connection) => {
if(err) throw err;
// Use the connection
connection.query('SELECT id FROM customer limit 10;', (error, results, fields) => {
// And done with the connection.
connection.release();
// Handle error after the release.
if (error) callback(error);
else callback(null,results);
});
});
};
Thanks!
running into some issues trying to figure out an Azure Function (node.js-based) can connect to our mysql database (also hosted on Azure). We're using mysql2 and following tutorials pretty much exactly (https://learn.microsoft.com/en-us/azure/mysql/connect-nodejs, and similar) Here's the meat of the call:
const mysql = require('mysql2');
const fs = require('fs');
module.exports = async function (context, req) {
context.log('JavaScript HTTP trigger function processed a request.');
if (req.query.fname || (req.body && req.body.fname)) {
context.log('start');
var config = {
host:process.env['mysql_host'],
user: process.env['mysql_user'],
password: process.env['mysql_password'],
port:3306,
database:'database_name',
ssl:{
ca : fs.readFileSync(__dirname + '\\certs\\cacert.pem')
},
connectTimeout:5000
};
const conn = mysql.createConnection(config);
/*context.log(conn);*/
conn.connect(function (err) {
context.log('here');
if (err) {
context.error('error connecting: ' + err.stack);
context.log("shit is broke");
throw err;
}
console.log("Connection established.");
});
context.log('mid');
conn.query('SELECT 1+1',function(error,results,fields) {
context.log('here');
context.log(error);
context.log(results);
context.log(fields);
});
Basically, running into an issue where the conn.connect(function(err)... doesn't return anything - no error message, no logs, etc. conn.query works similarly.
Everything seems set up properly, but I don't even know where to look next to resolve the issue. Has anyone come across this before or have advice on how to handle?
Thanks!!
Ben
I believe the link that Baskar shared covers debugging your function locally
As for your function, you can make some changes to improve performance.
Create the connection to the DB outside the function code otherwise it will create a new instance and connect every time. Also, you can enable pooling to reuse connections and not cross the 300 limit that the sandbox in which Azure Functions run has.
Use the Promises along with async/await
You basically can update your code to something like this
const mysql = require('mysql2/promise');
const fs = require('fs');
var config = {
host: process.env['mysql_host'],
user: process.env['mysql_user'],
password: process.env['mysql_password'],
port: 3306,
database: 'database_name',
ssl: {
ca: fs.readFileSync(__dirname + '\\certs\\cacert.pem')
},
connectTimeout: 5000,
connectionLimit: 250,
queueLimit: 0
};
const pool = mysql.createPool(config);
module.exports = async function(context, req) {
context.log('JavaScript HTTP trigger function processed a request.');
if (req.query.fname || (req.body && req.body.fname)) {
context.log('start');
const conn = await pool.getConnection();
context.log('mid');
await conn.query('SELECT 1+1', function(error, results, fields) {
context.log('here');
context.log(error);
context.log(results);
context.log(fields);
});
conn.release();
}
};
PS: I haven't test this code as such but I believe something like this should work
Debugging on serverless is challenging for obvious reasons. You can try one of the hacky solutions to debug locally (like Serverless Framework), but that won't necessarily help you if your issue is to do with a connection to a DB. You might see different behaviour locally.
Another option is to see if you can step debug using Rookout, which should let you catch the full stack at different points in the code execution and give you a good sense of what's failing and why.
How to use mysql connection with serverless framework.connection should be available in my component functions without creating mysql connection each time in component function
Tried like this
var mysql = require('mysql');
module.exports.respond = function(event, cb) {
var pool = mysql.createPool({
connectionLimit : 100,
host : 'hostname',
user : 'username',
password : 'password',
database : 'databasename',
debug : false
});
var message='';
pool.getConnection(function(err,connection){
if(err) {
message='Could not connect to database';
} else {
message="Database is connected";
}
var response = {
message: message
};
return cb(null, response);
});
};
but above code will be only available for current function,want to make common thing for mysql connection in serverless framework,can not find proper document about how to use mysql in serverless framework
I am writing answer of my own question
make database.js file in component/lib folder
code of database.js
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'hostname',
user : 'username',
password : 'password',
database : 'databasename'
});
connection.connect();
module.exports = connection;
created object like this in component/lib/index.js file
var connection = require("../lib/database.js");
Can use connection variable to write mysql query like this in component/lib/index.js
module.exports.respond = function(event, cb) {
var query="SELECT * from table_name";
connection.query(query,function(err,rows) {
})
};
I believe you have a Component created in your Serverless Framework based project that contains multiple lambda functions. And now you want to write the MySQL connection code such that this code block is available for re-use in all your lambda functions of that component.
If this is the ask, then Serverless does provide a "lib" folder inside your Component directory, which you can utilize to write common code logic to be re-used. Since you have a NodeJS-based runtime for your component, there should be an "index.js" file inside your Component folder -
your_serverless_project_directory/component_name/lib/index.js
The first thing you want to do is to add the MySQL connection code logic to a function/method in index.js.
Serverless should have already included for you this entire lib/ folder in all your lambda function's handler.js code like this -
var lib = require('../../lib');
Therefore, the next/final thing you want to do is re-use your connection function/method (in all the lambda functions belonging inside your Component) like this -
module.exports.handler = function(event, context) {
lib.mySQLConnection();
};
Hope this helps, let me know how it goes.
To build off of Normal Goswami's answer:
You've specified the database here in the connection. My lambdas each need different databases, so in the connection code just leave off the database:
var mysql = require('mysql');
var connection = mysql.createConnection({
host : 'hostname',
user : 'username',
password : 'password'
// no database here
});
connection.connect();
module.exports = connection;
And then use an oddly named function to change the database in each lambda function:
connection.changeUser({database: database}, function(err) {
if (err) { throw err; }
});
connection.query(sql, function(err, rows, fields) {
// etc
}
You could also look into using a database connection pool.
you have to make connection out of function, as we are doing it with mongodb
we are making mongodb connection out side of Lambda Function.
my code snippet from https://github.com/malikasinger1/serverles-practice/tree/master/mongodb-connection:
var mongoose = require("mongoose");
var dbURI = 'mongodb://localhost/mydatabase';
mongoose.connect(dbURI);
mongoose.connection.on('connected', function () {//connected
console.log("Mongoose is connected");
// process.exit(1);
});
module.exports.signup = (event, context, cb) => {
//doing signup here
}
in your cace it will be something like this most probably:
var mysql = require('mysql');
//make connection here
var pool = mysql.createPool({
...
});
pool.getConnection(function(err,connection){
...
});
module.exports.respond = function(event, cb) {
//use connection here
};
I am assuming you are using serverless framework on AWS.
Although you can create a connection and assign it to a frozen variable, it's not guaranteed that your lambda won't create a new connection. Here is why:
The best way so far (in my personal opinion) is to create a separate lambda function for db related operations and invoke this function through other lambdas. Here is the flow:
client -> registerUserLambda -> dbLambda -> DATABASE
However, the thing about lambdas is that when there are too many requests, there will be new containers created to handle other requests. That is, new connections will be created. Therefore the concept of connection pools does not work well for now on serverless lambdas.