ETIMEDOUT connecting to MySql from Lambda - mysql

I am trying to connect to a MySql RDS instance from a Lambda function and getting an ETIMEDOUT error
The Lambda is not part of a VPC
The RDS instance is available publicly, I can connect to it from my laptop using MySqlWorkbench
The RDS instance's security group has inbound rules configured for all ports and 0.0.0.0/0
The Lambda's execution role has many policies (probably too many) including RDSFullAccess, LambdaVPCAccessExecutionRole, ec2:*, even AdministratorAccess!)
Again, the code executes locally, connects to and queries the RDS instance just fine. Executing the same code in Lambda throws the ETIMEDOUT error
Similar posts are resolved by adding the Lambda to the RDS instance's VPC, or by configuring the inbound rules on the database's security group. Nothing seems to work.
Since I can connect to the database from my laptop just fine, my hunch is that it's a problem with the Lambda
Are there additional policies I should attach to the Lambda's execution role?
Is there any other reason the Lambda would time out connecting to a publicly available database?
Additional info:
The Lambda is not running in a VPC. It runs on Node and connects to MySql using the mysql package v2.18.1 and is deployed using Serverless with the following config:
foo:
handler: functions/handlers.foo
timeout: 20
events:
- http:
path: /path/{pathParameter}/foo
method: get
cors: true
caching:
enabled: true
ttlInSeconds: 3600
cacheKeyParameters:
- name: request.path.pathParameter
In the Lambda I try connecting with this function (which, again, works fine when I execute the function on my laptop):
function openDbConnection() {
let connection = mysql.createConnection({
host: 'db-name.cgwxrjuo6oyd.us-east-1.rds.amazonaws.com',
user: process.env.DB_USER,
password: process.env.DB_PASS,
database: 'db-name'
});
try {
connection.connect(function(err) {
console.log("Database is ", connection.state)
if (err) {
return console.error('error: ' + err.message);
}
console.log('Connected to the MySQL server.');
});
} catch (error) {
console.log("Database is ", connection.state)
console.log("Error connecting to MySql: ", error);
}
return connection;
}
The database username and password are retrieved from environment variables that are published to the Lambda with Serverless using a .env.yml file:
provider:
name: aws
runtime: nodejs12.x
lambdaHashingVersion: '20201221'
environment: ${file(.env.yml):}
Below are the Cloudwatch logs for a single execution, which I'm having trouble making sense of. Entries appear out of sequence:
| timestamp | message |
|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1626209636788 | START RequestId: e3e3ceb7-bb55-4c3e-8392-38e08401f679 Version: $LATEST |
| 1626209636791 | 2021-07-13T20:53:56.790Z adfc1cd5-e4be-40b7-970c-d38acabeb199 INFO Database is disconnected |
| 1626209636791 | 2021-07-13T20:53:56.791Z adfc1cd5-e4be-40b7-970c-d38acabeb199 ERROR error: connect ETIMEDOUT |
| 1626209636791 | 2021-07-13T20:53:56.791Z adfc1cd5-e4be-40b7-970c-d38acabeb199 INFO An error occured querying MySql: connect ETIMEDOUT |
| 1626209636792 | 2021-07-13T20:53:56.792Z adfc1cd5-e4be-40b7-970c-d38acabeb199 INFO Database is disconnected |
| 1626209636792 | 2021-07-13T20:53:56.792Z adfc1cd5-e4be-40b7-970c-d38acabeb199 ERROR error: Connection lost: The server closed the connection. |
| 1626209636793 | 2021-07-13T20:53:56.792Z adfc1cd5-e4be-40b7-970c-d38acabeb199 INFO An error occured querying MySql: Connection lost: The server closed the connection. | 1626209636803 | END RequestId: e3e3ceb7-bb55-4c3e-8392-38e08401f679 |
| 1626209636803 | REPORT RequestId: e3e3ceb7-bb55-4c3e-8392-38e08401f679 Duration: 9.32 ms Billed Duration: 10 ms Memory Size: 1024 MB Max Memory Used: 79 MB |
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Related

Serverless, connection to AWS RDS

i'm attempting to connect to an AWS RDS database, however I can't seem to get my function to even attempt to connect.
This is my function:
export const connectTest = async (event, context) => {
let mysql = require('mysql');
let connection = mysql.createConnection({
host : process.env.RDS_HOSTNAME,
user : process.env.RDS_USERNAME,
password : process.env.RDS_PASSWORD,
port : process.env.RDS_PORT
});
let result = await connect(connection);
connection.end();
return {
statusCode: 200,
body: JSON.stringify({
message: result
})
};
};
const connect = () => new Promise((resolve, reject) => connection => {
connection.connect(function(err) {
if (err) {
reject(`Error ${err.message}`);
}
resolve('Connected');
});
});
And this is the response I get after running the following command:
Command: serverless invoke local --function connectTest
λ serverless invoke local --function connectTest
Serverless: DOTENV: Loading environment variables from .env:
Serverless: - RDS_HOSTNAME
Serverless: - RDS_USERNAME
Serverless: - RDS_PASSWORD
Serverless: - RDS_PORT
Serverless: Bundling with Webpack...
As you can see the script just ends, I don't get either of the messages.
If you want to conenct to an RDS instance from a Lambda function (I assume this is what you mean by Serverless), then check the details here: https://docs.aws.amazon.com/lambda/latest/dg/services-rds-tutorial.html
To connect to RDS from code or a tool running on your development machine (it does matter if the code is Java, JavaScript, Python, etc), you need to perform these tasks:
Ensure that the RDS instance is configured to allow Public Access.
Set the inbound rules to let the IP address of your development machine connect.
For information about setting up security group inbound rules, see Controlling Access with Security Groups.

Can't connect to RDS via AWS.RDS.Signer

I'm trying to connect to my MySQL RDS from a Lambda via AWS.RDS.Signer with the following code and fake credentials:
1 const DB_REGION = 'ap-southeast-2a'
2 const DB_HOST = 'dbinstance.ddtev8utygt.ap-southeast-2.rds.amazonaws.com'
3 const DB_PORT = 3306
4 const DB_USER = 'anyuser'
5 const DB_NAME = 'anydb'
6
7 const signerOptions = {
8 region: DB_REGION,
9 hostname: DB_HOST,
10 port: DB_PORT,
11 username: DB_USER
12 }
13
14 const signer = new AWS.RDS.Signer(signerOptions)
15 const token = await signer.getAuthToken()
16
17 const config = {
18 host: DB_HOST,
19 user: DB_USER,
20 password: token, // "Password123"
21 database: DB_NAME,
22 ssl: 'Amazon RDS',
23 authPlugins: {
24 mysql_clear_password: () => () => token
25 }
26 }
but I always get this error
"Access denied for user 'anyuser'#'172.14.1.12' (using password: NO)"
I'm not entirely sure if that is needed for the AWS.RDS.Signer but I selected this option of my database:
Password and IAM database authentication
Authenticates using the database password and user credentials through AWS IAM users and roles.
NOTE: If I swap the password from token to "Password123" on line 20 I can successfully connect to my RDS.
Am I missing something here or does AWS.RDS.Signer only work with RDS Proxy?
By the way: the getAuthToken function gives me something like that (token truncated)
"dbinstance.ddtev8utygt.apsoutheast2.rds.amazonaws.com:3306/Action=connect&DBUser=anyuser&&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAU7VGXF6UCWYZCFEG%2F20210318%2Fap-southeast-2a%2Frds-db%2Faws4_request&X-Amz-Date=20210318T105145Z&X-Amz-Expires=900&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEAsaDmFwLXNvdXRoZWFzdC0yIkgwRgIhAKg8ibwNJ4E3hSOuq7HtDFvqHxmTlpOUk3I6EH2%2B9VdOV3RQ%2F03xiVdvjhEBkHqEXHQ%3D&X-Amz-Signature=749d931f74873e6c2c0d4fec94f0743f42efd5aa95ca0ac0f05c4bef30e3bd4d&X-Amz-SignedHeaders=host"
I finally can connect to my RDS via Lamdba using IAM (aka AWS.RDS.Signer)
So what was the problem?
Short story:
I used the wrong region in my Policy and in the props for the AWS.RDS.Signer.
I assumed that the Availability zone (ap-southeast-2a) equals the region, but that's not true. The correct region is actually described when the AWS command line creds get created. To find out the region invoke cat ~/.aws/config from the terminal. My default region was actually region=ap-southeast-2.
Long story:
When I started with AWS.RDS.Signer I followed the instructions here
When it came to extracting the DB info for the Policy, I used this command
aws rds describe-db-instances --db-instance-identifier <MY INSTANCE NAME> --query "DBInstances[*].DbiResourceId" --region ap-southeast-2a
but I got this error Could not connect to the endpoint URL: "https://rds.ap-southeast-2a.amazonaws.com/"
After a bit of Googling, I realised that the region is not right. That gave me a hint to change the region in the policy and in the code of my Lambda function.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rds-db:connect"
],
"Resource": [
"arn:aws:rds-db:<my-region>:<my-account-id>:dbuser:<my-db-resource-id>/<my-db-username>"
]
}
]
}
Then I created an extra user (ssluser) in the DB to connect with the AWS.RDS.Signer token
CREATE USER 'ssluser' IDENTIFIED WITH AWSAuthenticationPlugin as 'RDS';
GRANT ALL PRIVILEGES ON <MY DB NAME>.* TO 'ssluser'#'%';
GRANT USAGE ON <MY DB NAME>.* TO 'ssluser'#'%' REQUIRE SSL;
The next step was to add the above policy to the EC2 instance, ssh into the instance, install the MySQL client yum install mysql and try to connect to the instance using the token
mysql --host=dbinstance.chteb5kjtggo.ap-southeast-2.rds.amazonaws.com --port=3306 --ssl-ca=rds-combined-ca-bundle.pem --enable-cleartext-plugin --user=ssluser --password=`aws rds generate-db-auth-token --hostname dbinstance.chteb5kjtggo.ap-southeast-2.rds.amazonaws.com --port 3306 --region ap-southeast-2 --username ssluser`
After successfully connect to the RDS without providing a password I only had to attach the policy to my Lambda and change the username and region in my Lambda code to
1 const DB_REGION = 'ap-southeast-2'
2 const DB_HOST = 'dbinstance.ddtev8utygt.ap-southeast-2.rds.amazonaws.com'
3 const DB_PORT = 3306
4 const DB_USER = 'ssluser'
5 const DB_NAME = 'anydb'
I hope that will help someone in the future.

Error: Cannot enqueue Query after fatal error. Expressjs. Mysql

I have my Express js connect to multiple dbs. Which works everytime I startup my app. But As soon as my connection to my database goes stale... the connection returns an error code of PROTOCOL_CONNECTION_LOST. Which is normal for a mysql when a connection is idle. My mysql server is deployed in AWS RDS which also works just fine.
The problem is, everytime my express app encounters the PROTOCOL_CONNECTION_LOST error, it should reconnect to the database, which in fact also works. BUUT when I try to make queries to my MYSQL db. It returns a Error: Cannot enqueue Query after fatal error. error. I've been dealing with this for a while, and my workaround is to restart the express app everytime. hope someone else has encountered this and could give an advice.
Here is my sample code for connecting to db:
var mysql = require('mysql');
var mysqlConn
// mysqlConn.connect();
function handleDisconnect() {
mysqlConn = mysql.createConnection({
host: 'aws_instance***',
user: '******',
password: '*****',
database: 'my_db',
multipleStatements: true
});
mysqlConn.connect(function (err) {
if (err) {
console.log('ERROR CONNECT admin:', err.code + '--' + err.address);
setTimeout(handleDisconnect, 2000);
} else {
console.log('Connected to DB')
}
});
mysqlConn.on('error', function (err) {
console.log('ERROR admin', err.code + '--' + err.address);
if (err.code === 'PROTOCOL_CONNECTION_LOST') { // Connection to the MySQL server is usually
console.log("Connection to db lost!")
handleDisconnect(); // lost due to either server restart, or a
} else {
console.log(err) // connnection idle timeout (the wait_timeout
throw err; // server variable configures this)
}
});
}
handleDisconnect();
module.exports = {
mysqlConn: mysqlConn,
};
Then here is my output logs as shown in my server logs.
ERROR db PROTOCOL_CONNECTION_LOST--undefined
Connection to db lost!
Connected to db
OPTIONS /verify-token/ 200 0.285 ms - 4
Error: Cannot enqueue Query after fatal error.
POST /verify-token/ 500 1.332 ms - 95
OPTIONS /auth/login 200 0.793 ms - 4
Error: Cannot enqueue Query after fatal error.
POST /login 500 1.564 ms - 58
OPTIONS /login 200 0.687 ms - 4
Error: Cannot enqueue Query after fatal error.
POST /login 500 1.467 ms - 58
While there are workarounds, they apparently don't work for everyone. The suggestion in the documentation is to use connection pooling instead of manually managing individual connections.

How can I interact with a RDS MySQL instance with Node.js Lambda function?

I am attempting to access a MySQL database hosted on amazon RDS through amazon Lambda. I have .js files that I can run through cmd line on windows, but when I transfer to Lambda, I cannot connect to the database. I researched this issue thoroughly, and even after following this guide: Redstapler AWS, I receive a "process exited before completing request" error message.
My Code, copied from the above tutorial
var mysql = require('mysql');
var pool = mysql.createPool({
host: "",
user: "",
password: "",
database: ""
});
exports.handler = (event,context,callback) => {
context.callbackWaitFOrEmptyEventLoop = false;
pool.getConnection(function(err,connection){
if (err) throw err;
connection.query("SELECT * FROM testdata limit 10",
function(error,result,fields){
connection.release();
if (error) callback(error)
else callback(null,result)
});
});
};
Error Message received from Amazon Lambda
Response:
{
"errorMessage": "RequestId: b5151db1-6db8-11e8-8004-1b9e8072561c Process exited before completing request"
}
Request ID:
"b5151db1-6db8-11e8-8004-1b9e8072561c"
Function Logs:
START RequestId: b5151db1-6db8-11e8-8004-1b9e8072561c Version: $LATEST
2018-06-11T20:48:01.478Z b5151db1-6db8-11e8-8004-1b9e8072561c Error: Handshake inactivity timeout
at Handshake. (/var/task/node_modules/mysql/lib/protocol/Protocol.js:164:17)
at emitNone (events.js:86:13)
at Handshake.emit (events.js:185:7)
at Handshake._onTimeout (/var/task/node_modules/mysql/lib/protocol/sequences/Sequence.js:129:8)
at ontimeout (timers.js:386:14)
at tryOnTimeout (timers.js:250:5)
at Timer.listOnTimeout (timers.js:214:5)
--------------------
at Protocol._enqueue (/var/task/node_modules/mysql/lib/protocol/Protocol.js:145:48)
at Protocol.handshake (/var/task/node_modules/mysql/lib/protocol/Protocol.js:52:23)
at PoolConnection.connect (/var/task/node_modules/mysql/lib/Connection.js:130:18)
at Pool.getConnection (/var/task/node_modules/mysql/lib/Pool.js:48:16)
at exports.handler (/var/task/main.js:11:6)
END RequestId: b5151db1-6db8-11e8-8004-1b9e8072561c
REPORT RequestId: b5151db1-6db8-11e8-8004-1b9e8072561c Duration: 10074.63 ms Billed Duration: 10100 ms Memory Size: 1280 MB Max Memory Used: 27 MB
RequestId: b5151db1-6db8-11e8-8004-1b9e8072561c Process exited before completing request
This tutorial is highly rated and seems reputable, but I am unable to replicate its success. The error seems to imply that the connection.release is located in the wrong spot, or that I need to have another way of ending the connection and returning it to the pool.
Check to see that RDS instance security groups allow access from lambda IP address range (https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html) or place the lambda in VPC from which RDS instance is accessible.By default lambda is not in VPC...
Similar question:
updated : AWS Lambda is not able to connect to MySQL
Put console log statements to know if you are able to connect to RDS or not.
...
connection.release();
console.log(result);
if (error) callback(error)
...
Make sure you call pool.end() before calling callback function else lambda function execution won't be exited properly so you will get the timeout error.

initiating sequelizer mysql connection via tunnel-ssh module

I'd like to connect to a MySQL database using Sequelizer. Right now, I'm getting a Connection Refused Error.
To access the database, I have to SSH in. According to Mick Hansen here: https://github.com/sequelize/sequelize/issues/3753, one way to SSH in is to use tunnel-ssh to establish the tunnel, then initiate Sequelizer.
My (unsuccessful) approach so far has been to initiate the tunnel, then when the tunnel opens, test whether Sequelizer has authenticated.
Update
Host: DigitalOcean
CLI Success: I can 1) ssh into digitalocean server 2) login into mysql from the server and 3) access all database information as the root user.
Sequel Pro: I can also log into the database using Sequel Pro.
MySQL 127.0.0.1:3306: Based on the mysql/my.cnf file, the port is 3306 and the bind-address is 127.0.0.1. The config file also says instead of skip-networking, the default is to listen only on localhost, if that's relevant.
socketPath -> Error Connection Switching from TCP to socket seems to sometimes work for this type of problem, but when I tried it, I continued to get a connection refused error.
2 Error Types - "All Configured Authentication Methods Failed" and "Error Connection Refused"
Thanks for the help!
Code:
// sequelize config
var sequelize = new Sequelize('database', 'user', 'pass', {
host: '127.0.0.1',
dialect: 'mysql',
port: 3306,
pool: {
max: 10,
min: 0,
idle: 20000
}
});
// tunnel config
var config = {
user:'user',
host:'sshHost',
port:22,
dstHost:'127.0.0.1',
dstPort:3306,
srcHost:'127.0.0.1',
srcPort:3306,
localHost:'127.0.0.1',
localPort: 3306,
privateKey:require('fs').readFileSync('/path/to/key')
};
var tunnel = require('tunnel-ssh');
// initiate tunnel
tunnel(config, function (error, server) {
//....
if(error) {
console.error(error);
} else {
console.log('server:', server);
// test sequelize connection
sequelize
.authenticate()
.then(function(err) {
console.log('Connection established');
})
.catch(function(err) {
console.error('unable to establish connection', err);
})
}
})
When my config is set to the object above, I get an "All configuration methods failed error".
If I change my config to the below, I get a "Sequelize Error Connection Refused" error.
// tunnel config
var config = {
user:'user',
host:'sshHost',
port:22,
dstHost:'127.0.0.1',
dstPort:3306,
//srcHost:'127.0.0.1',
//srcPort:3306,
//localHost:'127.0.0.1',
//localPort: 3306,
privateKey:require('fs').readFileSync('/path/to/key')
};
localPort is the port listening on your local system. Currently you have it defined as 27000 but your Sequelize config is set to 3306.