Lambda + Sequelize randomly getting SequelizeConnectionError ETIMEDOUT - mysql

Question

We are using Sequelize within Aws Lambda and for the most part everything is working great however randomly it is erroring out with the following error:
ETIMEDOUT {"name":"SequelizeConnectionError","parent":{"errorno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","fatal":true},"original":{"errorno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","fatal":true}}
We are using Rds - mysql 8.0.15, Serverless framework, serverless-http, serverless-webpack.
Here is our file configuration.
//db.js
... import all models
const sequelize = new Sequelize(
process.env.DATABASE,
process.env.DB_USER,
process.env.DB_PASSWORD,
{
host: process.env.DB_HOST,
port: process.env.STAGE === "dev" ? 3306 : 31304,
dialect: "mysql",
dialectOptions: { decimalNumbers: true },
pool: {
max: 10,
min: 0
}
}
);
const models = {};
// Initialize models
modules.forEach(module => ...
export default models;
//handler.js
import express from "express";
import serverless from "serverless-http";
import db from "./db";
const app = express();
app.use(async (req, res, next) => {
try {
const email = "get email from jwt ...";
req.user = await db.user.findOne({
where: { email }
});
return next();
} catch (e) {
logger.warn("An error occurred" , e);
res.status(500).send({ message: e.message });
}
});
app.use("/api", api);
app.get("*", (req, res) =>
res.status(404).json({ errorCode: 0, message: "Unrecognized route" })
);
const handler = serverless(app);
module.exports.handler = async (event, context) => {
context.callbackWaitsForEmptyEventLoop = false;
return handler(event, context);
};
I thought potentially that we reached the max mysql connections (which for my instance is 66) however the rds dashboard shows the most we have is in the 40's.
What are we doing wrong?

Answer 1

Although you say you are not reaching the maximum connections, you still might want to try creating an Amazon RDS Proxy for your Lambda function to access, hit it with a high load, and see if you are able to reproduce the error.
You don't really have enough logs to diagnose the issue, if the above does not work you will need to dive deeper, potentially enable more RDS logging to see if that tells you the problem.
Other ways you can troubleshoot the issue is if you right the same queries you are executing in another language/framework, simulate and see if problems persists.
You may also want to check Cloudwatch metrics for any other tells which could give you a clue as to what the problem is. Graph your Lambda resource metrics and RDS instance metrics on the same chart to see if there are any patterns with when the Lambda function errors and what your DB is doing, such as if the error occurs if your write or read latency increases.
If the issue persists, and you are not able to solve it, the best you can probably do is implementing retries, which will simply mask the issue, but if the boss is at you for a solution, this might be your best bet.
Hope my suggestions help, I've had similar issues with DB+Lambda & DB+ECS and found those to be effective troubleshooting strategies.