Node.js MySQL ERR_HTTP_HEADERS_SENT - mysql

This is the function in question:
castVote: function (req, res, ip, mysql) {
// POST return codes
const ALREADY_VOTED = '100';
const SUCCESS = '200';
const FAILURE = '300';
// Create connection to db
// Keep connection while the client is connected
var con = mysql.createConnection({
host: sqlHOST,
user: sqlUSER,
password: sqlPASS,
database: dbNAME
});
// Connect to db
con.connect(function (err) {
// MySQL error, return
if (err) {
res.send(FAILURE);
return;
}
});
// Get link voted for
var link = req.body.song_voted;
// Check if user has already voted
con.query(`SELECT * FROM VotedUsers WHERE ip = '${ip}'`, function (err, result, fields) {
if (err) {
res.send(FAILURE);
return;
}
// User hasn't voted
if (!result.length) {
// Cast vote here
// ...
// Add user's IP to voted list
con.query(`INSERT INTO VotedUsers (ip) VALUES ('${ip}')`, function (err, result) {
if (err) {
res.send(FAILURE);
return;
}
});
res.send(SUCCESS);
return;
}
// User already voted
else {
res.send(ALREADY_VOTED);
return;
}
});
}
I call the function like this, every time the user clicks a button
mysql_backend.castVote(req, res, uipv4, mysql);
(mysql is the mysql module. uipv4 is the user's ip).
Every time I click said button, This is the error output:
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
I've got 2 questions:
What's wrong with my code?
Is there a better way of managing a mysql connection for each client (not opening a new one every time the user clicks the button)

1. What is wrong with your code
I can see a couple of things.
Firstly, the most obvious problem is your con.connect and con.query functions.
You have them implemented incorrectly. con.connect takes a callback that all your implementation should be in i.e. all your query logic, once you have created a connection.
Anyway the con.connect function is not required. You can remove it as con.query will create a connection for you.
Secondly, and this is causing the error that you are seeing. You need to check your application to ensure that the res.send (or equivalent) is only being invoked once per request.
So for example, this is going to throw the error you are seeing.
// User hasn't voted
if (!result.length) {
// Cast vote here
// ...
// Add user's IP to voted list
con.query(`INSERT INTO VotedUsers (ip) VALUES ('${ip}')`, function (err, result) {
if (err) {
res.send(FAILURE);
return;
}
});
res.send(SUCCESS);
return;
}
If the insert fails, res.send(SUCCESS) has already being executed, then the con.query callback is executed and the res.send(FAILURE) will be called.
This is just one example. You will need to check your code through the entire request to ensure that a response is only ever sent once. If you send more than once you will see the ERR_HTTP_HEADERS_SENT (or a similar error - depending on the function you invoke on the response object). The key takeaway is that you should only send a response once and only once! If the below is not fixing your error you need to check the rest of your application to make sure that you are not sending a response again e.g. after you call mysql_backend.castVote(req, res, uipv4, mysql);
To fix the above problem depends on the desired result.
If you want to return to the caller that the error has occurred, you must wait until the callback has returned. e.g.
if (!result.length) {
// Cast vote here
// ...
// Add user's IP to voted list
con.query(`INSERT INTO VotedUsers (ip) VALUES ('${ip}')`, function (err, result) {
if (err) {
res.send(FAILURE);
return;
}
res.send(SUCCESS);
return;
});
}
2. Is there a better way of Managing Connections
Yes - use connection pools. Connection pooling allows you to limit the number of connections that can be created, e.g. if you have 100 parallel requests and a maximum pool size of 10, you will only ever create 10 connections.

Related

Testing an AWS Lambda function

I need to create a lambda function to act as the middleman between a mobile Java app and an AWS RDS MySQL database. The idea is to submit queries from the mobile app and then send them off to the lambda function, which will then return the query. I have a basic MySQL query set up in my AWS lambda:
var mysql = require('mysql');
var config = require('./config.json');
var pool = mysql.createPool({
host : config.dbhost,
user : config.dbuser,
password : config.dbpassword,
database : config.dbname
});
exports.handler = (event, context, callback) -> {
context.callbackWaitsForEmptyEventLoop = false;
pool.getConnection(function(err, connection) {
if (err) throw err; // not connected!
// Use the connection
connection.query('select Album from record', function (error, results, fields) {
// When done with the connection, release it.
connection.release();
// Handle error after the release.
if (error) callback(error);
else callback(null, results[0].Album);
// Don't use the connection here, it has been returned to the pool.
});
});
};
And all that I am currently trying to do is get this code to run and output what the query will return. I've seen tutorials where people seem to just click test and have the code run, but it keeps asking me to create a test, and I'm not sure what exactly I would need to do to test this function.
EDIT: I realized I was missing a small change in my lambda uploaded code, but I am now getting an error on line 10 saying there is an unexpected token >.
I'm not sure what's wrong here, as the tutorial I watched seems to have the same exact thing.
Since you're not passing in any parameters through the context, you can just create a test with the defaults or an empty object {}, and click Test in the console. It will invoke your Lambda function as if it had been called from your mobile app, and you can debug from there.

Node.JS and MySQL - queries lock up and execute extremely slowly

I am getting strange behavior using Node.JS and MySQL with this driver - https://github.com/mysqljs/mysql
Essentially, I have a button on the frontend that triggers an app.get that makes a query in the database and I can happily use the results in my backend.
This works nicely, until I press the button 4-5 times in a second, where as the queries lock up and I have to wait for 2-3 minutes until they continue executing. I have a similar write function that behaves the same way.
Is it possible this is a problem, because I'm trying to execute the exact same query asynchronously? I.e. do I have to limit this from the front end or is it a backend problem?
Any ideas on how to debug what exactly is going on?
// database.js
var mysql = require('mysql');
var pool = mysql.createPool({
connectionLimit: 100,
host : 'localhost',
user : 'secret',
password : 'secret',
database : 'mydb'
});
exports.getConnection = function(callback) {
pool.getConnection(function(err, connection) {
callback(err, connection);
});
};
// dbrw.js
var con = require('../config/database');
function read(id, done) {
con.getConnection(function(err, connection){
if(!err){
connection.query("SELECT * FROM users WHERE id = ?",[id], function(err, rows) {
connection.release();
if (err)
done(err);
if (rows.length) {
console.log("rows " + JSON.stringify(rows));
done(rows[0].progress);
};
});
}
else {
console.log(err);
}
});
}
exports.read = read;
// routes.js
var dbrw = require('./dbrw.js');
app.get('/read', isLoggedIn, function(req, res) {
dbrw.read(req.user.id, function(result) {
console.log(result);
});
});
// Frontend - angular app.js
$scope.tryread = function() {
$http.get('/read');
}
Thanks in advance for any input.
I see a few issues:
function read(id, done) {
con.getConnection(function(id, connection){...}
}
Notice how you overwrite the id passed to read by giving that same name to an argument of the callback to getConnection.
Also, your Express route doesn't actually end the request by sending back a response, which will make your browser time out the connection. At some point, it will even refuse to send more requests because too many are still pending.
So make sure to end the request:
app.get('/read', isLoggedIn, function(req, res) {
dbrw.read(req.user.id, function(result) {
console.log(result);
res.end(); // or `res.send(result)`
});
});
And a tip: you should use the callback calling convertion for Node, where the first argument represents an error (if there is any) and the second argument represents the return value.

has exceeded the 'max_user_connections' resource

I have a MySQL, Express, Angular, NodeJS application and sometimes when I log in I get the following error in my node console:
TypeError: Cannot read property 'query' of undefined
The error occurs in my passport.local.js file, this is the line:
connection.query('SELECT * FROM users WHERE username LIKE ?', [username], function (err, user) {
This is the passport function
passport.use(new LocalStrategy(
function(username, password, done) {
console.log('app.js');
pool.getConnection(function(err, connection) {
console.log('err: ' + err);
console.log(connection);
connection.query('SELECT * FROM users WHERE username LIKE ?', [username], function (err, user) {
if (err) throw err;
for (var i = user.length - 1; i >= 0; i--) {
var current = user[i];
}
if(current){
if(bcrypt.compareSync(password, current.password)){
return done(null, user);
} else {
return done(null, false);
}
} else {
console.log('no user');
return done(null, false);
}
});
connection.release();
});
}
));
I require my pool in the top of my file
var pool = require('../../config/connection');
When the error occurs the:
console.log(connection);
Gets:
undefined
I also log the error:
console.log('err: ' + err);
Shows:
err: Error: ER_USER_LIMIT_REACHED: User 'bfe4a8980ede74' has exceeded the 'max_user_connections' resource (current value: 10)
I am assuming that your max_user_connections is set to 10. Please increase the max_user_connection value.
show global variables like '%connections%';
Will help you in giving the no of connections you have set. Increase the number of connections If its less than 25 or 50. Max connections can be more than 16000 I guess, it all depends on your cpu, no of threads It can handle etc.
SET GLOBAL max_user_connections=100;
max_user_connections is a dynamic variable, which means you can directly run this query. You donot have to shutdown mysql.
The error you're getting is stating the issue: your MySQL server is only allowing 10 connection per user, and that limit has been reached.
The default for the mysql connection pool also happens to be 10, which is cutting it really close. If any other MySQL client besides your Express app is connected to the database with the same user credentials, you may run into that particular error. I would suggest increasing max_user_connections in the MySQL configuration.
Aside from that, there's another issue with your code: it's releasing the connection before the query has finished, which may lead to unexpected behaviour. Move the call to connection.release() to inside the callback:
pool.getConnection(function(err, connection) {
...
connection.query('SELECT * FROM users WHERE username LIKE ?', [username], function (err, user) {
connection.release();
...
});
});
If this is a common way you're using MySQL (get a connection, perform a query, release the connection), you can make life a bit easier by using pool.query() instead. See this example.
And finally, if you're working with async code, don't throw errors but pass them to the callback (and make sure that you actually handle them, because you're not handling any errors from pool.getConnection now, besides logging them):
pool.getConnection(function(err, connection) {
if (err) return done(err);
...
});

How to structure Node.js script with MySQL so that connection doesn't close prematurely

So I am making my first attempt with Node and I can't really wrap my head around how to work with the MySQL connection. The script is somewhat simplified like this
var mysql = require('mysql');
var connection = mysql.createConnection({
host : '192.168.40.1',
user : 'user',
password : 'password',
database : 'database'
});
function DoSomething(connection, item, callback) {
connection.query(
'SELECT COUNT(*) AS count FROM another_table WHERE field=?',
item.field,
function (err, results) {
if (err) throw err;
if (results.length > 0 && results[0].count >= 1) {
callback(err, connection, item, 'Found something')
}
});
}
function DoSomethingElse(connection, item, callback) {
// Similar to DoSomething()
}
function StoreResult(err, connection, item, reason) {
if (err) throw err;
connection.query(
'INSERT INTO result (item_id, reason) VALUES (?, ?)',
[item.id, reason],
function (err, results) {
if (err) throw err;
});
}
connection.query('SELECT * FROM table WHERE deleted=?', [0], function (err, results)
{
if (err) throw err;
results.forEach(function (item, index) {
DoSomething(connection, item, StoreResult);
DoSomethingElse(connection, item, StoreResult);
});
});
connection.end();
What I am having trouble with (as far as I can tell) is that since DoSomething() it seems that connection.end() is called before all of the DoSomething()'s have finished causing errors that queries can't be performed when the connection is closed.
I tried playing around with the async library, but I haven't gotten anywhere so far. Anyone with some good advice on how to do this?
The problem with your code is that you're closing the connection synchronously while an asynchronous request is still being handled. You should call connection.end() only after all query callbacks have been called.
Since you are doing multiple queries, this means using some way to wait for all their results. The simplest way is to nest every next call into the callback of the previous one, but that way leads to the pyramid of doom. There are a number of popular libraries that solve this, but my own preference is for async.
Using async I would rewrite your code as follows:
async.waterfall([function(next) {
connection.query('SELECT * FROM table WHERE deleted=?', [0], next); // note the callback
},
function(results, next) {
// asynchronously handle each results. If they should be in order, use forEachSeries
async.forEach(results, function(item, next) {
// handle in parallel
async.parallel([function(done) {
DoSomething(connection, item, function(err, connection, item, reason) {
// This is a hack, StoreResult should have a callback which is called
// after it's done, because the callback is now being called too soon
StoreResult(err, connection, item, reason);
callback(err);
});
}, function(done) {
DoSomethingElse(connection, item, function(err, connection, item, reason) {
// This is a hack, StoreResult should have a callback which is called
// after it's done, because the callback is now being called too soon
StoreResult(err, connection, item, reason);
callback(err);
}], function(err) {
// this callback is called with an error as soon as it occurs
// or after all callbacks are called without error
next(err);
});
}, function(err) {
// idem
next(err);
});
}], function(err, results) {
// idem
// Do whatever you want to do with the final error here
connection.end();
});
This also allows you to solve a possible issue with the order of your queries in the forEach: They are started in order, but are not guaranteed to finish in order due to their asynchronous nature.
Close your connection after you have done everything you want in the script.
When programming in asynchronous language, keep in mind that the real ending point of your script is the last asynchronous callback, instead of the last line like other scripts (e.g. PHP).
Note that you don't want to simply ignore the connection.end(); as the underlying MySQL driver will keep the connection alive and your script will stuck in the last line forever until you kill it.
This is the modified version of your code.
connection.query('SELECT * FROM table WHERE deleted=?', [0], function (err, results)
{
if (err) throw err;
results.forEach(function (item, index) {
DoSomething(connection, item, StoreResult);
DoSomethingElse(connection, item, StoreResult);
});
// End your connection here
connection.end();
});

NodeJS mysql query returns empty result

i'm new to NodeJS (duh!).
I know it executes functions asynchronous but I still cannot see what causes this phenomenon:
I am using the express and mysql modules and trying to execute an SQL query based on request parameters. It is supposed to be a simple validation API feature where the server is going to lookup a user in a database by listening on a specific URL for two request parameters (user and passwd).
The problem is that the SQL query always returns an empty object as result when I do this using the request parameters in the query.
However, if i hard code the query and run it outside the app.get(...) I get the desired result! But I need this to work on demand by request...
(I'm not intending to use GET-request later on, this example is for debugging purposes :))
What am i doing wrong here?
Here's my code:
// Server and Mysql setup here
var app = require('express').createServer(),
SERVER_PORT = 8080;
var Client = require('mysql').Client,
client = new Client(),
...
// User, password and database setup here, cropped out from this example //
// ...
function validateUser(user, passwd, callback) {
client.query('SELECT date FROM '+CUSTOMERS_TABLE+' WHERE email="'+user+'" AND passwd="'+passwd+'";',
function selectCb(err, results, fields) {
if (err) {
throw err;
}
console.log(fields);
callback(results);
});
}
app.get('/', function(req, res){
var url_parts = url.parse(req.url, true);
var query = url_parts.query;
if((typeof query[REQ_PARAM_USER] != 'undefined' && typeof query[REQ_PARAM_PASSWD] != 'undefined')
&& (query[REQ_PARAM_USER] != '' && query[REQ_PARAM_PASSWD] != '')) {
validateUser(REQ_PARAM_USER, REQ_PARAM_PASSWD, function(results) {
console.log(results);
});
}
res.end("End")
});
app.listen(SERVER_PORT);
console.log('Server running at port '+SERVER_PORT);
Oh, and by the way, console.log(fields) outputs the correct fields! But why not the results?
You are passing the wrong parameters to validateUser:
validateUser(REQ_PARAM_USER, REQ_PARAM_PASSWD, // ...
What you really want:
validateUser(query[REQ_PARAM_USER], query[REQ_PARAM_PASSWD], // ...
Edit: A few other issues with your code:
You don't have to parse the url. Express does this for you, and the query is available as req.query.
You shouldn't throw in asynchronous code. It will give unexpected results. Instead, stick to the nodejs paradigm of passing (err, results) to all callbacks, and do proper error checking where you can -- i.e., in your verifyUser, pass along the error with the callback and check for errors in your get handler. Either res.send(500) (or something) when you get an error, or pass it along to the express error handler by calling next(err).
validateUser(query[REQ_PARAM_USER], query[REQ_PARAM_PASSWD], function(err, results) {
if(err) {
console.error(err);
res.send(500);
} else {
console.log(results);
res.send(results);
}
});
Never pass query parameters directly to something like an SQL query. Instead, use parameters for your SQL query:
client.query('SELECT date FROM '+CUSTOMERS_TABLE+' WHERE email=? AND passwd=?', [user, passwd], // ...