ECONNRESET error crashing NodeJS application - mysql

I am kind of new to coding and I needed some help with one error that I'm getting.
I'm running a NodeJS application (a Discord.js bot), my bot has the function to register a user's ID to a MySQL database when the user types a certain command. Sometimes it works just fine, but other times it crashes with an ECONNRESET error:
events.js:292
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:205:27)
Emitted 'error' event on Connection instance at:
at Connection._handleProtocolError (C:\Users\Celeron\Desktop\DiscordBot\node_modules\mysql\lib\Connection.js:423:8)
at Protocol.emit (events.js:315:20)
at Protocol._delegateError (C:\Users\Celeron\Desktop\DiscordBot\node_modules\mysql\lib\protocol\Protocol.js:398:10)
at Protocol.handleNetworkError (C:\Users\Celeron\Desktop\DiscordBot\node_modules\mysql\lib\protocol\Protocol.js:371:10)
at Connection._handleNetworkError (C:\Users\Celeron\Desktop\DiscordBot\node_modules\mysql\lib\Connection.js:418:18)
at Socket.emit (events.js:315:20)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
errno: 'ECONNRESET',
code: 'ECONNRESET',
syscall: 'read',
fatal: true
}

First of all, you should catch the error, so you app can handle it properly and doesn't crash when mysql connection is closed for any odd reason. Try either with connection.on('error', ...) or with try-catch blocks.
For keeping an open connection, you should either reconnect on close. Or simply use mysql's pooling connection, which handles automatic reconnection very well, with a single code change.
PS: Pooling multiple connections is a generally good idea for async apps, like servers, but it's safe to maintain a single connection via pooling (connectionLimit : 1) just for automatic reconnection itself.
PPS: Mysql's inactivity timeout can be configured in server's my.cnf

I fixed the problem.
The crash happened because of the application inactivity on the database, and when the con. becomes inactive for a couple seconds the mysql server closes de connection making the bot to crash when the code requires some database resource.
The solution was to create a function to keep constant activity in the DB connection.
Here is the code:
con.query('SELECT 1', (err) => {
if (err) throw err
console.log('Tum tum') \\ Heartbeat noises to know if it worked xD
})
}
setInterval(keepAlive, 15000)```

Related

Node.JS Azure MySQL { [Error: Cannot enqueue Query after fatal error.] code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR', fatal: false }

I have app which connects to free Azure MySQL service and make queries in a loop. I use single connection to make those queries (I am limited to 4 connections), and process takes usually ~5 seconds. When I run app locally, it works perfectly fine, but after deployment do azure I get
{ [Error: Cannot enqueue Query after fatal error.] code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR', fatal: false }
In my logs. How can I prevent this, and why does it work correctly locally, but not on Azure?
There was an answered SO thread as the same issue as yours, you can refer to Node JS Mysql PROTOCOL ENQUEUE AFTER FATAL ERROR and try to solve the issue.

NodeJS: MySQL sometimes raises ETIMEDOUT error

I'm currently developing an application using NodeJS.
However, often the server throws this error, and I can't interact with mysql.
[Error: read ETIMEDOUT]
code: 'ETIMEDOUT',
errno: 'ETIMEDOUT',
syscall: 'read',
fatal: true }
{ [Error: Cannot enqueue Query after fatal error.] code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR', fatal: false }
Does someone have a solution to fix that?
Thanks
From your question, I assume that you can work with your database perfectly but this error happens often after a given time of that connection being up...
Supposing you are using node-mysql
Source : https://github.com/felixge/node-mysql#error-handling
Your error terminates your connection to the database :
err.fatal: Boolean, indicating if this error is terminal to the connection object.
If the error is not from a MySQL protocol operation, this property will not be defined.
Note: 'error' events are special in node. If they occur without an
attached listener, a stack trace is printed and your process is
killed.
tl;dr: This module does not want you to deal with silent failures. You
should always provide callbacks to your method calls. If you want to
ignore this advice and suppress unhandled errors, you can do this:
// I am Chuck Norris:
connection.on('error', function() {});
From this you could check connection status and perform a reconnect if needed.
You could also try to connect manually to your mysql service and change request timeout :
wait_timeout” : the amount of seconds during inactivity that MySQL will wait before it will close a connection on a non-interactive connection in seconds.
http://www.serveridol.com/2012/04/13/mysql-interactive_timeout-vs-wait_timeout/
https://support.rackspace.com/how-to/how-to-change-the-mysql-timeout-on-a-server/
Try to use mysql2 over mysql npm pakcage. it will solve your problem.
full explanation here

MySql on Azure Throwing Connection Timeouts

I have 2 Virtual Machines on Azure in the same Virtual Network.
One virtual machine runs a NodeJs process which is responsible for MySQL operations.
Other virtual machine runs a MySQL instance. I can connect to it from the other VM and from the NodeJs process fine.
Sometimes it will fail and throw an error about Connection timeout when acquiring a connection from the pool.
My connection string uses a local IP address from within the Virtual Network to access the database so it should have this much delay to exceed a 10 second timeout. When it works it's rapid, I mean really fast! But sometimes it just breaks and randomly starts working again. Anyone ever come across this?
If it's any help this is a MySQL instance based on Ubuntu Server 15.10.
Exception:
{
"error": {
"name": "Error",
"status": 500,
"message": "connect ETIMEDOUT",
"errorno": "ETIMEDOUT",
"code": "ETIMEDOUT",
"syscall": "connect",
"fatal": true,
"stack": "Error: connect ETIMEDOUT
at PoolConnection.Connection._handleConnectTimeout (projectdir/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Connection.js:375:13)
at Socket.g (events.js:180:16)
at Socket.EventEmitter.emit (events.js:92:17)
at Socket._onTimeout (net.js:327:8)
at Timer.unrefTimeout [as ontimeout] (timers.js:412:13)
--------------------
at Protocol._enqueue (projectdir/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:135:48)
at Protocol.handshake (projectdir/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:52:41)
at PoolConnection.connect (projectdir/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Connection.js:123:18)
at Pool.getConnection (projectdir/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Pool.js:45:23)
at MySQL.executeSQL (projectdir/node_modules/loopback-connector-mysql/lib/mysql.js:200:12)
at projectdir/node_modules/loopback-connector-mysql/node_modules/loopback-connector/lib/sql.js:408:10
at projectdir/node_modules/loopback-datasource-juggler/lib/observer.js:175:9
at doNotify (projectdir/node_modules/loopback-datasource-juggler/lib/observer.js:93:49)
at MySQL.ObserverMixin._notifyBaseObservers (projectdir/node_modules/loopback-datasource-juggler/lib/observer.js:116:5)
at MySQL.ObserverMixin.notifyObserversOf (projectdir/node_modules/loopback-datasource-juggler/lib/observer.js:91:8)"
}
}
Per my experience, there are 2 situations may raise your issue.
The connections reaches the max_connection number of MySQL server, and there are no available connections for a new connection client. In this situation, you may check your code, whether you release the connection after MySQL operations.
On the other hand, when you get the timeout exception, you may login on Azure manage portal, in monitor page of your VM portal, to check whether the metrics reach the bottleneck of the VM which will also occur your issue. In this situation, you can scale on your VM to enlarge your VM hardwares.
The Node.js mysql module has a few options.
One of these options is the 'connectTimeout', which defaults to 10000ms (which is roughly 10 seconds).
If nothing is done within those 10 seconds, the connection closes automatically.
What the solution could be for you problem, is using pooling connections.
With this, you create a connection pool. Everytime a query needs to be executed, it takes a connection from the pool uses it and when it expires it automatically returns to the pool, ready to be restarted, thus no more connection timeout errors.
I have a process running all the time and when it idles for a while the connections in the pool are in a sleeping state. Eventually MySQL purges those connections based on the wait_timeout setting in my.cnf. Once this happens and I try to use a connection it will fail because the module assumes the connection is still live and tries to use it only to get a timeout or connection exception.
To prevent this you can either overwrite the mysql module code to support "connection lifetime" in the pool or stop using the pool and manage your own connections.

nodejs runtime stops responding when testing under stress

I'm basically checking all the routes via request module with mocha.
https://www.npmjs.com/package/request
I'm doing a stress test, by opening two console windows side by side and running them simultaneously. Most of the time tests are successful, but then an instant comes when the tests fail without timeout error, and from postman I've this specific route that stops responding.
it happens once in around 7 times, and I'm wondering what I could do to figure this out.
Edit:
Increased to 4 console windows running tests simultaneously, they ran fine couple of times but then start to timeout.
even no console output on app.get, app.post etc. routes.
Any suggestions?
Edit
Caught some request errors based on the suggestion within tests.
Uncaught AssertionError: { [Error: connect ECONNREFUSED]
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect' } == null
The corresponding code for the above error is
request({url: endpoint + "/SignIn?emailAddress=" + emailAddress + "&password=" + password}, function (error, response, body) {
assert.equal(error, null);
Edit 2
Dig further deep with console statements and noticed the mysql connection callback was not called. Attaching a screenshot and noticing some connection limit, is it because of this? I'm using connection pools though.
logs says forcing close of threads.
Probable Answer:
This thread helped with the issue.
https://github.com/felixge/node-mysql/issues/405
I set the waitForConnections: false and then started to see the error ->
[Error: No connections available.]
so it seems to me that system was waiting for the connections but test runner didn't wait and ended up with timeout error.
It also seems there's some limit on the maximum number of connections, though I was calling release on connections after each query, not sure how this works on production systems out there? do we have a limit there?
You are running out of tcp connections. You need to make few changes in system and application level, to make it handle more load.
1. Change your connection setting to keepAlive, wherever possible.
2. On unix, you have ulimit, i.e., the maximum number of file handles that any process can hold at any instant. Remember, in unix every socket is also a file.
3. Manage your time out settings, based on the response time of your database server or another web server.
You'll have to do similar changes at each level of handling request, if you have a multi-tier architecture.

Authentication mysql to .Net

I have Mysql DB connected to .NET MVC. When i connect remote it some times works fine and some times i have this error msg:
MySql.Data.MySqlClient.MySqlException (0x80004005): Authentication to
host 'ServerName' for user 'UserName' using method
'mysql_native_password' failed with message: Reading from the stream
has failed. ---> MySql.Data.MySqlClient.MySqlException (0x80004005):
Reading from the stream has failed. ---> System.IO.IOException: Unable
to read data from the transport connection: An established connection
was aborted by the software in your host machine. --->
System.Net.Sockets.SocketException: An established connection was
aborted by the
Please how can i make Authentication ? Any ideas to solve it?
This is a common error when some kind of deadlock occurs on the database more commonly a connection has been open a long time and gets shut down at the database end even though it is still in-use in your web application.
In my code, I used a suggestion from Microsoft, that retries the database command if it fails. The link is here: http://social.technet.microsoft.com/wiki/contents/articles/4235.retry-logic-for-transient-failures-in-windows-azure-sql-database.aspx Similar things exist for other non-Azure problems that use RetryPolicy.
What this means in a simple (Azure in my case) example is:
private static RetryPolicy<SqlAzureTransientErrorDetectionStrategy> retryPolicy = new RetryPolicy<SqlAzureTransientErrorDetectionStrategy>(new Incremental(3, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(1)));
// Inside your database function
retryPolicy.ExecuteAction(() =>
{
db.ExecuteStoredProc("procRecordMetric", cmd);
});
In my case, this was caused by ssl being enabled on the mysql server.
Here is a snippet from my.cnf
#ssl-ca = /etc/ssl/mysql/ca-cert.pem
#ssl-cert = /etc/ssl/mysql/server-cert.pem
#ssl-key = /etc/ssl/mysql/server-key.pem
Commenting these 3 lines fixed the issue for me.