MySQL giving "read ECONNRESET" error after idle time on node.js server - mysql

I'm running a Node server connecting to MySQL via the node-mysql module. Connecting to and querying MySQL works great initially without any errors, however, the first query after leaving the Node server idle for a couple hours results in an error. The error is the familiar read ECONNRESET, coming from the depths of the node-mysql module.
A stack trace (note that the three entries of the trace belong to my app's error reporting code):
Error
at exports.Error.utils.createClass.init (D:\home\site\wwwroot\errors.js:180:16)
at new newclass (D:\home\site\wwwroot\utils.js:68:14)
at Query._callback (D:\home\site\wwwroot\db.js:281:21)
at Query.Sequence.end (D:\home\site\wwwroot\node_modules\mysql\lib\protocol\sequences\Sequence.js:78:24)
at Protocol.handleNetworkError (D:\home\site\wwwroot\node_modules\mysql\lib\protocol\Protocol.js:271:14)
at PoolConnection.Connection._handleNetworkError (D:\home\site\wwwroot\node_modules\mysql\lib\Connection.js:269:18)
at Socket.EventEmitter.emit (events.js:95:17)
at net.js:441:14
at process._tickCallback (node.js:415:13)
This error happens both on my cloud Node server and MySQL server as well as a local setup of both.
My questions:
Does this problem appear to be a disconnection of Node's connection to my MySQL server(s), perhaps due to a connection lifetime limitation?
When using connection pools, node-mysql is supposed to gracefully handle disconnections and prune them from the pool. Is it not aware of the disconnect until I make a query, thus making the error unavoidable?
Considering that I see the "read ECONNRESET" error a lot in other StackOverflow posts, should I be looking elsewhere from MySQL to diagnose the problem?
Update: After more browsing, I think my issue is a duplicate of this one. It appears his connection is disconnecting as well, but no one has suggested how to keep the connection alive or how to address the error outside of failing on the first query back.

I reached out to the node-mysql folks on their Github page and got some firm answers.
MySQL does indeed prune idle connections. There's a MySQL variable "wait_timeout" that sets the number of second before timeout and the default is 8 hours. We can set the default to be much larger than that. Use show variables like 'wait_timeout'; to view your timeout setting and set wait_timeout=28800; to change it.
According to this issue, node-mysql doesn't prune pool connections after these sorts of disconnections. The module developers recommended using a heartbeat to keep the connection alive such as calling SELECT 1; on an interval. They also recommended using the node-pool module and its idleTimeoutMillis option to automatically prune idle connections.

If this happens when establishing a single reused connection, it can be avoided by establishing a connection pool instead.
For example, if you're doing something like this...
var db = require('mysql')
.createConnection({...})
.connect(function(err){});
do this instead...
var db = require('mysql')
.createPool({...});

Does this problem appear to be a disconnection of Node's connection to my MySQL server(s), perhaps due to a connection lifetime limitation?
Yes. The server has closed its end of the connection.
When using connection pools, node-mysql is supposed to gracefully handle disconnections and prune them from the pool. Is it not aware of the disconnect until I make a query, thus making the error unavoidable?
Correct, but it should handle the error internally, not pass it back to you. This appears to be a bug in node-mysql. Report it.
Considering that I see the "read ECONNRESET" error a lot in other StackOverflow posts, should I be looking elsewhere from MySQL to diagnose the problem?
It is either a bug in the node-MySQL connection pool implementation, o else you haven't configured it properly to detect failures.

I have been also facing the same issue. Apparently it was happening because one of the backend process has been triggered on table which was being referred in my api.
This caused table to go in lock wait state and my query request got failed with connection reset. Though i'm wondering why i didn't receive lock wait error .

Related

Azure database for MySQL DB 5.7 Transient handling in .net core

I am creating .net core 2.1 MVC application and using Azure database for MySQL DB 5.7.
I have read below links but seems they are applicable for MS SQL DB.
https://learn.microsoft.com/en-us/azure/mysql/concepts-high-availability
https://learn.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific
Transient handling for MySQL not possible? Help me link to MYSQL related similar pages.
A transient error, also known as a transient fault, is an error that will resolve itself. Most typically these errors manifest as a connection to the database server being dropped. Also new connections to a server can't be opened. Transient errors can occur for example when hardware or network failure happens.
Transient errors should be handled using retry logic. Situations that must be considered:
An error occurs when you try to open a connection
An idle connection is dropped on the server side. When you try to issue a command it can't be executed
An active connection that currently is executing a command is dropped.
The first and second case are fairly straight forward to handle. Try to open the connection again. When you succeed, the transient error has been mitigated by the system. You can use your Azure Database for MySQL again. We recommend having waits before retrying the connection. Back off if the initial retries fail. This way the system can use all resources available to overcome the error situation. A good pattern to follow is:
Wait for 5 seconds before your first retry.
For each following retry, the increase the wait exponentially, up to 60 seconds.
Set a max number of retries at which point your application considers the operation failed.
Read more here.
And you can read more on how to troubleshoot connection issues to Troubleshoot connection issues to Azure Database for MySQL here.

NodeJS Server with mysql hangs

I am writing an application using NodeJS, Express, mysql, so far everything works fine, but when I run my application after sometime when mysql connection is interrupted my application throughs this exception and my application goes down.
Error: read ECONNRESET
at errnoException (net.js:901:11)
at TCP.onread (net.js:556:19)
From another stackquestion i came to know that i have to handle such uncaught exceptions like this.
process.on('uncaughtException', function(err) {
console.log('Caught exception: ' + err);
console.log(err.stack);
});
after this now my application does not exit, but instead it hangs up, so my question is how do I handle this exception so that mysql connection is ok even after this exception and my application does not hang up.
I'm not sure if you're using the node-mysql module for your project, but I was, and I encountered the same ECONNRESET issue. Here's a repeat of my answer on my issue:
I reached out to the node-mysql folks on their Github page and got some firm answers.
MySQL does indeed prune idle connections. There's a MySQL variable "wait_timeout" that sets the number of second before timeout and the default is 8 hours. We can set the default to be much larger than that. Use show variables like 'wait_timeout'; to view your timeout setting and set wait_timeout=28800; to change it.
According to this issue, node-mysql doesn't prune pool connections after these sorts of disconnections. The module developers recommended using a heartbeat to keep the connection alive such as calling SELECT 1; on an interval. They also recommended using the node-pool module and its idleTimeoutMillis option to automatically prune idle connections.
I found the solution and I am posting if someone else is facing the same problem this might help you guys as well.
First I caught all uncaught exceptions, which made my application not to exit abnormally.
Second the problem of hanging I had was because when server would close the connection, all my requested queries would fail and my server would simply hang up I guess its node-mysql bug, but I solved it using connecting pooling, in pooling if server close the connection its re-aquired after a second or so again so my problem was solved this way.
Here is how to make most out of node-mysql pooling

Node.js and MySQL "Too many connections" error

I'm using Node.js to run a web-server for my web application. I'm also using the node-mysql module to interface with a MySQL server for all my persistent database needs.
Whenever there is a critical error within my Node.js application that crashes my app's process I get an email sent to me. So, I keep getting this email with an error saying "Too many connections". Here's an example of the error:
Error: Too many connections
at Function.Client._packetToUserObject (/apps/x/node_modules/mysql/lib/client.js:394:11)
at Client._handlePacket (/apps/x/node_modules/mysql/lib/client.js:307:43)
at Parser.EventEmitter.emit (events.js:96:17)
at Parser.write.emitPacket (/apps/x/node_modules/mysql/lib/parser.js:71:14)
at Parser.write (/apps/x/node_modules/mysql/lib/parser.js:576:7)
at Socket.EventEmitter.emit (events.js:96:17)
at TCP.onread (net.js:396:14)
As you can see all it tells me is that the error is coming from the mysql module, but it doesn't tell me where in my application code the issue is occurring.
My application opens a db connection anytime I need to run one or more queries. I immediately close the connection after all my queries and data has been collected. So, I don't understand how I could be exceeding the 151 max_connections limit.
Unless there is a place in my code where I forgot to call db.end() to close the connection, I don't see how my app would leak like this. Even if there was such a mistake, I wouldn't get these emails sent by the dozens. Yesterday, I received almost 100 emails with roughly the same error. How could this be happening? If my application had leaked and allocated connections over time, as soon as the first error occurred the app process would crash and all connections would be lost, preventing the app to crash again. Since I received ~100 emails, this means the app crashed ~100 times, and all within a short period of time. This could only mean that somewhere in my application a lot of connections where established in a short period of time, right?
How could I avoid this problem? This is very discouraging. All help is highly appreciated. Thanks
MySQL has a default MAX_CONNECTIONS = '100' not 151 unless you changed it. Also, in truth you have MAX_CONNECTIONS + 1. The plus 1 allows a root user to logon even after you have maxed out the conenctions in order to figure out what is actually being used. When your connections are maxed out try logging on as root and running the following command from MySQL.
mysql> SHOW FULL PROCESSLIST
Post the output of this command above. Once you actually know what is consuming your resources you can go about fixing it.It could easily be your code that is leaving open connections.
You should take a look at the follwoing documentation: Show Processlist
+1 for question. Investigations showed us that node-mysql opens the connections and doesn't close them. Because of that at one moment be reach the limit of max connections. The question is why node-mysql doesn't close the connections?

"MySQL server has gone away" error after few hours of inactivity

I have a Django app running on Apache with wsgi module.
After few hours of inactivity I get that error and I have to restart the Apache.
Any ideas?
Thanks
This error message mean that the database server has closed the connection to you. I guess this is caused because the connection is idle.
I believe you can get this fixed by adjusting the wait_timeout inside the configuration file of your mysql database server. The file is most commonly named "my.cnf".
This, however, is not considered as a good practice. I would like to suggest you to optimize the application you are writing to open the connection to mysql on demand - there's no point to keep it open if you are not actively using it for a long time.
If you need a quick fix, use the mysql_ping() function to check whether the connection is still alive and re-open if necessary.

A transport-level error has occurred when receiving results from the server [closed]

Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 6 years ago.
Improve this question
I'm getting a SQL Server error:
A transport-level error has occurred
when receiving results from the
server. (provider: Shared Memory
Provider, error: 0 - The handle is
invalid.)
I'm running Sql Server 2008 SP1, Windows 2008 Standard 64 bit.
It's a .Net 4.0 web application. It happens when a request is made to the server. It's intermittent. Any idea how I can resolve it?
The database connection is closed by the database server. The connection remains valid in the connection pool of your app; as a result, when you pickup the shared connection string and try to execute it's not able to reach the database. If you are developing Visual Studio, simply close the temporary web server on your task bar.
If it happens in production, resetting your application pool for your web site should recycle the connection pool.
Try the following command on the command prompt:
netsh interface tcp set global autotuning=disabled
This turns off the auto scaling abilities of the network stack
I had the same problem. I restarted Visual Studio and that fixed the problem
Transport level errors are often linked to the connection to sql server being broken ... usually network.
Timeout Expired is usually thrown when a sql query takes too long to run.
So few options can be :
Check for the connection in VPN (if used) or any other tool
Restart IIS
Restart machine
Optimize sql queries.
For those not using IIS, I had this issue when debugging with Visual Studio 2010. I ended all of the debugger processes: WebDev.WebServer40.EXE which solved the issue.
All you need is to Stop the ASP.NET Development Server and run the project again
If you are connected to your database via Microsoft SQL Server Management, close all your connections and retry.
Had this error when connected to another Azure Database, and worked for me when closed it.
Still don't know why ..
Look at the MSDN blog which details out this error:
Removing Connections
The connection pooler removes a connection from the pool after it has
been idle for a long time, or if the pooler detects that the
connection with the server has been severed.
Note that a severed connection can be detected only after attempting
to communicate with the server. If a connection is found that is no
longer connected to the server, it is marked as invalid.
Invalid connections are removed from the connection pool only when
they are closed or reclaimed.
If a connection exists to a server that has disappeared, this
connection can be drawn from the pool even if the connection pooler
has not detected the severed connection and marked it as invalid.
This is the case because the overhead of checking that the connection
is still valid would eliminate the benefits of having a pooler by
causing another round trip to the server to occur.
When this occurs, the first attempt to use the connection will detect
that the connection has been severed, and an exception is thrown.
Basically what you are seeing is that exception in the last sentence.
A connection is taken from the connection pool, the application does
not know that the physical connection is gone, an attempt to use it is
done under the assumption that the physical connection is still there.
And you get your exception.
There are a few common reasons for this.
The server has been restarted, this will close the existing connections.
In this case, have a look at the SQL Server log, usually found at:
C:\Program Files\Microsoft SQL Server\\MSSQL\LOG
If the timestamp for startup is very recent, then we can suspect that
this is what caused the error. Try to correlate this timestamp with
the time of exception.
2009-04-16 11:32:15.62 Server Logging SQL Server messages in file
‘C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\ERRORLOG’.
Someone or something has killed the SPID that is being used.
Again, take a look in the SQL Server log. If you find a kill, try to
correlate this timestamp with the time of exception.
2009-04-16 11:34:09.57 spidXX Process ID XX was killed by
hostname xxxxx, host process ID XXXX.
There is a failover (in a mirror setup for example) again, take a look in the SQL Server log.
If there is a failover, try to correlate this timestamp with the time
of exception.
2009-04-16 11:35:12.93 spidXX The mirrored database “” is changing roles from “PRINCIPAL” to “MIRROR” due to
Failover.
Was getting this, always after about 5 minutes of operation. Investigated and found that a warning from e1iexpress always occurred before the failure. This apparently is an error having to do with certain TCP/IP adapters. But changing from WiFi to hardwired didn't affect it.
So tried Plan B and restarted Visual Studio. Then it worked fine.
On closer study I noticed that, when working correctly, the message The Thread '<No Name>' has exited with code 0 occurred at almost exactly the time the run crashed in previous attempts. Some Googling reveals that that message comes up when (among other things) the server is trimming the thread pool.
Presumably there was a bogus thread in the thread pool and every time the server attempted to "trim" it it took the app down.
You get this message when your script make SQL Service stopped for some reasons. so if you start SQL Service again perhaps your problem will be resolved.
I know this may not help everyone (who knows, maybe yes), but I had the same problem and after some time, we realized that the cause was something out of the code itself.
The computer trying to reach the server, was in another network, the connection could be established but then dropped.
The way we used to fix it, was to add a static route to the computer, allowing direct access to the server without passing thru the firewall.
route add –p YourServerNetwork mask NetworkMask Router
Sample:
route add –p 172.16.12.0 mask 255.255.255.0 192.168.11.2
I hope it helps someone, it's better to have this, at least as a clue, so if you face it, you know how to solve it.
I got the same error in Visual Studion 2012 development environment, stopped the IIS Express and rerun the application, it started working.
I had the same issue. I solved it, truncating the SQL Server LOG.
Check doing that, and then tell us, if this solution helped you.
For me the solution was totally different.
In my case I had an objectsource which required a datetimestamp parameter. Even though that ODS parameter ConvertEmptyStringToNull was true 1/1/0001 was being passed to SelectMethod. That in turn caused a sql datetime overflow exception when that datetime was passed to the sql server.
Added an additional check for datetime.year != 0001 and that solved it for me.
Weird that it would throw a transport level error and not a datetime overflow error.
Anyways..
In my case the "SQL Server" Server service stopped. When I restarted the service that enabled me to run the query and eliminate the error.
Its also a good idea to examine your query to find out why the query made this service stop
For me the answer is to upgrade the OS from 2008R2 to 2012R2, the solution of iisreset or restart apppool didn't work for me.
I also tried to turn of TCP Chimney Offload setting, but I didn't restart the server because it is a production server, which didn't work either.
We encountered this error recently between our business server and our database server.
The solution for us was to disable "IP Offloading" on the network interfaces.
Then the error went away.
One of the reason I found for this error is 'Packet Size=xxxxx' in connection string. if the value of xxxx is too large, we will see this error. Either remove this value and let SQL server handle it or keep it low, depending on the network capabilities.
It happened to me when I was trying to restore a SQL database and checked following Check Box in Options tab,
As it's a stand alone database server just closing down SSMS and reopening it solved the issue for me.
This occurs when the database is dropped and re-created some shared resources is still considering the database still exists, so when you re-run execute query to create tables in the database after it was re-created the error will not show again and Command(s) completed successfully. message will show instead of the error message Msg 233, Level 20, State 0, Line 0 A transport-level error has occurred when sending the request to the server. (provider: Shared Memory Provider, error: 0 - No process is on the other end of the pipe.).
Simply ignore this error when you are dropping and recreating databases and re-execute your DDL queries with no worries.
I faced the same issue recently, but i was not able to get answer in google.
So thought of sharing it here, so that it can help someone in future.
Error:
While executing query the query will provide few output then it will throw below error.
"Transport level error has occurred when receiving output from
server(TCP:provider,error:0- specified network name is no longer
available"
Solution:
Check the provider of that linked server
In that provider properties ,Enable "Allow inprocess" option for that particular provider to fix the issue.