EC2 MySQL Connection Drops using Private IP Works with Public IP - mysql

We recently split our single EC2 (t2.small) instance which was running (IIS and MySQL) into 2 - (t2.small) instances. We moved all the IIS logic to the new instance and updated the WebConfigs to connect to the old machine which is running MySQL. After firing everything up we noticed very random/sporadic exceptions from MySQL related to connection issues. Overall our connection to the DB works but it seems to drop under certain cases. We tried tweaking timeouts on both IIS and MySQL side but didn't help. The last thing we tried was instead of using the private (Internal) IP for the connection to the MySQL Db we used the public (Elastic IP). With the public IP everything works perfect and we do not get any connection exceptions
Does anyone have any insight into why using the private IP between the servers is inconsistent, but the public IP works very reliably?
MySQL Exception:
*MySql.Data.MySqlClient.MySqlException: Fatal error encountered during command execution. ---> MySql.Data.MySqlClient.MySqlException: Fatal error encountered attempting to read the resultset. ---> MySql.Data.MySqlClient.MySqlException: Reading from the stream has failed. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.LoadPacket()*
*System.Data.Entity.Core.EntityException: The underlying provider failed on Open. ---> MySql.Data.MySqlClient.MySqlException: Reading from the stream has failed. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)*

We ended up solving this issue by following this guide for windows. Setting the below TCP settings in the Registry.
https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\:
KeepAliveTime: 30000
KeepAliveInterval: 1000
TcpMaxDataRetransmissions: 10

Related

For MYSQL 5.7.31 - Which Timeout variable should be changed for a transport connection: Connection timed out

I'm using mysql 5.7.31 with C# on Linux ubuntu
When I programmatically delete a large table or drop a database I get this timeout
Stack=MySql.Data.MySqlClient.MySqlException (0x80004005): Fatal error encountered during command execution.
---> MySql.Data.MySqlClient.MySqlException (0x80004005): Fatal error encountered attempting to read the resultset.
---> MySql.Data.MySqlClient.MySqlException (0x80004005): Reading from the stream has failed.
---> System.IO.IOException: Unable to read data from the transport connection: Connection timed out.
---> System.Net.Sockets.SocketException (110): Connection timed out
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow, Int64& insertedId)
at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force)
at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
To fix this I've attempted
I added this to my connection string
Connect Timeout=60
I added this to my mysqld.cnf, Restarted and checked it show variables. The log file shows it timed-out after 42 seconds.
connect_timeout=60000
interactive_timeout=60000
wait_timeout=60000
I think the issue is "Unable to read data from the transport connection: Connection timed out."
But not sure which variable needs to be changed to fix this?
Thanks
connect_timeout is used only when the client and server are performing a handshake, during the initial client connection. Increasing this helps if the client gets errors like Lost connection to MySQL server.... The default is 10 seconds.
I've never had a reason to tune this variable, because if the client can't connect within 10 seconds, it's probably blocked by a firewall, or routing, or else the MySQL Server is simply not running or is listening on another port. Increasing the timeout will not help in these cases.
wait_timeout helps the client is idle for a long time between queries. The client is not considered idle if the query executes for a long time. The idle time starts when a query finishes. The default is 8 hours (28800 seconds). If your app is timing out during an SQL statement execution, then this variable is not relevant.
interactive_timeout is just like wait_timeout, but for "interactive" clients. The only client that is in this category is the mysql command-line client. You can ignore this timeout variable, unless you're working on the mysql CLI.
You might need to tune net_read_timeout or net_write_timeout. But I wouldn't expect either of these to be necessary.
Tip: Use XFS as the filesystem for the MySQL datadir on Linux. It's much faster to drop tables or schemas, regardless of how large they are. XFS gives better performance for other work too, not just dropping tables.

SSRS - Error with SSAS datasource on certain users

I'm currently setting up the security on SSRS reports at the customer location.
A few of the SSRS reports have a MD cube as a datasource.
When I'm trying to execute a report with an certain user I get an error showed below:
processing!ReportServer_0-1!520!09/28/2018-12:10:30:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: , Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source 'Cube'. ---> Microsoft.AnalysisServices.AdomdClient.AdomdConnectionException: The connection either timed out or was lost. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at Microsoft.AnalysisServices.AdomdClient.DimeRecord.ForceRead(Stream stream, Byte[] buffer, Int32 length)
at Microsoft.AnalysisServices.AdomdClient.DimeRecord.ReadHeader()
at Microsoft.AnalysisServices.AdomdClient.DimeReader.ReadRecord()
at Microsoft.AnalysisServices.AdomdClient.TcpStream.GetResponseDataType()
--- End of inner exception stack trace ---
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.EndRequest(Boolean useBinaryXml)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.SendMessage(Boolean endReceivalIfException, Boolean readSession, Boolean readNamespaceCompatibility)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.SupportsProperty(String propName)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.Connect(ConnectionInfo connectionInfo, Boolean beginSession)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.XmlaClientProvider.Connect(Boolean toIXMLA)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.ConnectToXMLA(Boolean createSession, Boolean isHTTP)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.Open()
at Microsoft.ReportingServices.DataExtensions.AdomdExceptionHandler.Execute(String connectionString, Action action)
at Microsoft.ReportingServices.DataExtensions.AdoMdConnectionBase.Open()
at Microsoft.ReportingServices.Diagnostics.DataExtensionConnectionBase.<>c__DisplayClass5.<OpenConnection>b__4()
--- End of inner exception stack trace ---;
processing!ReportServer_0-1!520!09/28/2018-12:10:30:: e ERROR: Data source 'Cube': An error has occurred. Details: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source 'Cube'. ---> Microsoft.AnalysisServices.AdomdClient.AdomdConnectionException: The connection either timed out or was lost. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
processing!ReportServer_0-1!520!09/28/2018-12:10:30:: e ERROR: An exception has occurred in data source 'Cube'. Details: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source 'Cube'. ---> Microsoft.AnalysisServices.AdomdClient.AdomdConnectionException: The connection either timed out or was lost. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
processing!ReportServer_0-1!520!09/28/2018-12:10:30:: i INFO: DataPrefetch abort handler called for Report with ID=. Aborting data sources ...
I have three AD user groups defined (admins, users and agents).
For the users in my "admins" and "users" these reports work fine.
For the "agents" users I get the error.
I have checked the roles on the SSAS MD cube and they all have the same security at this point.
Here are already a few things I've tried:
I have tried to delete the user in the "Membership" section and re-added him, without any luck.
Removed the SSRS reports (with the error) from the report server and re-added them, without any luck.
Restarted the SSRS service, without any luck
Executing the report with an "Admins" user, was successfull (no errors)
Executing the report with an "Users" user, was successfull (no errors)
Connected with an "Agents" user to the cube via Excel, was successfull
Tried executing the SSRS reports with serveral other "Agents" users, without any luck.
I'm out of ideas now, anyone else got some suggestions?
Note: I don't have access to there AD (this goes through another partner)
Thanks
Kevin

SSRS throws error on IP swap

We have swapped SQL existing IP of working DB server for new server and after that SQL reports when executed from web application throws error. Application work fine for CRUD operation; its reports fails.
But as we swap back IP to old server it works fine.
NOTE: We made windows firewall off.
Error log - Report View System Error for Report "ReportName" :- System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond XXXX:80 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception) --- End of inner exception stack trace --- at Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.GetSecureMethods() at Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.IsSecureMethod(String methodname) at Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.SetConnectionSSLForMethod(String methodname) at Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.ProxyMethodInvocation.Execute[TReturn](RSExecutionConnection connection, ProxyMethod1 initialMethod, ProxyMethod1 retryMethod)

Cannot connect to MySQL datebase [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 9 years ago.
Improve this question
We have a clan MySQL database that tracks player stats in a game. There person in charge of our MySQL is away at the moment and we are getting this following Error
[18:17:18 23] Error in Tablebuilder:
[18:17:18 23] Message: Unable to connect to any of the specified MySQL hosts.
[18:17:18 23] Native: -2147467259
[18:17:18 24] Source: MySql.Data
[18:17:18 24] StackTrace: at MySql.Data.MySqlClient.NativeDriver.Open()
at MySql.Data.MySqlClient.Driver.Open()
at MySql.Data.MySqlClient.Driver.Create(MySqlConnecti onStringBuilder settings)
at MySql.Data.MySqlClient.MySqlPool.GetPooledConnecti on()
at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()
at MySql.Data.MySqlClient.MySqlPool.GetConnection()
at MySql.Data.MySqlClient.MySqlConnection.Open()
at PRoConEvents.CChatGUIDStatsLoggerBF3.tablebuilder( )
[18:17:18 24] InnerException: System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 50.63.244.184:3306
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at MySql.Data.Common.StreamCreator.CreateSocketStream (IPAddress ip, Boolean unix)
at MySql.Data.Common.StreamCreator.GetStreamFromHost( String pipeName, String hostName, UInt32 timeout)
at MySql.Data.Common.StreamCreator.GetStream(UInt32 timeout)
at MySql.Data.MySqlClient.NativeDriver.Open()
[18:17:18 24] Error: System.NullReferenceException: Object reference not set to an instance of an object.
at PRoConEvents.CChatGUIDStatsLoggerBF3.tablebuilder( )
Any ideas what the issue could be?
You should start by reading error messages. They look like gibberish but when you read closely, often there is a part it tells you what's going on.
[18:17:18 24] InnerException: System.Net.Sockets.SocketException:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 50.63.244.184:3306
You should now:
Execute telnet localhost 3306 from the database server and if it fails, it's an issue with the DB server
Execute telnet 50.63.244.184 3306 from the application server and if it fails, it's an issue with the network
You have two exceptions, them tested, but the meaning is clear:
"Unable to connect to any of the specified MySQL hosts."
and
"A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"
So in short: your webserver cannot connect to you database server. Your database server (the MySQL server process mysqld.exe or the server itself, like its OS, hardware, anything) might be stopped. Or the firewall was reset to block connections on the MySQL port 3306. It could be anything.
I assume you've got a standard webhost - ping them and ask them why your database server isn't operational.
...but before you embarrass yourself, make sure that your application is connecting to the right server (i.e. production DB, not dev or staging DB). Take a look at your connection string.

NHibernate + MySql Intermittent TimeoutException

I have just migrated my NHibernate app from SQL Server to MySql. All seems well except I am getting intermittent TimeoutExceptions. I can find a lot of info about this timeouts with MySql and even one blog post about timeouts with Hibernate + MySql + C3PO (whatever that is). But nothing about how to tweak NHib to stop this problem. Do you know?
Here is the stack trace:
NHibernate.TransactionException: Begin failed with SQL exception ---> System.TimeoutException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
--- End of inner exception stack trace ---
at MyNetworkStream.HandleOrRethrowException(Exception e)
at MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.NativeDriver.ReadOk(Boolean read)
at MySql.Data.MySqlClient.NativeDriver.SetDatabase(String dbName)
at MySql.Data.MySqlClient.Driver.SetDatabase(String dbName)
at MySql.Data.MySqlClient.MySqlConnection.ChangeDatabase(String databaseName)
at MySql.Data.MySqlClient.MySqlConnection.Open()
at NHibernate.Connection.DriverConnectionProvider.GetConnection()
at NHibernate.AdoNet.ConnectionManager.GetConnection()
at NHibernate.Impl.SessionImpl.get_Connection()
at NHibernate.Transaction.AdoTransaction.Begin(IsolationLevel isolationLevel)
--- End of inner exception stack trace ---
at NHibernate.Transaction.AdoTransaction.Begin(IsolationLevel isolationLevel)
at NHibernate.Transaction.AdoTransaction.Begin()
at NHibernate.AdoNet.ConnectionManager.BeginTransaction()
at NHibernate.Impl.SessionImpl.BeginTransaction()
I run NHibernate on Rackspace cloud with a MYSQL database. I found I was getting timeouts due to the fact that connections were being left semi open due to sleeping. Eventually I was running out of connections and my web site timed out.
In the end I added Pooling=false; into my connection string. I am not sure if this is the ultimate fix but it has stopped my problem.
You can check for sleeping connections by using show processlist in a MySQL editor...