NHibernate + MySql Intermittent TimeoutException - mysql

I have just migrated my NHibernate app from SQL Server to MySql. All seems well except I am getting intermittent TimeoutExceptions. I can find a lot of info about this timeouts with MySql and even one blog post about timeouts with Hibernate + MySql + C3PO (whatever that is). But nothing about how to tweak NHib to stop this problem. Do you know?
Here is the stack trace:
NHibernate.TransactionException: Begin failed with SQL exception ---> System.TimeoutException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
--- End of inner exception stack trace ---
at MyNetworkStream.HandleOrRethrowException(Exception e)
at MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.NativeDriver.ReadOk(Boolean read)
at MySql.Data.MySqlClient.NativeDriver.SetDatabase(String dbName)
at MySql.Data.MySqlClient.Driver.SetDatabase(String dbName)
at MySql.Data.MySqlClient.MySqlConnection.ChangeDatabase(String databaseName)
at MySql.Data.MySqlClient.MySqlConnection.Open()
at NHibernate.Connection.DriverConnectionProvider.GetConnection()
at NHibernate.AdoNet.ConnectionManager.GetConnection()
at NHibernate.Impl.SessionImpl.get_Connection()
at NHibernate.Transaction.AdoTransaction.Begin(IsolationLevel isolationLevel)
--- End of inner exception stack trace ---
at NHibernate.Transaction.AdoTransaction.Begin(IsolationLevel isolationLevel)
at NHibernate.Transaction.AdoTransaction.Begin()
at NHibernate.AdoNet.ConnectionManager.BeginTransaction()
at NHibernate.Impl.SessionImpl.BeginTransaction()

I run NHibernate on Rackspace cloud with a MYSQL database. I found I was getting timeouts due to the fact that connections were being left semi open due to sleeping. Eventually I was running out of connections and my web site timed out.
In the end I added Pooling=false; into my connection string. I am not sure if this is the ultimate fix but it has stopped my problem.
You can check for sleeping connections by using show processlist in a MySQL editor...

Related

For MYSQL 5.7.31 - Which Timeout variable should be changed for a transport connection: Connection timed out

I'm using mysql 5.7.31 with C# on Linux ubuntu
When I programmatically delete a large table or drop a database I get this timeout
Stack=MySql.Data.MySqlClient.MySqlException (0x80004005): Fatal error encountered during command execution.
---> MySql.Data.MySqlClient.MySqlException (0x80004005): Fatal error encountered attempting to read the resultset.
---> MySql.Data.MySqlClient.MySqlException (0x80004005): Reading from the stream has failed.
---> System.IO.IOException: Unable to read data from the transport connection: Connection timed out.
---> System.Net.Sockets.SocketException (110): Connection timed out
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.LoadPacket()
at MySql.Data.MySqlClient.MySqlStream.ReadPacket()
at MySql.Data.MySqlClient.NativeDriver.GetResult(Int32& affectedRow, Int64& insertedId)
at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force)
at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
at MySql.Data.MySqlClient.MySqlDataReader.NextResult()
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteReader(CommandBehavior behavior)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
To fix this I've attempted
I added this to my connection string
Connect Timeout=60
I added this to my mysqld.cnf, Restarted and checked it show variables. The log file shows it timed-out after 42 seconds.
connect_timeout=60000
interactive_timeout=60000
wait_timeout=60000
I think the issue is "Unable to read data from the transport connection: Connection timed out."
But not sure which variable needs to be changed to fix this?
Thanks
connect_timeout is used only when the client and server are performing a handshake, during the initial client connection. Increasing this helps if the client gets errors like Lost connection to MySQL server.... The default is 10 seconds.
I've never had a reason to tune this variable, because if the client can't connect within 10 seconds, it's probably blocked by a firewall, or routing, or else the MySQL Server is simply not running or is listening on another port. Increasing the timeout will not help in these cases.
wait_timeout helps the client is idle for a long time between queries. The client is not considered idle if the query executes for a long time. The idle time starts when a query finishes. The default is 8 hours (28800 seconds). If your app is timing out during an SQL statement execution, then this variable is not relevant.
interactive_timeout is just like wait_timeout, but for "interactive" clients. The only client that is in this category is the mysql command-line client. You can ignore this timeout variable, unless you're working on the mysql CLI.
You might need to tune net_read_timeout or net_write_timeout. But I wouldn't expect either of these to be necessary.
Tip: Use XFS as the filesystem for the MySQL datadir on Linux. It's much faster to drop tables or schemas, regardless of how large they are. XFS gives better performance for other work too, not just dropping tables.

EC2 MySQL Connection Drops using Private IP Works with Public IP

We recently split our single EC2 (t2.small) instance which was running (IIS and MySQL) into 2 - (t2.small) instances. We moved all the IIS logic to the new instance and updated the WebConfigs to connect to the old machine which is running MySQL. After firing everything up we noticed very random/sporadic exceptions from MySQL related to connection issues. Overall our connection to the DB works but it seems to drop under certain cases. We tried tweaking timeouts on both IIS and MySQL side but didn't help. The last thing we tried was instead of using the private (Internal) IP for the connection to the MySQL Db we used the public (Elastic IP). With the public IP everything works perfect and we do not get any connection exceptions
Does anyone have any insight into why using the private IP between the servers is inconsistent, but the public IP works very reliably?
MySQL Exception:
*MySql.Data.MySqlClient.MySqlException: Fatal error encountered during command execution. ---> MySql.Data.MySqlClient.MySqlException: Fatal error encountered attempting to read the resultset. ---> MySql.Data.MySqlClient.MySqlException: Reading from the stream has failed. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size) at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count) at MySql.Data.MySqlClient.MySqlStream.LoadPacket()*
*System.Data.Entity.Core.EntityException: The underlying provider failed on Open. ---> MySql.Data.MySqlClient.MySqlException: Reading from the stream has failed. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)*
We ended up solving this issue by following this guide for windows. Setting the below TCP settings in the Registry.
https://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\:
KeepAliveTime: 30000
KeepAliveInterval: 1000
TcpMaxDataRetransmissions: 10

SSRS - Error with SSAS datasource on certain users

I'm currently setting up the security on SSRS reports at the customer location.
A few of the SSRS reports have a MD cube as a datasource.
When I'm trying to execute a report with an certain user I get an error showed below:
processing!ReportServer_0-1!520!09/28/2018-12:10:30:: e ERROR: Throwing Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: , Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source 'Cube'. ---> Microsoft.AnalysisServices.AdomdClient.AdomdConnectionException: The connection either timed out or was lost. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at Microsoft.AnalysisServices.AdomdClient.DimeRecord.ForceRead(Stream stream, Byte[] buffer, Int32 length)
at Microsoft.AnalysisServices.AdomdClient.DimeRecord.ReadHeader()
at Microsoft.AnalysisServices.AdomdClient.DimeReader.ReadRecord()
at Microsoft.AnalysisServices.AdomdClient.TcpStream.GetResponseDataType()
--- End of inner exception stack trace ---
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.EndRequest(Boolean useBinaryXml)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.SendMessage(Boolean endReceivalIfException, Boolean readSession, Boolean readNamespaceCompatibility)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.SupportsProperty(String propName)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.Connect(ConnectionInfo connectionInfo, Boolean beginSession)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.XmlaClientProvider.Connect(Boolean toIXMLA)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.ConnectToXMLA(Boolean createSession, Boolean isHTTP)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.Open()
at Microsoft.ReportingServices.DataExtensions.AdomdExceptionHandler.Execute(String connectionString, Action action)
at Microsoft.ReportingServices.DataExtensions.AdoMdConnectionBase.Open()
at Microsoft.ReportingServices.Diagnostics.DataExtensionConnectionBase.<>c__DisplayClass5.<OpenConnection>b__4()
--- End of inner exception stack trace ---;
processing!ReportServer_0-1!520!09/28/2018-12:10:30:: e ERROR: Data source 'Cube': An error has occurred. Details: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source 'Cube'. ---> Microsoft.AnalysisServices.AdomdClient.AdomdConnectionException: The connection either timed out or was lost. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
processing!ReportServer_0-1!520!09/28/2018-12:10:30:: e ERROR: An exception has occurred in data source 'Cube'. Details: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source 'Cube'. ---> Microsoft.AnalysisServices.AdomdClient.AdomdConnectionException: The connection either timed out or was lost. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
processing!ReportServer_0-1!520!09/28/2018-12:10:30:: i INFO: DataPrefetch abort handler called for Report with ID=. Aborting data sources ...
I have three AD user groups defined (admins, users and agents).
For the users in my "admins" and "users" these reports work fine.
For the "agents" users I get the error.
I have checked the roles on the SSAS MD cube and they all have the same security at this point.
Here are already a few things I've tried:
I have tried to delete the user in the "Membership" section and re-added him, without any luck.
Removed the SSRS reports (with the error) from the report server and re-added them, without any luck.
Restarted the SSRS service, without any luck
Executing the report with an "Admins" user, was successfull (no errors)
Executing the report with an "Users" user, was successfull (no errors)
Connected with an "Agents" user to the cube via Excel, was successfull
Tried executing the SSRS reports with serveral other "Agents" users, without any luck.
I'm out of ideas now, anyone else got some suggestions?
Note: I don't have access to there AD (this goes through another partner)
Thanks
Kevin

SSRS throws error on IP swap

We have swapped SQL existing IP of working DB server for new server and after that SQL reports when executed from web application throws error. Application work fine for CRUD operation; its reports fails.
But as we swap back IP to old server it works fine.
NOTE: We made windows firewall off.
Error log - Report View System Error for Report "ReportName" :- System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond XXXX:80 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception) --- End of inner exception stack trace --- at Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.GetSecureMethods() at Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.IsSecureMethod(String methodname) at Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.SetConnectionSSLForMethod(String methodname) at Microsoft.Reporting.WebForms.Internal.Soap.ReportingServices2005.Execution.RSExecutionConnection.ProxyMethodInvocation.Execute[TReturn](RSExecutionConnection connection, ProxyMethod1 initialMethod, ProxyMethod1 retryMethod)

Can't Connect to Remote MySQL Database some times

I have a Mysql DB Connected to ASP.NET MVC. In local it's working fine but when connected to a remote server sometimes it works fine and sometimes it has problems.
Any ideas to solve this problem would be appreciated.
The underlying provider failed on Open.
*************************
MySql.Data.MySqlClient.MySqlException (0x80004005): Authentication to host 'aleppo-cuisine.com' for user 'xxx' using method 'mysql_native_password' failed with message: Reading from the stream has failed. ---> MySql.Data.MySqlClient.MySqlException (0x80004005): Reading from the stream has failed. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at MySql.Data.Common.MyNetworkStream.HandleOrRethrowException(Exception e)
at MySql.Data.Common.MyNetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.TimedStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at MySql.Data.MySqlClient.MySqlStream.ReadFully(Stream stream, Byte[] buffer, Int32 offset, Int32 count)