IIS SQL connection fails under heavy load; reset IIS fixes temporarily - sql-server-2008

Server 1: SQL Server 2008 Standard Edition
Server 2: Windows 2008 Server R2, IIS 7.0
A web-site on Server 2 requires data in SQL Server on Server 1.
Everything works fine for a while (weeks sometimes). Then, under heavy load, Server 2 reports it cannot connect SQL Server on Server 1. Once IIS on Server 2 reports it cannot connect to SQL on Server 1, it does not get better until IIS on Server 2 is restarted. Perhaps restarting the application pool would work as well as a full IIS reset. I'm not certain.
I've tried changing the connection string to increase the connection pool size to unreasonably large values (1,000). The failures still happen.
The web-site is written in C#. The data access layer is NHibernate.
Here is the start of the exception:
[SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)]
System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject) +428
System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +65
System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +117
System.Data.SqlClient.SqlConnection.Open() +122
NHibernate.Connection.DriverConnectionProvider.GetConnection() +60
NHibernate.Impl.SessionFactoryImpl.OpenConnection() +39
Any ideas why this is happening? How to diagnose/fix it? I'm frustrated and considering ripping out NHibernate, which will take months and probably lead to many other kinds of problems.

I think you have to increase connection timeout value and connection pooling as well. So that it won't release connection to SQL Server in heavy load.

The fix for me was to configure SQL server to allow an unlimited number of connections (set to value of "0"). I still had a few connection issues as described in the original post, but no where near as much. So I also setup a direct connection to the SQL server via the secondary Ethernet port on each server (Webserver to SQL Server),and kept the connection private using 196.168.1.x between the servers, and using it's own Ethernet switch not connected to the public switch.

Related

Intermittent SQL Server Connection Issue

I have been experiencing an intermittent SQL connection problem that I can't resolve. First of all I will explain the setup:
Windows 2008 R2 Standard SP1 (and all patches delivered through windows update):
This server runs IIS 7.5
The ASP.NET MVC 4 Web App runs within it's own App Pool
The App Pool identity is used to set permission on the folder containing the code
The Web App passed a username and password in the connection string - server xxx; database=xxx; Persist Security Info=True; User ID=xxx; Password=xxx; Pooling=False; Network=DBMSSOCN;
elmah used to log errors
Windows 2008 R2 Standard SP1 (and all patches delivered through windows update):
This server runs SQL Server 2008 R2 Web Edition SP2
TCP/IP is the only enabled Client Protocol
These are VM's hosted by our cloud provider. The cloud provider has investigated, from an infrastructure point of view, but cannot find any problems.
The error messages generated are as follows:
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 9 - Associating port with I/O completion mechanism failed)
I've been unable to find any information on error code 9.
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - An operation was attempted on something that is not a socket.) ---> System.ComponentModel.Win32Exception (0x80004005): An operation was attempted on something that is not a socket
While these errors are generated, IIS will show the 'friendly error' page. It usually takes around 15 minutes for the issueto resolve itself or an IISReset. When this issue first occurred I downloaded a SQL Load Generator from codeplex and ran test up to 300 simultaneous connections with multiple query's and everything was ok. I also ran another test while the web app was generating the SQL exceptions - all connections from the load test succeeded. This really confused me!?!
No Event Logs are created during the outage.
To me the error message indicates a problem connecting to the SQL box, however the load test run during the most recent outage contradicts this - yet IIS is still serving content (configured error message).
I realize that this is a difficult question, however I would be very grateful for any advice regarding techniques to find the cause of the error. Any information on what error 9 relates to would also be helpful.

SQLCMD Unable to complete login process due to delay in opening server connection

Sqlcmd: Error: Microsoft SQL Server Native Client 10.0 : Unable to
complete login process due to delay in opening server connection.
When I execute the first query in SQL Server 2008 R2 there was no problems but when I executed the second query there is problem shown above. My database is connected to the local PC and there is no remote connection.
Can anyone help me for finding the above remedy?
Raise your timeouts (connection and command)
Give the sql server more memory
Stop any heavy background processes
Play with netlibs, turn off shared memory and tcpip+netbios turn on and vice versa
Disabling the shared memory worked like a charm for me.
Instructions:
Start SQL Server Configuration Manager.
In SQL Server Configuration Manager, expand SQL Server Network Configuration, and then click Protocols for MSSQLSERVER.
Right-click Shared Memory, and then click Disable.
Click SQL Server Services, right-click SQL Server (MSSQLSERVER), and then click restart.
My office switched to a MultiSubnetFailover (load-balanced) system for our SQL set-up. Our connection manager previously had been set to OLEDB, and as it turns out OLEDB cannot accommodate MultiSubnetFailover=True; that's why we were getting the above error message.
As a result, we switched to an ODBC connection manager instead, and that worked.

SQL Server "network-related or instance-specific error" once a day or so (perplexed!)

We are experiencing the same error as this StackOverflow Q ...
System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 0 - A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)
at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)
at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
at System.Data.SqlClient.SqlConnection.Open()
at System.Data.Linq.SqlClient.SqlConnectionManager.UseConnection(IConnectionUser user)
at System.Data.Linq.SqlClient.SqlProvider.get_IsSqlCe()
at System.Data.Linq.SqlClient.SqlProvider.InitializeProviderMode()
at System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query)
... except that in the referenced StackOverflow Q, they need to restart SQL Server once the error occurs - and we do not. We'll get this error once a day, or once every few days - and all is fine after the error occurs, until the next time it occurs.
This makes us think it's not a "forgot to close connections" issue. We have a moderately busy ASP.NET 4.0 WebForms / SQL Server 2008 R2 app; but we're quite positive we're not exceeding the max # of database connections.
Any thoughts on this problem, or an approach to diagnose?
Thought I would comment on our progress with this.
While none of the SQL Server documentation/articles/blogs mention that this error can be caused by server busyness, I found a forum posting where some seasoned IT pro named Matt Neerincx states that it can be, as follows:
Possible reasons for this error include:
1. Poor network link from client to server.
2. Server is very busy (meaning high CPU) and cannot respond to new connection attempts.
3. Server is running out of memory (so high memory usage for SQL).
4. tcp-ip layer on client is over-saturated with connection attempts so tcp-ip layer rejects the connection.
5. tcp-ip layer on server side is over-staturated with connection attempts and so tcp-ip layer is rejecting new connections.
6. With SQL 2005 SP2 and later there could be a custom login trigger that rejects your connection.
You can increase the connect timeout to potentially alleviate issues #2, #3, #4, #5. Setting a longer connect timeout means the driver will try longer to connect and may eventually succeed.
To determine the root cause of these intermittent failures is not super easy to do unfortunately. What I normally do is start by examining the server environment, is the server constantly running in high CPU for example, this points to #2. Is the server using a hugh amount of memory, this points to #3. You can run SQL Profiler to monitor logins and look for patterns of logins, perhaps every morning at 9AM there is a flurry of connections etc...
So we are presently walking down this path - reducing the # of queries that execute at the same time in some of our batch queries, optimizing some of our queries, etc.
Also, in our app connection string, we increased the connection timeout, and set Min Pool Size to 20 (thinking it's good to try to ensure some existing, unused connections for the app to grab, rather than needing to establish a new connection).
At this moment, it's been almost 48 hours without receiving the error; making us very hopeful.

Connection String for Remote Connections to SQL Server 2008

I am trying to connect remotely to SQL Server 2008 R2 Express on a server running Windows Server 2008 R2. I receive the following error:
A network-related or instance-specific error occurred while
establishing a connection to SQL Server. The server was not found or
was not accessible. Verify that the instance name is correct and that
SQL Server is configured to allow remote connections. (provider: TCP
Provider, error: 0 - A connection attempt failed because the connected
party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond.)
I think the problem might be related to my connection string, because I am not sure exactly what it should be. I understand the connection string should take the form:
Data Source=123.123.123.123\InstanceName;Initial Catalog=MyDBName;user id=MyUserName;password=MyPassword;
The instance name of my SQL Server installation takes the form ComputerName\SQLEXPRESS. This is what I see when I log in to Management Studio on the server. So should the connection string begin with
Data Source=123.123.123.123\ComputerName\SQLEXPRESS
This looks wrong because of all the slashes. I have tried to omit the computer name thus:
Data Source=123.123.123.123\SQLEXPRESS
I also read somewhere that you can specify a port so I have also tried
Data Source=123.123.123.123,1433\SQLEXPRESS
because I understand that 1433 is the TCP port used by SQL Server and
Data Source=123.123.123.123,1434\SQLEXPRESS
because I understand that 1434 is the UDP port used by SQL Server browser.
I have also tried every combination of these and they all give the same error.
These are the steps I have taken to enable remote browsing on the server:
In Management Studio, right click the instance, go to Properties, Security and check SQL Server and Windows Authentication Mode. On the connections tab I have checked "Allow remote connections to this computer".
In SQL Server Configuration Manager I have enabled all four connection options (Shared Memory, Named Pipes, TCPIP and Via) under every node they occur. Under protocols for SQL Express I have tried specifying port 1433 and also leaving it blank with TCP Dynamic Ports set to 0 (which I think is meant to enable dynamic ports).
I have created firewall exceptions for TCP Port 1433, UDP Port 1434, and program exceptions for sqlservr.exe and sqlbrowser.exe.
If someone could tell me which of the four versions of the DataSource part of the connection string I should be using it would be a great help, even if it did not solve the problem. It would mean that everything I try subsequently I would only have to test once instead of four times.
Thanks in advance for any help!
Where did you enter the firewall exceptions, on your machine, on the server, or both?
The second connection string is the only really valid one. Can you telnet to the server on port 1433? Can you connect to the instance from Management Studio remotely and the problem is only from your code where the connection string is? Did you try running the same code on the server? Did you try forcing TCP/IP (vs. named pipes/shared memory etc) adding the following parameter to your connection string:
Network=DBMSSOCN;
This question comes up quite a lot and I'm sure you're hitting something that others have hit before. Have you gone through this article, "How to troubleshoot connecting to the SQL Server Database Engine"? Also there are 20+ questions on here that mention this error message and Express, according to this search result. I suggest double-checking your settings against some of the items in those answers that solved the issue for other users.

A transport-level error has occurred... (Existing connection closed by the server) Sql Server 2008

I have a web application which is currently running on Windows XP operating system with SQL Server 2005 database and IIS 6.0.
Now, we are trying to upgrade it to IIS 7.0 on Windows Server 2008 and SQL server 2008 database.
When I run the application then it is throwing "A transport-level error has occurred when sending the request to the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)" randomly at different database calls.
I have verified using "sp_who" that only one connection which I am using is opened.
Can anyone tell me, what could be the cause of this issue?
This blog post by Michael Aspengren explains the error message "A transport-level error has occurred when sending the request to the server."
Maybe the database is set to auto-close? This used to be the default for databases created on the older MSDE/Express edition.
ALTER DATABASE YourDatabaseName SET AUTO_CLOSE OFF;
More:
The SQL Server 2005 Express Edition Overview states:
Features like Auto-Close and the ability to copy databases as files are enabled by default in SQL Server Express ...
I can't remember which service pack changed the default, but the New Database UI in the SP3 version of Management Studio Express defaults the Auto Close setting to false.
This is an error that occurrs when connecting to database due to a lower layer network-related error that you cannot handle from Sql Server.
It would be great that the API used to conenct handled this error but AFAIK it is not. The only way to solve this is to retry connecting to the database when this error occurrs.
You should try to reconnect at least a couple of times more to get sure that the error is not persistent and if it is the case then throw the exception. My experience tells me that trying to reconnect, the 99.9% of times you will succed.
Often, it's the database being offline/closed or the SQL Instance being restarted/offline
In my case. We have DELL R720, NICs are on board and PCI Broadcom NetXtreme Gigabit Ethernet cards. We run into two issues.
Network speed is very slow for HyperV VMs. The solution is to disable the "Virtual Machine queues" of the HyperV host Broadcom nic property windows. It is in the Advanced tab.
SQL client having this transport level issue. It only happened once in every 10-15 min. At the same time the RDP service kick out client from time to time also. The solution is to disable the "Large Send Offload V2" of the HyperV host Broadcom nic property windows. It is in the Advanced tab.
Check if you have MultipleActiveResultSets=true; set in your connection string.
Transport level errors are often linked to the connection to sql server being broken ... usually network.
Timeout Expired is usually thrown when a sql query takes too long to run. Is this the scenario in your case? Like stored proc taking too much time to execute or are there are batch jobs that are executed?
Use clearpool(urSQLconnection), along with dispose. This will clear the dirty connection. and you can open new connection.
Got this trying to execute commands from multiple threads on one connection object in .net, that was a stupid mistake. Locking it or using a separate connection per thread fixes that.
In my case it was McAffee that was pre-installed. Can tell if it was conflict with other solutions or not. The connection was on VPN, on WiFI. Maybe this will help someone in future.