Can someone explain why we are experiencing a total consumption of the connection pool when performing queries inside TransactionScope resulting in
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
I have reduced our problem to the following:
SomeNonTransactionalCode()
{
// This code will execute with just one connection to the database
// and complete without exception
List<BusinessEntity> beList;
for (int i = 0; i < 101; i++)
{
BusinessEntityRepository beRepo = new BusinessEntityRepository();
beList = beRepo.ReadAll().ToList();
}
}
SomeTransactionalCode()
{
// This code will cause the connections to the database to increment
// with every iteration eventually timing out after 100 connections
using (TransactionScope transactionScope = new TransactionScope())
{
List<BusinessEntity> beList;
for (int i = 0; i < 101; i++)
{
BusinessEntityRepository beRepo = new BusinessEntityRepository();
beList = beRepo.ReadAll().ToList();
}
transactionScope.Complete();
}
}
EDIT
After Omer's answer below I think the problem is explained better like this:
SomeNonTransactionalCode()
{
// This code will execute with just one connection to the database
List<BusinessEntity1> be1List;
BusinessEntity1Repository be1Repo = new BusinessEntity1Repository();
be1List = be1Repo .ReadAll().ToList();
List<BusinessEntity2> be2List;
BusinessEntity2Repository be2Repo = new BusinessEntity2Repository();
be2List = be2Repo .ReadAll().ToList();
List<BusinessEntity3> be3List;
BusinessEntity3Repository be3Repo = new BusinessEntity3Repository();
be3List = be3Repo.ReadAll().ToList();
}
SomeTransactionalCode()
{
// This code will cause three seperate connections to the database
using (TransactionScope transactionScope = new TransactionScope())
{
// note this is simplified - the code below could be in unknown nested
// methods make creating of the repos prior to calling not possible
List<BusinessEntity1> be1List;
BusinessEntity1Repository beRepo1 = new BusinessEntity1Repository();
be1List = be1Repo.ReadAll().ToList();
List<BusinessEntity2> be2List;
BusinessEntity2Repository beRepo2 = new BusinessEntity2Repository();
be2List = be2Repo.ReadAll().ToList();
List<BusinessEntity3> be3List;
BusinessEntity3Repository beRepo3 = new BusinessEntity3Repository();
be3List = be3Repo.ReadAll().ToList();
transactionScope.Complete();
}
}
Surely this is not expected behaviour? I have read nothing that explains why this might be happening. I can only assume it is something to do with how we have implemented our repositories. Hopefully the following will give a good enough description of the implementation.
public class BusinessEntityRepository
{
private BusinessEntityDal Dal { get; set; }
public BusinessEntityRepository()
{
this.Dal = new BusinessEntityDal ();
}
public IQueryable<BusinessEntity> ReadAll()
{
IQueryable<BusinessEntity> query = null;
if (Dal != null)
{
query = Dal.ReadAll();
}
//-Return
return query;
}
}
public class BusinessEntityDal : BaseDal
{
public IQueryable<BusinessEntity> ReadAll()
{
var result = from de in this.DC.BusinessEntityTable
select new BusinessEntity
{
Property1 = Column1,
Property2 = Column2,
// etc...
};
//-Return
return (result);
}
}
public abstract class BaseDal
{
protected OurDataContext DC;
public BaseDal()
{
// create a DataContext
this.DC = new OurDataContext();
}
}
public class OurDataContext : System.Data.Linq.DataContext
{
private static readonly string _cn = // some static connection string taken from web.config
public OurDataContext()
: base(OurDataContext._cn)
{
}
}
Our connection string is fairly conventional and leaves the number of connections in the pool at the default 100 (hence the 101 iterations to test the issue in my code above).
You are creating new DataContext references inside the for loop.
for (int i = 0; i < 101; i++)
{
BusinessEntityRepository beRepo = new BusinessEntityRepository();
beList = beRepo.ReadAll().ToList();
}
It is keeping all those in different transactions. If you just put repo init code outside the for loop and perform all operations in one context, it will be fine.
using (TransactionScope transactionScope = new TransactionScope())
{
List<BusinessEntity> beList;
BusinessEntityRepository beRepo = new BusinessEntityRepository();
for (int i = 0; i < 101; i++)
{
beList = beRepo.ReadAll().ToList();
}
//do some other things with same context
transactionScope.Complete();
}
Related
I have connect Processing and SQL by using database library "de.Bezier.data.sql".
I don't know How can I get the name of columns in a specific Table.
I get the correct name of database, but i got the following as result of name of columns "Tables_in_sql7363100"
import de.bezier.data.sql.*;
MySQL sql;
String[] tableNames;
String[] columnNames;
void setup() {
size(300, 300);
database_connection();
if (connect) {
tableNames = sql.getTableNames();
for (int i=0; i<tableNames.length; i++) {
println(tableNames[i]);
}
columnNames = sql.getColumnNames();
for (int i=0; i<ColumnNames.length; i++) {
println(columnNames[i]);
}
}
}
void draw() {
background(255);
}
void database_connection() {
sql = new MySQL(this, "ServerName", "DataBase", "DUN", "PW");
if (sql.connect()) {
connect = true;
connect_status = "Conected";
} else {
connect = false;
connect_status = "Connection Failed";
}
}
There are 2 problems with what I'm seeing. The first one, which interests you, is that you didn't select any table. That's why you don't get a list of columns. You can fix this by using a simple query:
sql.query("SELECT * FROM myTable");
But that's not the only thing: you're not accounting for lag. It may work for now on a local database because lag is really low, but this won't fly with something which is over the internet. Here's an example where I show columns from a public test database and how long it takes to get the result from my query back:
import de.bezier.data.sql.*;
MySQL sql;
String user = "rfamro";
String pass = "";
String server = "mysql-rfam-public.ebi.ac.uk:4497";
String database = "Rfam";
String[] columnNames;
void setup() {
size(300, 300);
sql = new MySQL(this, server, database, user, pass);
}
void draw() {
if (columnNames != null) {
println("It took " + millis() + "ms to get this data back.");
for (String s : columnNames) {
println(s);
}
noLoop();
} else if (sql.connect()) {
sql.query("SELECT * FROM family");
sql.next(); // only use .next when you know you have data
columnNames = sql.getColumnNames();
}
}
From here, it takes between 2-7 seconds to get the data back. You'll understand that, the setup() method running taking about a couple milliseconds, you won't have any results back by then.
Hope this helps. Have fun!
I am trying to execute a package programmatically. The SSIS packages were deployed using the Project deployment model. I used the following as the example:
http://microsoft-ssis.blogspot.com/2013/01/call-ssis-2012-package-within-net.html
// Connection to the database server where the packages are located
SqlConnection ssisConnection = new SqlConnection
(#"Data Source=SSIS_SERVER;Initial Catalog=master;
Integrated Security=SSPI;");
// SSIS server object with connection
IntegrationServices ssisServer = new IntegrationServices(ssisConnection);
// The reference to the package which you want to execute
// (Note: formatted for ease of reading)
PackageInfo ssisPackage = ssisServer
.Catalogs["SSISDB"]
.Folders["DEV_FOLDER"]
.Projects["TestParentChildDeployment"]
.Packages["Child.dtsx"];
// Add execution parameter to override the default asynchronized execution.
//If you leave this out the package is executed asynchronized
//Collection<PackageInfo.ExecutionValueParameterSet> executionParameter
// = new Collection<PackageInfo.ExecutionValueParameterSet>();
var executionParameter = new Collection<PackageInfo
.ExecutionValueParameterSet>();
executionParameter.Add
(new PackageInfo.ExecutionValueParameterSet
{
ObjectType = 50
, ParameterName = "SYNCHRONIZED"
, ParameterValue = 1
}
);
// Get the identifier of the execution to get the log
long executionIdentifier = ssisPackage.Execute (false, null, executionParameter); // Times out here <<<<
Here is the error: Message=Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
It runs successfully once in a while, but more often than not, it fails. Any ideas?
Here is my solution. Instead of calling the package synchronously, call is asynchronously and use a ManualResetEvent in the calling code to block the main thread until the package completes.
namespace Anthony.Staging.Integration.Tests
{
using System;
using System.Data;
using System.Data.SqlClient;
using System.Diagnostics;
using System.Linq;
using System.Threading;
using Microsoft.SqlServer.Management.Common;
using Microsoft.VisualStudio.TestTools.UnitTesting;
//Add references via %WINDIR%\assembly\GAC_MSIL\:
/* Microsoft.SqlServer.ConnectionInfo
* Microsoft.SqlServer.Management.InterationServices
* Microsoft.SqlServer.Management.Sdk.Sfc
* Microsoft.SqlServer.Smo
*/
using Microsoft.SqlServer.Management.IntegrationServices;
[TestClass]
public class ConnectionTests
{
private const string connectionString = #"Data Source=Anthony\MSSQL11;Initial Catalog=SSISDB;Integrated Security=SSPI;";
private static SqlConnection connection;
private static ServerConnection conn;
private static IntegrationServices ssis;
private static string catalog = "SSISDB";
private static string folder = "Anthony.Integration";
[TestInitialize]
public void MyTestInitialize()
{
connection = new SqlConnection(connectionString);
ssis = new IntegrationServices(connection);
}
[TestMethod]
public void CanConnect()
{
Assert.IsNotNull(connection);
Assert.IsTrue(connection.State == ConnectionState.Open);
}
[TestMethod]
public void CanGetKnownPackage()
{
try
{
var package = this.GetPackage(catalog, folder, "Anthony.Staging", "_Extract_Full.dtsx");
Assert.IsNotNull(package);
}
catch (Exception ex)
{
Assert.Fail(ex.Message);
}
}
[TestMethod]
public void CanExecuteKnownPackage()
{
const int MAX_RETRIES = 20;
const int RETRY_INTERVAL_SECONDS = 10;
const int OPERATION_TIMEOUT_MINUTES = 10;
// get the package from the SSISCatalog and start it asynchronously
// because starting it synchronously will time out after 30 seconds.
var package = this.GetPackage(catalog, folder, "Anthony.Staging", "_Extract_Full.dtsx");
var executionIdentifier = package.Execute(false, null);
// block the main thread and kick off a timer immediately which checks execution status for an ssis execution identifier.
var mre = new ManualResetEvent(false);
var statusChecker = new StatusChecker(executionIdentifier, MAX_RETRIES);
var timer = new Timer(statusChecker.CheckStatus, new StatusCheckerState(mre), new TimeSpan(0, 0, 0), new TimeSpan(0, 0, RETRY_INTERVAL_SECONDS));
WaitHandle.WaitAny(new WaitHandle[] { mre }, (int)new TimeSpan(0, OPERATION_TIMEOUT_MINUTES, 0).TotalMilliseconds, false);
mre.Dispose();
timer.Dispose();
// get the results
var execution = ssis.Catalogs.Single(x => x.Name.Equals(catalog)).Executions.Single(x => x.Id.Equals(executionIdentifier));
var errors = execution.Messages.Where(m => m.MessageType == 120).Select(m => m.Message);
var warnings = execution.Messages.Where(m => m.MessageType == 110).Select(m => m.Message);
Assert.AreEqual(0, errors.Count());
Assert.AreEqual(Operation.ServerOperationStatus.Success, execution.Status);
}
class StatusCheckerState
{
public StatusCheckerState(ManualResetEvent waitHandle)
{
this.WaitHandle = waitHandle;
}
public ManualResetEvent WaitHandle { get; private set; }
}
class StatusChecker
{
private readonly long executionIdentifier;
private int invokeCount;
private ManualResetEvent waitHandle;
private readonly int maximumCount;
public StatusChecker(long executionIdentifier, int maxCount)
{
this.executionIdentifier = executionIdentifier;
invokeCount = 0;
maximumCount = maxCount;
}
// This method is called by the timer delegate.
public void CheckStatus(object state)
{
var localState = ((StatusCheckerState)state);
this.waitHandle = localState.WaitHandle;
if (invokeCount > 0)
{
Debug.WriteLine("Retry attempt: {0}", invokeCount);
}
invokeCount++;
if (invokeCount == maximumCount)
{
// Reset the counter and signal Main.
invokeCount = 0;
waitHandle.Set();
}
var execution = new IntegrationServices(connection).Catalogs.Single(x => x.Name.Equals(catalog)).Executions.Single(x => x.Id.Equals(executionIdentifier));
Debug.WriteLine("Status of execution " + executionIdentifier + " is " + execution.Status);
if (execution.Status == Operation.ServerOperationStatus.Success)
{
// Reset the counter and signal Main.
invokeCount = 0;
waitHandle.Set();
}
}
}
[TestCleanup]
public void MyTestCleanup()
{
if(connection.State == ConnectionState.Open)
connection.Close();
ssis = null;
}
private PackageInfo GetPackage(string catalog, string folder, string project, string package)
{
if (ssis.Catalogs.Contains(catalog))
{
var cat = ssis.Catalogs[catalog];
if (cat.Folders.Contains(folder))
{
var fold = cat.Folders[folder];
if (fold.Projects.Contains(project))
{
var proj = fold.Projects[project];
if (proj.Packages.Contains(package))
{
var pkg = proj.Packages[package];
return pkg;
}
}
}
}
throw new Exception("Cannot find package!");
}
}
}
Here is some code to solve the timeout... it's a bit dirty, but works great:
// Get the identifier of the execution to get the log
long executionIdentifier = ssisPackage.Execute(false, null, executionParameter);
/****
* This is a shameful workaround to not having a timeout override
* for PackageInfo.Execute.
****/
ExecutionOperation executionOperation = ssisServer.Catalogs["SSISDB"].Executions[executionIdentifier];
while (!(executionOperation.Completed))
{
executionOperation.Refresh();
System.Threading.Thread.Sleep(5000);
}
Source:
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/7f0967af-fdeb-4040-9c57-2fe14f2291b5/timeout-after-30-seconds-when-executing-package-via-net?forum=sqlintegrationservices
Got it working - Posted My solution below but will like to know if there is better way
Hello All
I am trying to create Domain Event for a newly created (after migration) domain object in my database.
for Objects without any internal child objects it worked fine by using Script Component. The problem is in how to get the child rows to add information to event object.
Ex. Customer-> Customer Locations.
I am creating Event in Script Component- as tranformation- (have reference to my Domain event module) and then creating sending serialized information about event as a column value. The input rows currently provide data for the parent object.
Please advise.
Regards,
The Mar
Edit 1
I would like to add that current I am doing processsing in
public override void Input0_ProcessInputRow(Input0Buffer Row)
I am looking for something like create a a data reader in this function
loop through data rows -> create child objecta nd add it to parent colelction
Still on google and PreExecute and ProcessInput Seems something to look at .
This is my solution. I am a total newbie in SSIS , so this may not be the best solution.
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
IDTSConnectionManager100 connectionManager;
SqlCommand cmd = null;
SqlConnection conn = null;
SqlDataReader reader = null;
public override void AcquireConnections(object Transaction)
{
try
{
connectionManager = this.Connections.ScriptConnectionManager;
conn = connectionManager.AcquireConnection(Transaction) as SqlConnection;
// Hard to debug failure- better off logging info to file
//using (StreamWriter outfile =
// new StreamWriter(#"f:\Migration.txt"))
//{
// outfile.Write(conn.ToString());
// outfile.Write(conn.State.ToString());
//}
}
catch (Exception ex)
{
//using (StreamWriter outfile =
// new StreamWriter(#"f:\Migration.txt"))
//{
// outfile.Write(" EEEEEEEEEEEEEEEEEEEE"+ ex.ToString());
//}
}
}
public override void PreExecute()
{
base.PreExecute();
cmd = new SqlCommand("SELECT [CustomerLocation fields] FROM customerlocationView where custid=#CustId", conn);
cmd.Parameters.Add("CustId", SqlDbType.UniqueIdentifier);
}
public override void PostExecute()
{
base.PostExecute();
/*
Add your code here for postprocessing or remove if not needed
You can set read/write variables here, for example:
Variables.MyIntVar = 100
*/
}
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
Collection<CustomerLocation> locations = new Collection<CustomerLocation>();
cmd.Parameters["CustId"].Value = Row.id;
// Any error always saw that reader reamians open on connection
if (reader != null)
{
if (!reader.IsClosed)
{
reader.Close();
}
}
reader = cmd.ExecuteReader();
if (reader != null)
{
while (reader.Read())
{
// Get Child Details
var customerLocation = new CustomerLocation(....,...,...,);
customerLocation.CustId = Row.id;
locations.Add(customerLocation);
}
}
var newCustomerCreated = new NewCustomerCreated(Row.id,,...,...,locations);
var serializedEvent = JsonConvert.SerializeObject(newCustomerCreated, Formatting.Indented,
new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.Objects, ReferenceLoopHandling = ReferenceLoopHandling.Ignore });
Row.SerializedEvent = serializedEvent;
Row.EventId = newCustomerCreated.EventId;
...
...
...
....
..
.
Row.Version = 1;
// using (StreamWriter outfile =
// new StreamWriter(#"f:\Migration.txt", true))
// {
// if (reader != null)
// {
// outfile.WriteLine(reader.HasRows);
//outfile.WriteLine(serializedEvent);
// }
// else
// {
// outfile.Write("reader is Null");
// }
//}
reader.Close();
}
public override void ReleaseConnections()
{
base.ReleaseConnections();
connectionManager.ReleaseConnection(conn);
}
}
One thing to note is that a different approach to create connection is to
get the connection string from connectionManager and use it to create OLEDB connection.
If I use this code in each Repository class then I get SQL profiling to work but I want to move that code from each class into the class where StructureMap handles the DB.
Example of a Repository class:
public DB CreateNewContext()
{
var sqlConnection = new SqlConnection(ConfigurationManager.ConnectionStrings["connection"].ConnectionString);
var profiledConnection = ProfiledDbConnection.Get(sqlConnection);
return DataContextUtils.CreateDataContext<DB>(profiledConnection);
}
public SqlRecipeRepository(DB dataContext)
{
_db = CreateNewContext();
}
Now I want the dataContext variable to be the profiled version and so come from my DBServiceRegistry class.
Here is the DBServiceRegistry class:
public class DBServiceRegistry : Registry
{
public DBServiceRegistry()
{
var sqlConnection = new SqlConnection(ConfigurationManager.ConnectionStrings["GetMeCooking.Data.Properties.Settings.server"].ConnectionString);
var profiledConnection = ProfiledDbConnection.Get(sqlConnection);
For<DB>().HybridHttpOrThreadLocalScoped().Use(() => DataContextUtils.CreateDataContext<DB>(profiledConnection));
//Original method just had this:
//For<DB>().HybridHttpOrThreadLocalScoped().Use(() => new DB());
}
}
This code does not cause any errors but I don't get the SQL profiling, what am I doing wrong?
The comment is correct, by creating the sql connection outwith the For line, you are overriding the scope command.
Far better to encapsulate the whole lot into an anonymous delegate
using System.Configuration;
using System.Data.SqlClient;
using System.Threading.Tasks;
using StructureMap;
using StructureMap.Configuration.DSL;
using Xunit;
public class DBServiceRegistry : Registry
{
private string connString = ConfigurationManager.ConnectionStrings["GetMeCooking.Data.Properties.Settings.server"].ConnectionString;
public DBServiceRegistry()
{
For<DB>().HybridHttpOrThreadLocalScoped().Use(
() =>
{
var sqlConnection = new SqlConnection(connString);
var profiledConnection = new StackExchange.Profiling.Data.ProfiledDbConnection(sqlConnection, MiniProfiler.Current);
return DataContextUtils.CreateDataContext<DB>(profiledConnection);
});
}
}
You can use unit tests to verify that the scope is correct (test syntax is xunit.net)
public class DBRegistryTests : IDisposable
{
private Container container;
public DBRegistryTests()
{
// Arrange (or test setup)
container = new Container(new DBServiceRegistry());
}
[Fact]
public void ConnectionsAreSameInThread()
{
// Create two connections on same thread
var conn1 = container.GetInstance<DB>();
var conn2 = container.GetInstance<DB>();
// Assert should be equal because hybrid thread is scope
// and test executes on same thread
Assert.Equal(conn1, conn2);
// Other assertions that connection is profiled
}
[Fact]
public void ConnectionAreNotSameInDifferentThreads()
{
var conn1 = container.GetInstance<DB>();
// Request second connection from a different thread
// (for < c# 4.0 use Thread instead of task)
var conn2 = new Task<DB>(() => this.container.GetInstance<DB>());
conn2.Start();
conn2.Wait();
// Assert that request from two different threads
// are not the same
Assert.NotEqual(conn1, conn2.Result);
}
public void Dispose()
{
// Test teardown
container.Dispose();
}
}
I'm wrote an insert method that uses linq and loops through 2 lists, the first being able to go up to 14k objects and the send about 8k objects.
Whenever I run this method, I always get "Transaction Timeout Exception". Can you help me improve this?
public void InsertNewInventoryGoodsEvents(List<GoodsEvent> eventsList, List<InventoryGoods> goodsList)
{
InventoryGoodsEvents ige = new InventoryGoodsEvents();
TransactionScope scope= new TransactionScope();
int i = 0;
const int batchSize = 50; // or even 50
foreach (InventoryGoods good in goodsList)
{
if (i == 50)
{
if (scope != null)
{
context.SubmitChanges();
}
i = 0;
}
try
{
foreach (GoodsEvent g in eventsList)
{
if (g.Gid == good.Gid)
{
ige = new InventoryGoodsEvents() { EventId = g.Id, InventoryGood = good.Id };
context.InventoryGoodsEvents.InsertOnSubmit(ige);
}
}
}
catch (Exception ex)
{
ex.ToString();
}
++i;
}
if (scope != null)
{
context.SubmitChanges();
scope.Complete();
}
}
the Foreach in a foreach can can be re-written as this-
var itsReallyJustAJoin = (from g in GoodEvent
join i in InventoryEvent on g.Gid equals i.Gid
select new {g.Id, good.Id}).ToList();
Then you can insert these however you'd like. It looks like you're doing batches of 50 right now? You can continue that if you'd like... This will take all of the work outside of the transaction scope though, so you can probably lob the whole bit in there at once, if your results aren't too huge.
Is it important that all of the inserts be done in the same transaction? If not, could you just use a new transaction for each batch of items you're inserting? That should prevent the transaction from being open for too long and timing out.
I have re-written the method after giving it a good thought and it turned out like this:
public void InsertNewInventoryGoodsEvents(List<GoodsEvent> eventsList, List<InventoryGoods> goodsList)
{
List<InventoryGoodsEvents> mylist = new List<InventoryGoodsEvents>();
InventoryGoodsEvents ige = new InventoryGoodsEvents();
foreach (GoodsEvent g in eventsList)
{
foreach (InventoryGoods ig in goodsList)
{
if (ig.Gid == g.Gid)
{
ige = new InventoryGoodsEvents();
ige.EventId = g.Id;
ige.InventoryGood = ig.Id;
mylist.Add(ige);
}
}
}
using (var scope = new TransactionScope())
{
foreach (InventoryGoodsEvents ip in mylist)
{
context.InventoryGoodsEvents.InsertOnSubmit(ip);
}
context.SubmitChanges();
scope.Complete();
}
}
I liked MikeM's idea of simplifying things but we'll see which method I'll keep.