Continue SSIS package when One Connection string is not reachable - ssis

I have a strange situation. I have an SSIS package and this package takes connection string from a table in SQL server something like
(Data Source=XYZ;Initial Catalog=Mail;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;) and there are many more connection string like this.
It uses this connection string to insert records to the desired SQL server. But, whenever a server is not reachable the SSIS package fails and stops execution.
I want to know how we can continue SSIS package if a particular server fails so that it doesn't effect other servers. Any help or comment is appreciated.

I assume the current package looks something like
Execute SQL Task (Get master list of connection strings)
ForEach (recordset) Enumerator (Assign current connection string to Variable)
Execute SQL Task (Inserts into table)
For each record you find, you assign that to a variable which is then used to drive the ConnectionString property of an OLE DB Connection manager.
Assuming that approximates the problem, you would need to add a precursor step to #3 which tests the validity of the connection string/manager. In SSIS, this mostly commonly be implemented through a Script Task. Rather than deal with Failing the Script Task, I'd also create an SSIS variable called IsConnectionValid and the result of the Script Task will be to set that to True or False depending on the state of the connection.
Psuedologic
Assumes Read only Collection is our variable #[User::ConnectionString]
Assumes Read/Write collection is our variable #[User::IsConnectionValid]
Assumes I can code without opening a text editor but the logic is sound
// re-initialize our state to false
Dts.Variables["User::IsConnectionValid"].Value = false;
// Access the current connection string
string cs = Dts.Variables["User::ConnectionString"].Value.ToString();
try
{
using (System.Data.SqlClient.SqlConnection conn = new System.Data.SqlClient.SqlConnection(cs))
{
conn.Open();
// If the connection didn't blow chow, then it's good *right now*
// The server could fall over in the next instant but this is the best we can do
Dts.Variables["User::IsConnectionValid"].Value = true;
}
}
catch(Exception ex)
{
// Swallow the error
;
}
Now that we have this script task running and safely evaluating our connection, then the last step is to change the Precedence Constraint between this new step 3 (script task) and the old step 3 (Execute SQL Task). Double click the connecting line and change it from the current default of OnSuccess to PrecedentAndConstraint (name approximate)
The constraint is simply #[User::IsConnectionValid] and it remains an AND conditional. This means that it will only give the Execute SQL Task the signal to fire if the variable is true and the preceding task didn't fail.

Related

SSIS ForEach Loop Container - How to dynamically change OLEDB Destination connection at run time

There are 4 Connection strings with different SQL Servers (which I set up in SSIS Connection Managers section):
Database name is same in all the servers:
SERVER DATABASE
dbTestServer dbFees (Main Server and Database)
dbTestServer1 dbFees1
dbTestServer2 dbFees1
dbTestServer3 dbFees1
dbTestServer is the OLEDB Source and other Servers are OLEDB Destination that needs to be updated everytime we run package.
Now, I want to take data from dbTestServer-dbFees and copy to all the other databases. I created a Dataflow task to copy data from dbTestServer to dbTestServer1.
But I need to put this data flow task inside ForEach Loop container to change the connection/Server dynamically so that it will work like:
First run- By default OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer1 and data is copied from dbFees to dbFees1.
Second run- OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer2 and data is copied from dbFees to dbFees1
Third run- OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer3 and data is copied from dbFees to dbFees1.
I need step by step solution as I am new to SSIS packages and I tried multiple solutions but NOTHING worked so far!
Appreciate your help!
Thank you
I suggest using FOR LOOP.
My logic is to increment variable on each loop and create an expression with a connection string and a number of iteration.
1st step is to create Connection Manager with server name dbTestServer1 and database name dbFees1
2nd step will be to add a connection manager to OLE DB Destination
3rd step is to create 2 variables: ConnString and Iteration.
For Iteration default value set to 1, because you need dbTestServer 1
ConnString you need to set like your initial connection string, just on place 1 in dbTestServer1 to set (DT_STR, 1, 65001)#[User::Iteration].
Like on next 2 pictures:
When you set variables, you need to set expression in OLE DB Connection Manager.
From drop-down select connection string and type #[User::ConnString].
And finally set FOR LOOP like on picture
NOTE: I can't test package because I don't have server names like you, but this is logic of how to solve your problem. And this is only solution for what you asked, you must create whole package on your own.
For main server and database, just add one OLE DB Source with static names for server name and database name.
And you don't need script task if you using my logic.
Here is the code I have used to dynamically change connection server/database inside C# Script task in SSIS:
Variables I pass to the C# Script task under ReadOnlyVariables:
(set these up in your Variables inside SSIS)
User::DatabaseListOnThisLoop_ConnectionString
User::DatabaseListOnThisLoop_DatabaseName
This is the name of the connection string I am dynamically change that is in my ConnectionMangers in SSIS:
SourceServerDBForClassification_Dynamic
FULL SCRIPT from my C# Script task inside SSIS. As long as you setup the variables and put the 2 in above in the ReadOnly section of the script task, you should be able to just copy/paste the entire code below into your C# Script task.
NOTE: The Namespace may give you an issue so may want to keep the one that is generated in your code when adding the script task.
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms; // dont think this is needed, I used this for message box for some testing, but leaving here just in case
namespace ST_f8d6dad17af541bbb0010c9fce3ccbb0
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
// get connection string from variable
string ServerConnectionStringOnThisLoop = Dts.Variables["DatabaseListOnThisLoop_ConnectionString"].Value.ToString();
string DatabaseOnThisLoop = Dts.Variables["DatabaseListOnThisLoop_DatabaseName"].Value.ToString();
// this could change depend on what type of connection you are using for provider and other settings
string DynamicConnectionString = "Data Source=" + ServerConnectionStringOnThisLoop + ";Initial Catalog=" + DatabaseOnThisLoop + ";Provider=SQLNCLI11.1;Integrated Security=SSPI;";
// Add the OLE DB connection manager set to existing connection
ConnectionManager SourceServerDBForClassification_Dynamic = Dts.Connections["SourceServerDBForClassification_Dynamic"];
// now set the dynamic connection above to the connection string passed in from SSIS package
SourceServerDBForClassification_Dynamic.ConnectionString = DynamicConnectionString;
// now set the package connection to the one we just created from using the variable from the SSIS package
Dts.Connections["SourceServerDBForClassification_Dynamic"].ConnectionString = SourceServerDBForClassification_Dynamic.ConnectionString;
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}

Unable to use expression on excel connection manager in SSIS 2017

I'm trying to loop through excel files in a directory and perform a data flow task in SSIS.
The For-Each Loop container seems pretty simple to set up:
I map to a variable called FileNameTemp.
Inside the For-Each Loop, I have a data flow task where the source object is an Excel Source with an Excel Connection Manager. I use the FileName temp to set the File Name of the ExcelFileName:
My problem is whenever I try to run the package, I get the error below:
[Connection manager "Excel Connection Manager"] Error: SSIS Error Code
DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code:
0x80004005. An OLE DB record is available. Source: "Microsoft Access
Database Engine" Hresult: 0x80004005 Description: "Failure creating
file.".
I found other similar posts. I definitely have permission to write to this folder. If I remove the expression and just open the same file over and over it works. I also set DelayValidation to true on pretty much every level.
Try removing the "C:..." from your expression definition. The For-Each file enumerator will give the full path.
In the future you can set a breakpoint on your data flow task and view the value of your variable that you set in the locals tab.
Same answer, just more verbose than #mike Baron's answer is that in the ForEach Loop Container, the radio button is checked for "Fully Qualified" with the result pushed into our variable #[User::FileNameTemp]
Each file found in the specified source folder C:\SourceCode\ExcelSourceFinancialReconcilliation is in turn going to be assigned to that variable in the form of
C:\SourceCode\ExcelSourceFinancialReconcilliation\file1.txt
C:\SourceCode\ExcelSourceFinancialReconcilliation\file2.csv
C:\SourceCode\ExcelSourceFinancialReconcilliation\file2.xls
Then, when we set the Expression on the Excel Connection Managers ExcelFilePath property, we need to just use #[User::FileNameTemp] As it stands, the expression is doubling up the path so that Excel is attempting to find
C:\SourceCode\ExcelSourceFinancialReconcilliation\file1.txt\C:\SourceCode\ExcelSourceFinancialReconcilliation\file1.txt
As a general rule, only use a direct variable in the Expressions associated to "objects" in SSIS. Property1 = #Variable The reason for this, is that you cannot put a break point to on the evaluation to determine why #Property1 = "Foo" + #Variable is invalid. If you create a custom variable #Property1Variable = "Foo" + #Variable and then assign #Property1 = #Property1Variable, you can put a breakpoint in the package and then inspect the value of the SSIS variable. It's much easier to find problems this way.
Possibly helpful other answers on the subject
https://stackoverflow.com/a/18640174/181965
https://stackoverflow.com/a/21536893/181965

SSIS Source SQL Statement to include variables

In my work place's SQL Server 2008 set up, we have set up a database table called DATE_RANGE_CONTROL with two columns: startDate and endDate of data type DateTime. Our SSIS packages extract data from an Oracle Database for a set date range so most of our SQL code in the DB Source Editor takes the form of:
SELECT COLUMN1,COLUMN2,COLUMN3,TRANSACTION_DATE
FROM SOURCE_DATABASE_TABLE
WHERE TRANSACTION_DATE BETWEEN startDate AND endDate
What I want to do is to set the startDate and endDate dynamically so that at the time of execution, the query gets the values from the DATE_RANGE_CONTROL table. I have looked at using the SQL Command from variable as an option but I cannot figure out how to feed the values of startDate and endDate from the DATE_RANGE_CONTROL table to the DB Source SQL command.
I have done something similar in the SQL SERVER 2000 DTS packages where I was setting the SQLCommand property of the data pump dynamically using the ACTIVEX script task. How do i do it in SSIS?
After spending a considerable amount of time on Google and looking through the Microsoft SQL Server 2008 SSIS documentation, I have finally cracked it. So proud of myself. The solution is thus:
In SQL Server Business Intelligence Development Studio, create a new SSIS package.
Create 3 variables, I called mine startDate, endDate and strDataSourceSQL. Make sure they all have a scope level of Package and ensure thay have Data Types of Int32, Int32 and String respectively..
On the Control Flow tab, drag a Script Task from the toolbox to the Control Flow pane. Right click it and choose edit from the pop up menu.
On the Script page, click on the ellipsis to the right of ReadWriteVariables. Tick the variables you created in step 1 i.e. user::startDate, user::endtDate and user::strDataSourceSQL. You can rename your script task on the General page. It is good practice.
On the Script page,click on the Edit Script... button to reveal a visual studio IDE pane with some pre-written code.
Make the following changes. At the top of the page, you will notice a few lines the import namespaces and classes to use. Edit the lines of code and make sure you have the following lines:
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Data.OleDb;
Go to the menu Project > Add Reference. On the .NET tab, make sure you select Microsoft.SqlServer.DTSPipelineWrap and Microsoft.SqlServer.DTSRuntimeWrap Credit to DotNetMonster for the tip.
In the public void Main() method, you will see a comment that says // TODO: Add your code hereWe will replace that line or add the following lines of code. By the way, I am using OLEDB for my data source and data destination.
//Declare variables
Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSConnectionManagerDatabaseParameters100 cmParams;
System.Data.OleDb.OleDbConnection oleDbConn;
ConnectionManager cm;
Int32 strStartDate;
Int32 strEndDate;
String startDateSQL = "SELECT StartDate FROM DATE_RANGE_CONTROL";
String endDateSQL = "SELECT EndDate FROM DATE_RANGE_CONTROL";
//Declare and assign Connection manager
cm = Dts.Connections["YOUR_OLEDB_CONNECTION_MANAGER_NAME"];
//Set Connection paramters
cmParams = cm.InnerObject as Microsoft.SqlServer.Dts.Runtime.Wrapper.IDTSConnectionManagerDatabaseParameters100;
//Instantiate the Oledb connection
oleDbConn = cmParams.GetConnectionForSchema() as OleDbConnection;
Assign the strStartDate and strEndDate variables, the result of running a function that we will display later. All it does provide an SQL command to the OLEDB connection.
strStartDate = (int)assignSQLCommand(startDateSQL,oleDbConn).ExecuteScalar();
strEndDate = (int)assignSQLCommand(endDateSQL, oleDbConn).ExecuteScalar();
//Assign the startDate and endDate variables the value of the SQL resultset
Dts.Variables["startDate"].Value = strStartDate;
Dts.Variables["endDate"].Value = strEndDate;
Declare a variable called dataSourceSQL of type string and assign your DataSource SQL code to it. Replace the date range values with the values of the variables above. For me, the code is as follows:
String dataSourceSQL = "SELECT COLUMN1,COLUMN2,COLUMN3,TRANSACTION_DATE
FROM SOURCE_DATABASE_TABLE
WHERE TRANSACTION_DATE BETWEEN '"+Dts.Variables["startDate"].Value +"' AND '"+Dts.Variables["endDate"].Value+"'";
Then assign the value of dataSourceSQL to your strDataSourceSQL package variable
Dts.Variables["strDataSourceSQL"].Value = dataSourceSQL ;
Outside the curly brackets of the public void Main() method, add the following function we discussed earlier.
public OleDbCommand assignSQLCommand(String SqlCommand, OleDbConnection oledbconn)
{
return new System.Data.OleDb.OleDbCommand(SqlCommand, oledbconn);
}
Go to the menu Build > Build [script name]. Save it and close the script page. Finally, all that is left is to create a Data Source task. Go to the Data Flow tab, drag an OLE DB DB Source to the pane. Right click on the OLE DB Source and choose edit. On the Connections Manager page, select your OLE DB connection manager and on the Data access mode choose SQL command from variable. In the variable name select user::strDataSourceSQL. As long as you set up your script task to run before your data flow, the job is done!
This is how I could do that.
In the OLE DB Source, enter your sql statement and enter ? as the parameter.
click [Parameters...] button and pick your variable from the list

ExcelConnection manager dynamic file path in SSIS

When i tried to give connection string in the form of a variable to "Excel Connection Manager", it gives me the below error.
the connection string format is not valid. it must consist of one or more components of the form x=y seperated by semicolons. This error occurs when a connection string with zero components is set on database connection manager.
Since you want a dynamic file path, when you are setting up the Expression for the Excel Connection Manager, you are probably selecting ConnectionString as a property in the Property Expressions Editor. This results in the error you specified in your question.
What you actually need to select is the ExcelFilePath property. Add your variable in the Expression field afterwards as you would normally do.
You should give us more information. What's the value of you variable when the error pops up? To exactly what property have you assigned this variable?
Anyway, I suspect that you didn't set [Delay validation] property of your connection manager to True - without it ssis check if you connection manager is ok, before you even assign value to the variable (which is dynamic and happens during execution in some loop, I suppose).

Seemingly getting wrong connection string from tabledef

Using Ms Access ...
I am trying to change the user to which I am connected to a SQL Server Database from Access. The startup form from my Access Application dynamically connects to the tables via an Altered version of a routine posted on Microsoft's KB. In Essence it deletes the tabledef that matches the name localName you call it with, Calls Currentdb.CreateTableDef to create a new one with the appropriate new connection string and appends it the the CurrentDb.TableDefs collection. As far as I can see, its the append function that actually contacts the database and throws an error if your connection string is wrong.
In my case, I did an original pass through all the tables in my database calling this routine with an old (standardised - ie I normally use this) string and all tables connect fine.
I am now trying a second pass through this start up form with a new user/password combination so as to test different permissions in the database.
I have then built a new connection string with a revised name and password and am currently single stepping through the function I just described.
I have a variable td which contains the newly created tabledef and I am just prior to appending it to the Currentdb.TableDefs collection
In the debugger "Immediate" window ?td.connect gives the connection string with the newly formed connection string correctly and ?td.name gives tblConfig as the table name.
Just to ensure the TableDef has been correctly deleted from the current collection I did ?CurrentDb.Tabledefs("tblConfig").Connect from the immediate window and was rewarded with a dialog box that said the collection didn't contain a member with that name
I then single step over the line
CurrentDb.TableDefs.Append td
And now call ?CurrentDb.Tabledefs("tblConfig").Connect again from the debuggers Immediate window and the response is the old connection string not the new one.
There is an OnError active in this routine, but It is not activated during the single stepping.
The only explanation I can think of for this, is that somehow the old connection string is stored on disk as a result of saving the database at some point and append doesn't copy across the connectionstring when it is finally appended.
Does anyone have a better explanation?
It turns out that the connection string is indeed cached
http://blogs.office.com/b/microsoft-access/archive/2011/04/08/power-tip-improve-the-security-of-database-connections.aspx
I had an incorrect password for the new connection string (one character was wrong) and it appears that access does not store the new username and password when that problem happens.
Somehow, despite deleting the old tabledef with the old username and password, Access was able to get it back - maybe it was cached, I don't know - and use it. Obviously that worked and so it carried on with that string rather than the new one.
It seems that because the old string worked, it didn't bother to throw an error, so I was now connected to the table with a server login that didn't have the same permissions as I expected.