I'm trying to loop through excel files in a directory and perform a data flow task in SSIS.
The For-Each Loop container seems pretty simple to set up:
I map to a variable called FileNameTemp.
Inside the For-Each Loop, I have a data flow task where the source object is an Excel Source with an Excel Connection Manager. I use the FileName temp to set the File Name of the ExcelFileName:
My problem is whenever I try to run the package, I get the error below:
[Connection manager "Excel Connection Manager"] Error: SSIS Error Code
DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code:
0x80004005. An OLE DB record is available. Source: "Microsoft Access
Database Engine" Hresult: 0x80004005 Description: "Failure creating
file.".
I found other similar posts. I definitely have permission to write to this folder. If I remove the expression and just open the same file over and over it works. I also set DelayValidation to true on pretty much every level.
Try removing the "C:..." from your expression definition. The For-Each file enumerator will give the full path.
In the future you can set a breakpoint on your data flow task and view the value of your variable that you set in the locals tab.
Same answer, just more verbose than #mike Baron's answer is that in the ForEach Loop Container, the radio button is checked for "Fully Qualified" with the result pushed into our variable #[User::FileNameTemp]
Each file found in the specified source folder C:\SourceCode\ExcelSourceFinancialReconcilliation is in turn going to be assigned to that variable in the form of
C:\SourceCode\ExcelSourceFinancialReconcilliation\file1.txt
C:\SourceCode\ExcelSourceFinancialReconcilliation\file2.csv
C:\SourceCode\ExcelSourceFinancialReconcilliation\file2.xls
Then, when we set the Expression on the Excel Connection Managers ExcelFilePath property, we need to just use #[User::FileNameTemp] As it stands, the expression is doubling up the path so that Excel is attempting to find
C:\SourceCode\ExcelSourceFinancialReconcilliation\file1.txt\C:\SourceCode\ExcelSourceFinancialReconcilliation\file1.txt
As a general rule, only use a direct variable in the Expressions associated to "objects" in SSIS. Property1 = #Variable The reason for this, is that you cannot put a break point to on the evaluation to determine why #Property1 = "Foo" + #Variable is invalid. If you create a custom variable #Property1Variable = "Foo" + #Variable and then assign #Property1 = #Property1Variable, you can put a breakpoint in the package and then inspect the value of the SSIS variable. It's much easier to find problems this way.
Possibly helpful other answers on the subject
https://stackoverflow.com/a/18640174/181965
https://stackoverflow.com/a/21536893/181965
Related
There are 4 Connection strings with different SQL Servers (which I set up in SSIS Connection Managers section):
Database name is same in all the servers:
SERVER DATABASE
dbTestServer dbFees (Main Server and Database)
dbTestServer1 dbFees1
dbTestServer2 dbFees1
dbTestServer3 dbFees1
dbTestServer is the OLEDB Source and other Servers are OLEDB Destination that needs to be updated everytime we run package.
Now, I want to take data from dbTestServer-dbFees and copy to all the other databases. I created a Dataflow task to copy data from dbTestServer to dbTestServer1.
But I need to put this data flow task inside ForEach Loop container to change the connection/Server dynamically so that it will work like:
First run- By default OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer1 and data is copied from dbFees to dbFees1.
Second run- OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer2 and data is copied from dbFees to dbFees1
Third run- OLEDB Source is set to dbTestServer and OLEDB Destination is set to dbServer3 and data is copied from dbFees to dbFees1.
I need step by step solution as I am new to SSIS packages and I tried multiple solutions but NOTHING worked so far!
Appreciate your help!
Thank you
I suggest using FOR LOOP.
My logic is to increment variable on each loop and create an expression with a connection string and a number of iteration.
1st step is to create Connection Manager with server name dbTestServer1 and database name dbFees1
2nd step will be to add a connection manager to OLE DB Destination
3rd step is to create 2 variables: ConnString and Iteration.
For Iteration default value set to 1, because you need dbTestServer 1
ConnString you need to set like your initial connection string, just on place 1 in dbTestServer1 to set (DT_STR, 1, 65001)#[User::Iteration].
Like on next 2 pictures:
When you set variables, you need to set expression in OLE DB Connection Manager.
From drop-down select connection string and type #[User::ConnString].
And finally set FOR LOOP like on picture
NOTE: I can't test package because I don't have server names like you, but this is logic of how to solve your problem. And this is only solution for what you asked, you must create whole package on your own.
For main server and database, just add one OLE DB Source with static names for server name and database name.
And you don't need script task if you using my logic.
Here is the code I have used to dynamically change connection server/database inside C# Script task in SSIS:
Variables I pass to the C# Script task under ReadOnlyVariables:
(set these up in your Variables inside SSIS)
User::DatabaseListOnThisLoop_ConnectionString
User::DatabaseListOnThisLoop_DatabaseName
This is the name of the connection string I am dynamically change that is in my ConnectionMangers in SSIS:
SourceServerDBForClassification_Dynamic
FULL SCRIPT from my C# Script task inside SSIS. As long as you setup the variables and put the 2 in above in the ReadOnly section of the script task, you should be able to just copy/paste the entire code below into your C# Script task.
NOTE: The Namespace may give you an issue so may want to keep the one that is generated in your code when adding the script task.
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms; // dont think this is needed, I used this for message box for some testing, but leaving here just in case
namespace ST_f8d6dad17af541bbb0010c9fce3ccbb0
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
// get connection string from variable
string ServerConnectionStringOnThisLoop = Dts.Variables["DatabaseListOnThisLoop_ConnectionString"].Value.ToString();
string DatabaseOnThisLoop = Dts.Variables["DatabaseListOnThisLoop_DatabaseName"].Value.ToString();
// this could change depend on what type of connection you are using for provider and other settings
string DynamicConnectionString = "Data Source=" + ServerConnectionStringOnThisLoop + ";Initial Catalog=" + DatabaseOnThisLoop + ";Provider=SQLNCLI11.1;Integrated Security=SSPI;";
// Add the OLE DB connection manager set to existing connection
ConnectionManager SourceServerDBForClassification_Dynamic = Dts.Connections["SourceServerDBForClassification_Dynamic"];
// now set the dynamic connection above to the connection string passed in from SSIS package
SourceServerDBForClassification_Dynamic.ConnectionString = DynamicConnectionString;
// now set the package connection to the one we just created from using the variable from the SSIS package
Dts.Connections["SourceServerDBForClassification_Dynamic"].ConnectionString = SourceServerDBForClassification_Dynamic.ConnectionString;
Dts.TaskResult = (int)ScriptResults.Success;
}
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
}
}
I have a strange situation. I have an SSIS package and this package takes connection string from a table in SQL server something like
(Data Source=XYZ;Initial Catalog=Mail;Provider=SQLNCLI11.1;Integrated Security=SSPI;Auto Translate=False;) and there are many more connection string like this.
It uses this connection string to insert records to the desired SQL server. But, whenever a server is not reachable the SSIS package fails and stops execution.
I want to know how we can continue SSIS package if a particular server fails so that it doesn't effect other servers. Any help or comment is appreciated.
I assume the current package looks something like
Execute SQL Task (Get master list of connection strings)
ForEach (recordset) Enumerator (Assign current connection string to Variable)
Execute SQL Task (Inserts into table)
For each record you find, you assign that to a variable which is then used to drive the ConnectionString property of an OLE DB Connection manager.
Assuming that approximates the problem, you would need to add a precursor step to #3 which tests the validity of the connection string/manager. In SSIS, this mostly commonly be implemented through a Script Task. Rather than deal with Failing the Script Task, I'd also create an SSIS variable called IsConnectionValid and the result of the Script Task will be to set that to True or False depending on the state of the connection.
Psuedologic
Assumes Read only Collection is our variable #[User::ConnectionString]
Assumes Read/Write collection is our variable #[User::IsConnectionValid]
Assumes I can code without opening a text editor but the logic is sound
// re-initialize our state to false
Dts.Variables["User::IsConnectionValid"].Value = false;
// Access the current connection string
string cs = Dts.Variables["User::ConnectionString"].Value.ToString();
try
{
using (System.Data.SqlClient.SqlConnection conn = new System.Data.SqlClient.SqlConnection(cs))
{
conn.Open();
// If the connection didn't blow chow, then it's good *right now*
// The server could fall over in the next instant but this is the best we can do
Dts.Variables["User::IsConnectionValid"].Value = true;
}
}
catch(Exception ex)
{
// Swallow the error
;
}
Now that we have this script task running and safely evaluating our connection, then the last step is to change the Precedence Constraint between this new step 3 (script task) and the old step 3 (Execute SQL Task). Double click the connecting line and change it from the current default of OnSuccess to PrecedentAndConstraint (name approximate)
The constraint is simply #[User::IsConnectionValid] and it remains an AND conditional. This means that it will only give the Execute SQL Task the signal to fire if the variable is true and the preceding task didn't fail.
Is it possible to read all rows from Excel sheet through Execute SQL Task from SSIS and reading each value in for loop container ?
You probably could and save the output to a variable, which you can use in the loop container. There may be a gotcha with permissions and/or linked server setup.
Here's another approach:
Create Data Flow
Create Data Connection to Excel file
Create Excel file source transformaion
Use Recordset Destination to populate a variable
Use the variable in your loop, setting Enumerator property to Foreach ADO Enumerator
When i tried to give connection string in the form of a variable to "Excel Connection Manager", it gives me the below error.
the connection string format is not valid. it must consist of one or more components of the form x=y seperated by semicolons. This error occurs when a connection string with zero components is set on database connection manager.
Since you want a dynamic file path, when you are setting up the Expression for the Excel Connection Manager, you are probably selecting ConnectionString as a property in the Property Expressions Editor. This results in the error you specified in your question.
What you actually need to select is the ExcelFilePath property. Add your variable in the Expression field afterwards as you would normally do.
You should give us more information. What's the value of you variable when the error pops up? To exactly what property have you assigned this variable?
Anyway, I suspect that you didn't set [Delay validation] property of your connection manager to True - without it ssis check if you connection manager is ok, before you even assign value to the variable (which is dynamic and happens during execution in some loop, I suppose).
Newbie to SSIS. I want to read an Excel spreadsheet, get a value, assign it to a package variable, and then use the variable in a SQL statement. I can read the excel and get the value, cannot figure out how to put the value into a variable (from the Excel source) so the SQL will run.
Easiest thing would probably be to have the excel source point to a RecordSet Destination. You can then Foreach-loop the object holding the recordset and extract the value(s) you want to work with that originated from Excel.