Task: Loop thru these excel files and insert data into SQL table but in the process i get an error and i don't know which it errored on.
My understanding is SSIS doesn't loop thru file in an random order but i get an error about CANNOT ACQUIRE CONNECTION FROM CONNECTIONMANAGER. Excel Source failed validation and returned error code.. I did set 64bitruntime to False. This happened on VS 2008/SQL Server 2008 R2 on Windows 7 OS. Initially i was able to run the whole process successfully on Windows XP- VS2008 /SQL Server 2008 R2.
Problem: How do i know which file system is going to iterate next if i have 70 odd files in a folder. The thing i get an error and i'm not sure which file SSIS is working on. However i do see files are executed and data is in SQL.
Let me know how to find which file SSIS is currently working or the next one it will work on.
You could add a Script Task and log the variable used in the foreach loop.
Add the variable as readonly variable in the script task editor and then add something like this in the main method (C#):
public void Main()
{
bool fireAgain = true;
Dts.Events.FireInformation(0, "Logging FELC variable", "File: " + Dts.Variables["User::FilePath"].Value.ToString(), string.Empty, 0, ref fireAgain);
Add a script task inside your ForEach container, immediately before you do the Excel processing. In the script task, add the variable you configured in your ForEach loop to hold the filename to the Read Only Variables. In the script itself, call the FireInformation event, which will add an informational message to the progress log in SSIS. In the FireInformation call, pass the value of your filename variable as the message argument.
This will let you see each file being processed, and which one it was processing when it failed.
FireInformation help: http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.dts.runtime.idtscomponentevents.fireinformation.aspx
Related
I have struggled and struggled with this issue. I've read a many similar questions concerning this topic, but I'm new at this and I'm reading "Greek to me". Perhaps someone will be willing to help me on an elementary level with this. I created an SSIS package with several steps. The first step is a script task that uses a VB8 script to pull the data from the SFTP server. The script reads like this:
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime
<System.AddIn.AddIn("ScriptMain", Version:="1.0", Publisher:="", Description:="")> _
<System.CLSCompliantAttribute(False)> _
Partial Public Class ScriptMain
Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
Enum ScriptResults
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
End Enum
Public Sub HealthSmart()
Dim Path As String, Path2 As String ', fileName As String, fo As Object, x As Object, y As Object, z As Object, rs As Object
Path = "\\RENYSQL1\share\Health_Smart\Received_Files\Test"
Path2 = "\\RENYMstr1\shared\CLIENT FOLDERS\HEALTHSMART\Original_PDF_Files\Test"
Shell("""C:\Program Files (x86)\Ipswitch\WS_FTP 12\wsftppro.exe"" -s ""Health_Smart:/Test/AMC_Test/*.pdf"" -d ""local:" & Path & "\"" -quiet", vbMaximizedFocus)
Shell("""C:\Program Files (x86)\Ipswitch\WS_FTP 12\wsftppro.exe"" -s ""Health_Smart:/Test/AMC_Test/*.pdf"" -d ""local:" & Path2 & "\"" -quiet", vbMaximizedFocus)
Dts.TaskResult = ScriptResults.Success
End Sub
End Class
It runs perfectly fine when the step is executed in the SSIS package. It also runs without error when I import the SSIS package into MSDB and select run package. But when I scedule it as a step in a job, it acts as if it ran perfectly. But the first step pulled not data.
I have tried setting the permissions of the SSIS package to a user that has full permissions to everything. I also tried setting the package protection level to DontSaveSensitive with no password.
On the SQL job side, I have tried using Windows authentication in the job as well as the login I mentioned earlier to run the package. Neither worked. Furthermore, I changed it to run on 32 bit - but to no avail.
I understand that the SQL job is run by the SQL agent. Does the agent have a specific login? People in my department say no.
The strange thing is that when I schedule the job to run, it runs and says it is successful. But the first step of getting the SFTP files with the above script runs, but does not pull any data. The Job Activity Monitor says it was Successful.
I have no idea what to do next. Be easy on me. I'm new at this.
While you certainly could have permissions issues if you are not receiving and error it would suggest the code is not executing. Looking at where you placed your code in your script I am thinking it is entirely possible that you did not put a call to your SUB in the Main () Sub. When creating a new script you should see something like:
Public Sub Main()
'
' Add your code here
'
Dts.TaskResult = ScriptResults.Success
End Sub
And basically where it says add your code here is where Microsoft was leading you to add your script. It is completely okay to do it in its own sub as you have, but if you do you need to call your sub within the Main() SUB like so:
Public Sub Main()
'
' Add your code here
'
HealthSmart()
Dts.TaskResult = ScriptResults.Success
End Sub
If you actually do have the Main SUB and calling your code. There could perhaps be an error within wsftppro.exe. Have you run the code outside of the script?
If permissions are causing the issue within the exe and not error you could try running via your dev environment as a user that has access to the locations and if it succeeds for you then you could need permissions for your SQL Agent and/or Service Account.
I am implementing a custom auditing framework, logging ETL events such as start, end, error, insertrows etc.
As well as logging at a package level, I'm implementing "session logging" where a sequence of package executions, i.e. a controller package that executes several packages, is a session. In order to keep track of the "session", the stored procedures always return a SessionLogID.
I was hoping I could map this result set to a project parameter as otherwise, I will have to save it to a user var and then pass it around between packages via parameters. This will mean every single package will have a Package Parameter and User Variable called SessionLogID. I don't want to do this if I don't need to.
Open to other suggestions.
Thanks,
Adam
Parameters cannot change at runtime. They are a set once kind of deal whereas variables can change at any time. You can set the variable once in the parent package and map the variable to the child package's using a parameter.
I have a very simple SSIS package and an absurdly simple Script task
var x = Dts.Variables["User::OrderPath"];
That fails with:
The element cannot be found in a collection. This error happens when
you try to retrieve an element from a collection on a container during
execution of the package and the element is not there.
I have a variable, OrderPath, that is scoped to the package. Any time I try to add the variable to the script's ReadOnlyVariables, it disappears whenever I execute the task.
This really shouldn't be this difficult so I assume I'm missing something monumentally basic. Any idea what that might be?
When accessing variables through the Dts.Variables collection, you do not prefix the variable name with the namespace; thus
var x = Dts.Variables["User::OrderPath"];
fails, while
var x = Dts.Variables["OrderPath"];
succeeds. (Assuming, of course, that OrderPath is added to either the ReadWriteVariables or ReadOnlyVariables task properties.)
See Using Variables in the Script Task in MSDN for another example.
Creating a package that will loop through all the servers in our different environments(DEV, UAT, PROD). We use service accounts and there is one service account for DEV and UAT and another for PROD. I am using a ForEach Loop Container/variables to set the connection string.
The issue: When the data flow in the loop trys to connect to the PROD servers they fail because they are using the DEV/UAT service account which obviously doesnt have access to PROD, which is fine. The problem is this kills the loop. Normally I would just put an event handler on the data flow and set the event handler's System Variable Propagate:OnError = False so that the error doesn't bubble up to the loop and kill it. Well this doesn't work because the OLE DB connection inside the data flow fails during validation(while the package is running) and apparently the Propagate = False setting only keeps an error from bubbling up if it occurs during the execution of a task and not the validation of a task.
I can set the MaximumErrorCount = 0 on everything including the package itself but that is a bit heavy handed and the package would always report that it ran successfully no matter what errors were encountered.
Running SQL Server 2008-R2
Link to an article on how to not kill your loop using the Propagate setting if someone isn't familiar with it.
One suggestion would be to put a Script Task before the Data Flow tasks that checks access to the connection string with a try-catch block and sets a variable upon failure, and then use that variable in a conditional split to determine whether to run the Data Flow or log that the connection string failed.
Alternatively, if you don't care about why it failed (since you already know it's because of permissions) you could just use a Precedence Constraint and only run Data Flows where the connection succeeded.
UPDATE:
Here's some working code for the Script Task:
public void Main() {
string connString = Dts.Variables["ConnectionStringToTest"].Value;
try {
using (OleDbConnection connection = new OleDbConnection()) {
connection.ConnectionString = connString;
connection.Open();
}
Dts.Variables["User::DatabaseCanConnect"].Value = true;
}
catch (Exception ex) {
Dts.Variables["User::DatabaseCanConnect"].Value = false;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
Create a variable called DatabaseCanConnect at the package scope. Set it to Boolean, it'll default to False.
Create a Script Task. Edit the ReadWriteVariables property and add your new variable.
Add to your ReadOnlyVariables whatever variable you're using to build the connection string out of your ForEach loop. I've named mine ConnectionStringToTest.
Add the script code above as your Main() function. Note that this using an OleDbConnection. This should mimic whatever connection you're using for the connection manager you're modifying in your data flow. So if it's a SQL connection, use a SqlConnection instead.
In your code, use the DatabaseCanConnect variable to determine flow from here. You can use it in a Precedence Constraint to prevent flow to the Data Flow, or use it in a Conditional Split (my preference, more visible to other developers) to log connection errors on failure and proceed as normal otherwise.
I am making a framework in SSIS to load files from configurable folders and match them to a logical job in the database. In this job a package name is configured and in SSIS I execute this package in runtime.
I want to programmatically load a package configuration for this package, depending on the job loaded. SSIS SQL Server package configuration is not an option, because that loads values to this package just once in runtime for the package itself, but I want to load a specific package configuration in runtime that has been stored with the job (job has one package, but has many package configurations)....
Schematically:
folderA -> file A.1 -> job A -> load package configuration for job A -> execute package in job A.
Is that possible?
We do something simliar using parent and child packages to run a standard package for differnt clients with differnt configuration values. The parent packge uses and enviroment variable and our configuration table to pull the configuration values for that particular process. The child table is configured to accept variables for the configuration which are sent from the parent package inthe execute package task. This also allows us to do some custom steps for a particular client in the parent package if need be (which is about 100% of the time here). So of you get one file form one client that they just cannot provide in the format the standard child import uses you can do transformation steps to get teh file ready for the standard import and then run the standard. Or you can add steps after the standrd package to send an email to the client with exceptions that they need to fix in their data for instance if only one client requires that.
You create Variables in the parent package for each piece of configuration information you want to send, typically to other variables or connection strings for the conmnections in the child package. You then put in an Excute package task that uses a connection to the child package.
In the child package you then go to the SSIS menu and choose package configurations and Add. Then for the type of configuration, you choose Parent Package variable. You will create one Parent package variable for each configuration item you want to send to the Child package. Things we send are things like the client_id, the connection strings to a client specific database, variables for things that might vary by client, etc.
We also store all our configurations in a table in a meta database where we store information about imports. So we set up our parent pacakge to use an environment variable to tell it which database to connect to to get the configuration information Then the second confiuration is to the SSISConfiguration table that stores the configuration information. We populate that information by server (it will vary by server generally, connection strings are different for dev, qa and prod) through an insert script that we run before testing the package.
For further detail, look in Books Online for execute package task and it wil show you how to set up the packages to pass variables.
I found the solution now. It is only possible by using a script task that uses the SSIS object model to create a package in runtime based on the SQL Server Application class where you can load the package by filename. After loading the package from file, I can read the configuration from file by xml or by SQL Server and add it in runtime to the child package configuration list.
Two important notes:
1) Parent variables are not passed to child package automatically.
Only when an execute package task is used the parent variables are passed to the child automatically. To get this working I search the variables in runtime and write the values in it, because I know the exact variables I want to pass to each child package.
2) When using SQL Server as a package configuration for a child package, you must also create a connection manager in runtime and add it to the connection manager collection of the package. when adding the package configuration to the child package, be sure that the name of that connection manager is part of the connection string.
Here is the code to prove it works:
//load the information of the job into these variables. Package is the File system deployed package on a share. Package configuration can be the package configuration in an xml file on a share, or a connection string when using SQL Server (this one is used here).
string package = this.Dts.Variables["Package"].Value.ToString();
string packageConfiguration = this.Dts.Variables["PackageConfiguration"].Value.ToString();
//create a package from package factory, by file.
Microsoft.SqlServer.Dts.Runtime.Application app = new Microsoft.SqlServer.Dts.Runtime.Application();
Package packageToRun = app.LoadPackage(package, null);
//------------------------------------------ CHILD PACKAGE VARIABLES PASSING
packageToRun.EnableConfigurations = true;
//add one extra package configuration for child package specific configuration
Configuration config = packageToRun.Configurations.Add();
config.Name = "MyConfig";
config.ConfigurationType = DTSConfigurationType.SqlServer;
config.ConfigurationString = packageConfiguration;
//use the name 'MyConnectionManager' in your packageConfiguration
ConnectionManager cm = packageToRun.Connections.Add("OleDb");
cm.Name = "MyConnectionManager";
//TODO: retrieve this from an environvariable to allow change in data source for DEV, QA, PROD, now temporarly fixed to this value
cm.ConnectionString = "Data Source=.;Initial Catalog=YYYYYYYYYY;Provider=SQLNCLI10.1;Integrated Security=SSPI;";
//For Parent-Child var passing, I used the technique to let all the parent variables being defined in the child packages.
//Other technique could be to allow the child package not define the parent variables, but then the child packages have to reference them from code
//------------------------------------------ PARENT VARIABLES PASSING
//Now check if these parent variables exist in child package and write the actual values in them
try
{
Variables vars = null;
VariableDispenser variableDispenser = packageToRun.VariableDispenser;
if (
packageToRun.Variables.Contains("User::XXXXXXXXXXXX") &&
)
{
packageToRun.VariableDispenser.LockForWrite("User::XXXXXXXXXXXX");
variableDispenser.GetVariables(ref vars);
packageToRun.Variables["User::XXXXXXXXXXXX"].Value = this.Dts.Variables["User::XXXXXXXXXXXX"].Value;
vars.Unlock();
packageToRun.Execute();
Dts.TaskResult = (int)ScriptResults.Success;
}
else
{
this.Dts.Events.FireError(0, string.Empty, "Child package: " + package + " has no required master variables defined or unable to unlock.", string.Empty, 0);
}
}
catch (Exception ex)
{
this.Dts.Events.FireError(0, string.Empty, ex.Message, string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}