I am trying to access the variable RowCount. However, I can't do :
since its assign value as 0 to row count even if have a result set
var rowCount=this.Variables.RowCount;
Where are you trying to access this variable, and are you trying to update it? SSIS variables can only be written to in the PostExecute method. To do this, start by adding the variable to the ReadWriteVariables field of the script component editor, then you can access this as done below.
Your variable is named rowCount. Are you looking to return the number of rows that go through the Data Flow Task? A Script Component is called once for each record in a Data Flow Task. To get the total number of records, use Row Count transformation instead and assign the variable to the result of this.
int rowCount;
public override void PreExecute()
{
base.PreExecute();
//get variable value before processing rows.
rowCount = Variables.RowCount;
}
public override void PostExecute()
{
base.PostExecute();
//update variable after records have been procssed
Variables.RowCount = rowCount;
}
In my project.params I have variables such as LampUserName of type string. In my
script task I'm then trying to read them like so:
foreach (var name in new string[] { "ServerName", "InitialCatalog", "UserName", "Password" }) {
if (!Dts.Variables.Contains(string.Format("$Project::Lamp{0}", name))) {
writer.WriteLine(name);
}
}
string server_name = (string)Dts.Variables["$Project::LampServerName"].Value;
string database = (string)Dts.Variables["$Project::LampInitialCatalog"].Value;
string username = (string)Dts.Variables["$Project::LampUserName"].Value;
string password = (string)Dts.Variables["$Project::LampPassword"].Value;
Every one of those prints out that it doesn't exist and then an exception is thrown. What am I doing wrong?
Are you referencing your project parameters from a Script Component within a Data Flow Task? If so, this differs from using a Script Task within the Control Flow. After adding the project parameter in the ReadOnlyVaraiables field, it is accessible as a property of the Variable object.
string database = Variables.DatabaseParameter.ToString();
I have developed one master package(Main.dtsx) and 3 child packages (Processor.dtsx).Note: Code is same for all child packages that picks up files from source location and process. To optimize the performance, I want that all these 3 child packages should run simultaneously on 10000 files in such a way that first child will pick 1st file and start execution , at the same time second will pick up 2nd file and so on. Please share the code if you have. I tried with 'MaxConcurrentExecutables' option but in that case all components access same file which is not expected.
This cannot be done with a Foreach Loop, but you can accomplish the task with a Script task:
Add 3 string variables to hold the file names (i.e. File1, File2, File3)
Pass the variables from the master package to each child package.
In each child package, configure an expression in the file connection manager to use the parameter as a connection string
At the end of each package, make sure that the file is moved from the source folder or renamed in such a way that it will be ignored in subsequent loops.
Set up a For loop that will end when all the files have been processed. You can add a boolean variable to the package like "ProcessingIsAllDone" and then set this in the script task.
At the top of the For loop add a script task and connect the execute package tasks with precedent constraints.
Use the script below to set the variables
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
namespace ST_e4ccd9cfaa4847ff86ec88c215c1961c
{
[Microsoft.SqlServer.Dts.Tasks.ScriptTask.SSISScriptTaskEntryPointAttribute]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
public void Main()
{
DirectoryInfo sourceDirectory = new DirectoryInfo(#"c:\temp");
int loops = 3;
foreach (FileInfo sourceFile in sourceDirectory.GetFiles("*.txt"))
{
if (loops == 0)
{
break;
}
string variableName = String.Format("File{0}", loops);
Dts.Variables[variableName].Value = sourceFile.FullName;
loops--;
}
if (sourceDirectory.GetFiles("*.txt").Length <= 3)
{
Dts.Variables["ProcessingIsAllDone"].Value = true;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
#region ScriptResults declaration
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
}
}
I am redirecting rows from a flat file source to a flat file destination. The default metadata in the redirected rows are:
The original flat file source row
The ErrorCode
The ErrorColumn
What I get in the output flat file is the source row (nice) and the error code (not nice, ex. -1071628249) and the error column (not nice since it's the internal ID of the column).
How can I transform the rows to output the error message (e.g. "The data was truncated.") and the column name as defined in the flat file source?
In other words, instead of ...,-1071607675,10 I'd like to see:
...,The data was truncated,Firstname
or alternatively (if the previous is not possible);
...,DTS_E_FLATFILESOURCEADAPTERSTATIC_TRUNCATED,Firstname.
Error message list is in the following location:
MSDN, Integration Services Error and Message Reference
https://learn.microsoft.com/en-us/sql/integration-services/integration-services-error-and-message-reference?view=sql-server-ver15
And column Id Number can be found in SSIS's Data Flow Task:
select the task component that generates the error, Advanced Editor, 'Input and Output Properties' tab, External Columns Properties.
Part of the question (adding the error description) can be achieved with a script component. This is described in Enhancing an Error Output with the Script Component.
It seems that the Dougbert blog has a solution to adding the column name, but it's far from simple. I'm amazed this is so difficult to do in SSIS; you'd think it was a basic need to know the name of the source and column.
There is a far simpler answer. Simply redirect the error output to a new destination file (CSV or whatever) and then enable a DataViewer on the error output....
It can be achieved using script component as transformation, Redirect error output to the script component and follow the steps to achieve what you are looking for.
(1) Open script component ,
Input Columns select
ErrorColumn
ErrorCode
Input and Output add Output columns
ErrorDescription (DT_STRING 500)
ErrorColumnDescription (DT_STRING 100)
(2) Edit Script
Paste the following code
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
#endregion
/// <summary>
/// This is the class to which to add your code. Do not change the name, attributes, or parent
/// of this class.
/// </summary>
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
var component130 = this.ComponentMetaData as IDTSComponentMetaData130;
if (component130 != null)
{
Row.ErrorDescription = component130.GetErrorDescription(Row.ErrorCode);
Row.ErrorColumnDescription = component130.GetIdentificationStringByID(Row.ErrorColumn);
}
}
Pragmatic Works appears to have an Error Output Description Transform that is a part of the Community Edition (Free) of the Product they call "Task Factory".
The Error Output Description Transform provides the user with a User Interface that can retrieve valuable information such as the ErrorCode, ErrorColumn, ErrorDescription, ComponentName (that generated the error), ColumnName (if known), ColumnType, and ColumnLength.
It also allows you to pass through any input columns to the Error Output. To be honest it is quite handy and has saved me hours of time troubleshooting my SSIS Packages.
//column error description
Row.ErrorDescription = this.ComponentMetaData.GetErrorDescription(Row.ErrorCode);
//we are getting column name with some extra information
String rawColumnName = this.ComponentMetaData.GetIdentificationStringByLineageID(Row.ErrorColumn);
//extracting positions of brackets
int bracketPos = rawColumnName.LastIndexOf('[')+1;
int lastBracketPos = rawColumnName.LastIndexOf(']');
//extracting column name from the raw column name
Row.ErrorColName = rawColumnName.Substring(bracketPos, (lastBracketPos - bracketPos));
Using SS2016 and above, it is easy:
https://www.mssqltips.com/sqlservertip/4066/retrieve-the-column-causing-an-error-in-sql-server-integration-services/
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
Row.ErrorDescription = this.ComponentMetaData.GetErrorDescription(Row.ErrorCode);
IDTSComponentMetaData130 componentMetaData = this.ComponentMetaData as IDTSComponentMetaData130;
Row.ErrorColumnName = componentMetaData.GetIdentificationStringByID(Row.ErrorColumn);
}
For anyone using SQL Server versions before SS2016, here are a couple of reference links for a way to get the Column name:
http://www.andrewleesmith.co.uk/2017/02/24/finding-the-column-name-of-an-ssis-error-output-error-column-id/
which is based on:
http://toddmcdermid.blogspot.com/2016/04/finding-column-name-for-errorcolumn.html
I appreciate we aren't supposed to just post links, but this solution is quite convoluted, and I've tried to summarise by pulling info from both Todd and Andrew's blog posts and recreating them here. (thank you to both if you ever read this!)
From Todd's page:
Go to the "Inputs and Outputs" page, and select the "Output 0" node.
Change the "SynchronousInputID" property to "None". (This changes
the script from synchronous to asynchronous.)
On the same page, open the "Output 0" node and select the "Output
Columns" folder. Press the "Add Column" button. Change the "Name"
property of this new column to "LineageID".
Press the "Add Column" button again, and change the "DataType"
property to "Unicode string [DT_WSTR]", and change the "Name"
property to "ColumnName".
Go to the "Script" page, and press the "Edit Script" button. Copy
and paste this code into the ScriptMain class (you can delete all
other method stubs):
public override void CreateNewOutputRows() {
IDTSInput100 input = this.ComponentMetaData.InputCollection[0];
if (input != null)
{
IDTSVirtualInput100 vInput = input.GetVirtualInput();
if (vInput != null)
{
foreach (IDTSVirtualInputColumn100 vInputColumn in vInput.VirtualInputColumnCollection)
{
Output0Buffer.AddRow();
Output0Buffer.LineageID = vInputColumn.LineageID;
Output0Buffer.ColumnName = vInputColumn.Name;
}
}
} }
Feel free to attach a dummy output to that script, with a data viewer,
and see what you get. From here, it's "standard engineering" for you
ETL gurus. Simply merge join the error output of the failing
component with this metadata, and you'll be able to transform the
ErrorColumn number into a meaningful column name.
But for those of you that do want to understand what the above script
is doing:
It's getting the "first" (and only) input attached to the script
component.
It's getting the virtual input related to the input. The "input" is
what the script can actually "see" on the input - and since we
didn't mark any columns as being "ReadOnly" or "ReadWrite"... that
means the input has NO columns. However, the "virtual input" has
the complete list of every column that exists, whether or not we've
said we're "using" it.
We then loop over all of the "virtual columns" on this virtual
input, and for each one...
Get the LineageID and column name, and push them out as a new row on
our asynchronous script.
The image and text from Andrew's page helps explain it in a bit more detail:
This map is then merge-joined with the ErrorColumn lineage ID(s)
coming down the error path, so that the error information can be
appended with the column name(s) from the map. I included a second
script component that looks up the error description from the error
code, so the error table rows that we see above contain both column
names and error descriptions.
The remaining component that needs explaining is the conditional split
– this exists just to provide metadata to the script component that
creates the map. I created an expression (1 == 0) that always
evaluates to false for the “No Rows – Metadata Only” path, so no rows
ever travel down it.
Whilst this solution does require the insertion of some additional
plumbing within the data flow, we get extremely valuable information
logged when errors do occur. So especially when the data flow is
running unattended in Production – when we don’t have the tools &
techniques available at design time to figure out what’s going wrong –
the logging that results gives us much more precise information about
what went wrong and why, compared to simply giving us the failed data
and leaving us to figure out why it was rejected.
Here is a solution that
Works at package runtime (not pre-populating)
Is automated through a Script Task and Component
Doesn't involve installing new assemblies or custom components
Is nicely BIML compatible
Check out the full solution here.
Here is the short version.
Create 2 Object variables, execsObj and lineageIds
Create Script Task in Control flow, give it ReadWrite access to both variables
Insert the following code into your Script Task
Dictionary<int, string> lineageIds = null;
public void Main()
{
// Grab the executables so we have to something to iterate over, and initialize our lineageIDs list
// Why the executables? Well, SSIS won't let us store a reference to the Package itself...
Dts.Variables["User::execsObj"].Value = ((Package)Dts.Variables["User::execsObj"].Parent).Executables;
Dts.Variables["User::lineageIds"].Value = new Dictionary<int, string>();
lineageIds = (Dictionary<int, string>)Dts.Variables["User::lineageIds"].Value;
Executables execs = (Executables)Dts.Variables["User::execsObj"].Value;
ReadExecutables(execs);
Dts.TaskResult = (int)ScriptResults.Success;
}
private void ReadExecutables(Executables executables)
{
foreach (Executable pkgExecutable in executables)
{
if (object.ReferenceEquals(pkgExecutable.GetType(), typeof(Microsoft.SqlServer.Dts.Runtime.TaskHost)))
{
TaskHost pkgExecTaskHost = (TaskHost)pkgExecutable;
if (pkgExecTaskHost.CreationName.StartsWith("SSIS.Pipeline"))
{
ProcessDataFlowTask(pkgExecTaskHost);
}
}
else if (object.ReferenceEquals(pkgExecutable.GetType(), typeof(Microsoft.SqlServer.Dts.Runtime.ForEachLoop)))
{
// Recurse into FELCs
ReadExecutables(((ForEachLoop)pkgExecutable).Executables);
}
}
}
private void ProcessDataFlowTask(TaskHost currentDataFlowTask)
{
MainPipe currentDataFlow = (MainPipe)currentDataFlowTask.InnerObject;
foreach (IDTSComponentMetaData100 currentComponent in currentDataFlow.ComponentMetaDataCollection)
{
// Get the inputs in the component.
foreach (IDTSInput100 currentInput in currentComponent.InputCollection)
foreach (IDTSInputColumn100 currentInputColumn in currentInput.InputColumnCollection)
lineageIds.Add(currentInputColumn.ID, currentInputColumn.Name);
// Get the outputs in the component.
foreach (IDTSOutput100 currentOutput in currentComponent.OutputCollection)
foreach (IDTSOutputColumn100 currentoutputColumn in currentOutput.OutputColumnCollection)
lineageIds.Add(currentoutputColumn.ID, currentoutputColumn.Name);
}
}
4. Create Script Component in Dataflow with ReadOnly access to lineageIds and the following code.
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
Dictionary<int, string> lineageIds = (Dictionary<int, string>)Variables.lineageIds;
int? colNum = Row.ErrorColumn;
if (colNum.HasValue && (lineageIds != null))
{
if (lineageIds.ContainsKey(colNum.Value))
Row.ErrorColumnName = lineageIds[colNum.Value];
else
Row.ErrorColumnName = "Row error";
}
Row.ErrorDescription = this.ComponentMetaData.GetErrorDescription(Row.ErrorCode);
}
I connected to the SSIS Error message ref webpage with excel using the get data from web on the data tab. Saved the table in a sheet in excel, then imported it to SQL Server. Then joined it to my error rows table on the decimal code to get the description, and then created a view out of it. Thought this might be useful for those that don't want to mess with the script task.
I was pulling my hair for last couple of days. I did everything that is mentioned everywhere but the package/c# was throwing an error. Finally when I decided to give up, I found that my ErrorColumn was coming up as 0 (Zero) because the error was in entire row due to PK/FK constraint violation.
So I modified the script as below:
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
Row.ErrorDescription = this.ComponentMetaData.GetErrorDescription(Row.ErrorCode);
var componentMetaData130 = this.ComponentMetaData as IDTSComponentMetaData130;
if (componentMetaData130 != null)
{
if (Row.ErrorColumn == 0) //Checking if the Column is zero
{
Row.ColumnName = "Entire Row. Check PK FK constraints"; //Hardcoded error message
}
else
{
Row.ColumnName = componentMetaData130.GetIdentificationStringByID(Row.ErrorColumn);
}
}
}
For usual process: https://learn.microsoft.com/en-us/sql/integration-services/extending-packages-scripting-data-flow-script-component-examples/enhancing-an-error-output-with-the-script-component?view=sql-server-2017
Why ErrorColumn value is Zero?: SSIS 2016 - ErrorColumn is 0 (zero)
Hope that helps !!!
1.Script Task: set arrays of (A) inventory count and (B) StoreNr
2.Data flow task: Use the list variables in where clauses (to filter and thereby speed up performance)
*Script task must read from server A and Data flow task from server B.
I dont want to use linked server and dont want to filter downstream the dataflow, but instead want to filter through the where clauses in the dataflow source (OLE DB).
You may do it in two Data Flows.
In first:
Select value to be used in where from source table
Store this values in string variable ListToBeFetched as comma separated list using Srcipt Component as destination witch code similar to:
using System.Text;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
StringBuilder sb;
public override void PreExecute()
{
base.PreExecute();
sb = new StringBuilder();
}
public override void PostExecute()
{
base.PostExecute();
Variables.IdListToBeFetched = sb.ToString().TrimEnd(',');
}
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (!Row.Value_IsNull)
{
sb.AppendFormat("{0},", Row.Value);
}
}
}
Do the same with second list.
In second Data Flow set dynamic generated query as sql command in OLE DB Source (taken from Jamie Thomson blog):
Create a new variable called SourceSQL
Open up the properties pane for SourceSQL variable (by pressing F4)
Set EvaluateAsExpression=TRUE
Set Expression to "select * from table where columnToBeSearched in (" + #[User::ListToBeFetched] + ")"
For your OLE DB Source component, open up the editor
Set Data Access Mode="SQL Command from variable"
Set VariableName = "SourceSQL"