I have a package with a bunch of oledb Destinations, using SSIS 2016 - which is supposed to show the exact column that generated the error.The ErrorColumn shows (0) zero, therefore I am unable to trap the column that generated the error.
Using the script below (with code that assigns "Unknown column" but it does not help, it just avoids the script fail):
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
//IDTSComponentMetaData130 componentMetaData = ComponentMetaData as IDTSComponentMetaData130;
//var component130 = this.ComponentMetaData as IDTSComponentMetaData130;
//if (component130 != null)
//{
// System.Windows.Forms.MessageBox.Show(component130.GetIdentificationStringByID(Row.ErrorColumn));
// Row.ErrorColumnName = component130.GetIdentificationStringByID(Row.ErrorColumn);
//}
IDTSComponentMetaData130 componentMetaData = ComponentMetaData as IDTSComponentMetaData130;
if (componentMetaData != null)
{
//
if (Row.wkpErrorColumn != 0)
Row.wkpErrorColumnName = componentMetaData.GetIdentificationStringByID(Row.wkpErrorColumn);
else
Row.wkpErrorColumnName = "Unknown column";
}
else
{
Row.wkpErrorColumnName = "Cannot determine";
}
Row.wkpErrorDescription = ComponentMetaData.GetErrorDescription(Row.wkpErrorCode);
}
In SSIS 2008 (and I believe also SSIS 2016), a zero error column identifies an error that affects the entire row. In the example below, I have created a package that contains only 1 data flow task in the control flow and redirecting all errors and truncations to the error output. Placed a row count task just to have somewhere to send them to.
Also placed data viewers in both error flows to see the data coming out of it.
In a package consuming data from a flat file into an OLE DB Destination:
Using these values as input data
And having name as the PK of the table
We get a PK violation, check the error description and the error column
So when the error affects the entire record, it gets an error column value of 0.
Hope this helps.
I found that
When the error is generated by the PK, the error affect the entire row, and the error column is 0
When the error is generated by a Fk, The error affect only the column, and you get the error column value different then 0
Related
I received a flat file that cannot be generated in other way. The delimited is a comma and the text qualifier is a double quote. The problem is that sometimes a have a double quote in the value. In example:
"0","12345", "Centre d"edu et de recherche", "B8E7"
Because of the double quote in the value, I received this error:
[Flat File Source [58]] Error: The column delimiter for column "XYZ" was not found.
[Flat File Source [58]] Error: An error occurred while processing file "C:\somefile.csv" on data row 296.
What can I do to process this file?
I use SSIS 2016 with Visual Studio 2015
You can use the Flat File Source error output to redirect bad rows to another flat file and correct values manually while all valid rows will be processed.
There are many links online to learn more about Flat File Source Error Output:
Flat File source Error Output connection in SSIS
How to Avoid Package Design Flaws When Sourcing Data From Flat Files
Flat File Source Editor (Error Output Page)
Update 1 - Workaround using Script Component and conditional split
Since Flat File error output is not working you can use a script component with a conditional split to filter bad rows, the following update is a step by step guide to implement that:
Add a Flat File connection manager, Go To advanced Tab, Delete all columns except one column and change it length to 4000
Add a script component, Go to Input and Output Column Tab, add desired output columns (in this example 4 columns) and add a Flag Column of type DT_BOOL
Inside the Script Component write the following script to check if the number of columns is 4 then Flag = True which means this is a valid row else set Flag as False which mean that this is a bad row:
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
if (!Row.Column0_IsNull && !String.IsNullOrWhiteSpace(Row.Column0))
{
string[] cells = Row.Column0.Split(new string[] { "\",\"" }, StringSplitOptions.None);
if (cells.Length == 4)
{
Row.Col1 = cells[0].TrimStart('\"');
Row.Col2 = cells[1];
Row.Col3 = cells[2];
Row.Col4 = cells[3].TrimEnd('\"');
Row.Flag = true;
}
else
{
bool cancel;
Row.Flag = false;
}
}
else
{
Row.Col1_IsNull = true;
Row.Col2_IsNull = true;
Row.Col3_IsNull = true;
Row.Col4_IsNull = true;
Row.Flag = true;
}
}
}
Add a conditional split to split rows based on Flag column
Map the Valid Rows output to the OLEDB Destination, and the Bad Rows output to another flat file where you only map Column0
my JSON Object is varies based on some conditins. when i am trying to access the value from json object when it is not present it's througing an error message saying that value is not defined. How to avoid such kind of situations??
IntentSlots: '{"cnumberslot":{"name":"cnumberslot","value":"C186206"}}' - condition works perfectly.
IntentSlots: '{"cnumberslot":{"name":"cnumberslot"}}' } - throws an error.
my condition is like below,
if(IntentSlots.cnumberslot.value !== ""){
}
Based on the condition if the property is not available add an empty value to that property because if u leave it empty it will throw that error.
or use different if condition
if(IntentSlots.hasOwnProperty('value') && IntentSlots.cnumberslot.value !== ""){ }
I am working with EZApi to assist in creating a package to stage data for transformation. It is working in terms of data movement. When opening the package in the designer however there are warning messages surrounding the Derived Column and the InputColumns being set to read only.
Warning 148 Validation warning. Staging TableName:
{AA700319-FC05-4F06-A877-599E826EA833}: The "Additional
Columns.Inputs[Derived Column Input].Columns[DataSourceID]" on
"Additional Columns" has usage type READONLY, but is not referenced by
an expression. Remove the column from the list of available input
columns, or reference it in an expression. StageFull.dtsx 0 0
I can manually change them in the designer to be Read/Write or unselect them and the warning goes away. I am unable to get this to work programmatically however.
I have tried removing the columns from the metadata which works but doesn't remove them from the component so the columns are still created in the xml.
XML section
<externalMetadataColumn refId="Package\Full\Staging TableName\DestinationStaging TableName.Inputs[OLE DB Destination Input].ExternalColumns[DataSourceID]" dataType="i4" name="DataSourceID" />
When I try to go to the underlying object and delete the column using component.DeleteInput(id) I get an error message stating that the input column cannot be removed.
0xC0208010
-1071611888
DTS_E_CANTDELETEINPUT
An input cannot be deleted from the inputs collection.
Here is the code I am using to create a data flow task with an OLEDB Source, Derived Column, and OLE DB Destination.
Note that the input columns are not present until after the derived column is attached to the Source: dc.AttachTo(source);
public class EzMyDataFlow : EzDataFlow
{
public EzMyDataFlow(EzContainer parent, EzSqlOleDbCM sourceconnection,
EzSqlOleDbCM destinationconnection, string destinationtable, string sourcecomannd, string dataflowname)
: base(parent)
{
Name = dataflowname;
EzOleDbSource source = new EzOleDbSource(this);
source.Connection = sourceconnection;
source.SqlCommand = sourcecomannd;
source.AccessMode = AccessMode.AM_SQLCOMMAND;
source.Name = string.Format("Source_{0}", dataflowname);
EzDerivedColumn dc = new EzDerivedColumn(this);
dc.Name = "Additional Columns";
// Setup DataSourceID
string columnName = DBSchema.ReportFoundationalColumns.DataSourceID;
dc.InsertOutputColumn(columnName);
dc.SetOutputColumnDataTypeProperties(columnName, DataType.DT_I4, 0, 0, 0, 0);
var c = dc.OutputCol(columnName);
var property = c.CustomPropertyCollection["Expression"];
property.Name = "Expression";
property.Value = "#[TM::SourceDatabaseID]";
property = c.CustomPropertyCollection["FriendlyExpression"];
property.Name = "FriendlyExpression";
property.Value = "#[TM::SourceDatabaseID]";
dc.AttachTo(source);
EzOleDbDestination destination = new EzOleDbDestination(this);
destination.Table = destinationtable;
destination.Connection = destinationconnection;
destination.Name = string.Format("Destination{0}", dataflowname);
destination.AttachTo(dc);
}
}
I've 10 flat files(.dat format) in a folder which need to be uploaded into database everyday at a scheduled time.
All files related information are present in a database like File name, file path, table name, column names and delimiter.
We need to check each file exists or not, if not, need to log an entry, "File Not Found".
If the file exists, it needs to check for a trailer record(the last record in the file which says, Count=00001000, It has to be the count of number of records in that particular file).
If the trailer record not exists, then need to log an entry "No trailer record found".If the trailer record says zero count, then a log entry has to be made "Zero count" and also, if the counts of the file are not matching a log entry is needed, "Count mismatch".
If all the conditions are satisfied then data need to be loaded into database for each of the file.
Please suggest your ideas to implement the above scenario. Thanks!!!
Following solution may help you to resolve the issue.
Use the For each loop container with "Item" enumerator. Since you have 10 files and if something missing you need raise then you should use this. File enumerator just iterate through the files, not raises any error.
Following are Steps.
Create following SSIS package with variables.
FileFullPath
IsValidated
For each loop enumerator should be configured as following screenshots.
Configuartion in collection:
configuration in Variable section
Inside the container have a script task. you have to mention the FileFullPath as readonly variable and IsValidate as read and write like the following screen.
Click Edit script and insert the following code.
public void Main()
{
Dts.Variables["IsValidated"].Value = true;
string fileFullPath = Dts.Variables["FileFullPath"].Value.ToString();
if (!File.Exists(fileFullPath))
{
var msg = String.Format("File is not available in location : {0}", fileFullPath);
Dts.Events.FireError(0, "Dat file loading", msg, string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
//Read last line
String lstLine = File.ReadLines(fileFullPath).Last();
int totalCount = 0;
bool talierExists = int.TryParse(lstLine, out totalCount);
if (!talierExists)
{
var msg = String.Format("No tailer row found and last line is : {0}", lstLine);
Dts.Events.FireError(0, "Dat file loading", msg, string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
//Total count
int fullCount = File.ReadLines(fileFullPath).Count();
if (fullCount != totalCount)
{
var msg = String.Format("No of count is not matching, tailer count = {0} and full count={1}");
Dts.Events.FireError(0, "Dat file loading", msg, string.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
Dts.Variables["IsValidated"].Value = true;
Dts.TaskResult = (int)ScriptResults.Success;
}
After that have your Data flow. Connect the script task with your data flow and right click on the connector and go to edit and configure as follows.
Your SSIS package will looks like follows.
Hope this helps!
I got this error while debugging in SSIS:
Error: 0xC0049064 at Data Flow Task, Derived Column [70]: An error occurred while attempting to perform a type cast.
Error: 0xC0209029 at Data Flow Task, Derived Column [70]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "Derived Column" (70)" failed because error code 0xC0049064 occurred, and the error row disposition on "output column "EVENT_DT" (118)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Error: 0xC0047022 at Data Flow Task: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Derived Column" (70) failed with error code 0xC0209029. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
Error: 0xC0047021 at Data Flow Task: SSIS Error Code DTS_E_THREADFAILED. Thread "WorkThread0" has exited with error code 0xC0209029. There may be error messages posted before this with more information on why the thread has exited.
Information: 0x40043008 at Data Flow Task, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x40043009 at Data Flow Task, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x4004300B at Data Flow Task, DTS.Pipeline: "component "DataReaderDest" (143)" wrote 0 rows.
Task failed: Data Flow Task
Warning: 0x80019002 at Package: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (4) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
My expression is:
(DT_DBTIMESTAMP)(SUBSTRING(EVENT_D,7,4) + "-" +
SUBSTRING(EVENT_D,4,2) + "-" +
SUBSTRING(EVENT_D,1,2) + EVENT_T)
My original data are in this sequence:
EVENT_D: DD/MM/YYYY
EVENT_T: HH:MM:SS
Any help are appreciated. I have try changing my expression numerous time but still fails.
I suspect that there are some date time values which are not in the correct format .So SSIS throws error while parsing them .
In order to find to incorrect date time value from your source table try to redirect the error rows from Derived Transformation and check the incorrect data using a data viewer
The problem with substring values are if the string data are not in the correct format then the Derived Transformation will throw error .
In your OLEDB source component try to write the sql and get only the correct datetime values
Ex:
Select col1,
case when isDate(EVENT_D) = 1 THEN EVENT_D ELSE NULL
END as [EVENT_D],
Col2,EVENT_T,other columns
from yourTable
in your derived transformations use your code to convert into DT_DBTIMESTAMP type .
Else Try to use a script component and parse the EVENT_D and EVENT_T values and convert to datetime values. No need to use Derived column with all those substring values
create a New Output column Valid_D with the datatype as DT_DBTIMESTAMP.Select the 2 input columns EVENT_D and EVENT_T in the available input Columns in Script Transformation Editor
VB.NET code
Dim result As DateTime
If DateTime.TryParseExact(Row.EVENT_D, "dd/MM/yyyy",
Nothing, Globalization.DateTimeStyles.None, result) Then
Row.ValidD = result.Add (TimeSpan .Parse (Row.EventT ) );
End If
C# code
DateTime result;
if (DateTime.TryParseExact(Row.EventD, "dd/MM/yyyy",
CultureInfo.InvariantCulture, DateTimeStyles.None,out result))
{
Row.ValidD = result.Add (TimeSpan .Parse (Row.EventT ) );
}
Now you can use Valid_D column which is of DateTime type in your subsequent transformations
Update : The problem with your syntax is you cannot add date+ time in the string format . You need to parse individually both EVENT_D and EVENT_T .
(DT_DBTIMESTAMP)(SUBSTRING(EVENT_D,7,4) + "-" +
SUBSTRING(EVENT_D,4,2) + "-" +
SUBSTRING(EVENT_D,1,2) + EVENT_T)
So your syntax is not valid.
The isDate function shows NULL for 30/04/2012 because as per MSDN
The return value of ISDATE depends on the settings set by
SET DATEFORMAT, SET LANGUAGE and Configure the default language
Server Configuration Option.
but it returns 1 for the value 01/05/2012 and 02/05/2012 because it takes as MM/dd/YYYY
so the 1st date is Jan 5th 2012 instead of May 1st 2012
So the soultion is use script component and parse these values into valid date and time and then use it in your subsequent transformations .
Please check my script transformation code above
Update 2:
I think your using SSIS 2005 .The code should be
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Dim result As DateTime
If DateTime.TryParseExact(Row.EVENTD, "dd/MM/yyyy", Nothing, Globalization.DateTimeStyles.None, result) Then
Row.ValidD = result.Add(TimeSpan.Parse(Row.EVENTT))
End If
End Sub
after Script transformation ,you don't need to use Derived component .The result obtained in the column Valid_D contains the valid value which is of datetime format
If you wanted to do this without a script, you could have just changed your expression from
(DT_DBTIMESTAMP)(SUBSTRING(EVENT_D,7,4) + "-" +
SUBSTRING(EVENT_D,4,2) + "-" +
SUBSTRING(EVENT_D,1,2) + EVENT_T)
to
(DT_DBTIMESTAMP)(SUBSTRING(EVENT_D,7,4) + "-" +
SUBSTRING(EVENT_D,4,2) + "-" +
SUBSTRING(EVENT_D,1,2) + " " + EVENT_T)
Not having the space between the date and the time portions is what was most likely causing your original error. You were getting
2012-04-3012:25:37
instead of
2012-04-30 12:25:37
I faced the same problem while converting a string containing date to datetimestamp field. It is all due to spaces in the field.
So we need to check for space and then convert it to datetimestamp.
Below is the expression for the same.
LEN(TRIM([Field1])) == 1 ? NULL(DT_DBTIMESTAMP) : (DT_DBTIMESTAMP)(SUBSTRING([Field1],1,4) + "-" + SUBSTRING([Field1],5,2) + "-" + SUBSTRING([Field1],7,2) + " " + SUBSTRING([Field1],10,2) + ":" + SUBSTRING([Field1],12,2))