I have the following simple SSIS Package to process a comma delimited file:
I'm having problems with date columns - specifically MoveInDate which the input column has data in the format '20090731000000'.
It goes into a database column MoveInDt which has a Datetime data type and data ends up in the format:
'2009-07-31 00:00:00.000'
In SSIS I have specified the input column to be DT_DBTIMESTAMP. I'm unable to figure out how to do a simple 'if MoveInDate is blank, load null, else change data to 'YYYY-MM-DD 00:00:00.000'
Right now I can't even seem to figure out how to load a hardcoded date
This fails
As does
"2019-01-01"
Or
(DT_DBTIMESTAMP)"2019-01-01"
(DT_DBTIMESTAMP)(DT_DATE)"2019-01-01"
Error: 0xC02020A1 at Data Flow Task, Flat File Source [11]: Data conversion failed. The data conversion for column "Move In Date" returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
Error: 0xC0209029 at Data Flow Task, Flat File Source [11]:
Looking up that message I've found posts about the data types not matching on the input to output column, but looking at it ...
Checking the flat file input column, it is defined as DT_DBTIMESTAMP
And the destination column is also defined as DT_DBTIMESTAMP when looking at input output properties.
I've looked at many questions here and elsewhere and have tried reducing the input to one simple row in the input file. I've tried recreating the package several times.
If the data is stored in the following format in the flat file
20090731000000
Then leave the source column data type as DT_STR and add a dervied column with the following expression:
(ISNULL([MoveInDate]) || [MoveInDate] == "") ? NULL(DT_DBTIMESTAMP) :
(DT_DBTIMESTAMP)(LEFT([MoveInDate],4) + "-" +
SUBSTRING([MoveInDate],5,2) + "-" +
SUBSTRING([MoveInDate],7,2) + " " +
SUBSTRING([MoveInDate],9,2) + ":" +
SUBSTRING([MoveInDate],11,2) + ":" +
SUBSTRING([MoveInDate],13,2) )
If data is stored in the following format:
2009-07-31 00:00:00
Use the following expression:
(ISNULL([MoveInDate]) || [MoveInDate] == "")
? NULL(DT_DBTIMESTAMP)
: (DT_DBTIMESTAMP)[MoveInDate]
If data is stored in the following format:
2009-07-31
Use the following expression:
(ISNULL([MoveInDate]) || [MoveInDate] == "")
? NULL(DT_DBTIMESTAMP)
: (DT_DBTIMESTAMP)([MoveInDate] + " 00:00:00")
Related
I have a SSIS package that inserts data from csv into SQL database.
This file has a data column that the format is different from SQL. I am getting error because the SSIS says that the data can not be convert.
The datacolumn is like this 11/9/2022 12:00:00 AM this data is related to 9th of november.
I need to use derived column to convert this data into dd/mm/yyyy.
I was trying using SUBTRING but I noticed that as firts character are the month we do not have the same number of character before the first / and the same for day.
I was wondering if I could try something like that
SUBSTRING([Dt],FINDSTRING("/",[Dt],1) + 1,FINDSTRING("/",[Dt],2) - 1) + "/" + LEFT([Dt],FINDSTRING("/",[Dt],1) - 1) + "/" + SUBSTRING([Dt],FINDSTRING("/",[Dt],2) + 1,4)
But I am getting the same error
How can I do to convert 11/9/2022 into this 09/11/2022 (dd/mm/yyyy) using derived column?
You were on the right path, but had the wrong syntax for FINDSTRING - the string to search comes first.
After a bit of trial and error I came up with this:
((FINDSTRING([Dt],"/",2) - FINDSTRING([Dt],"/",1)-1)==1?"0":"") + SUBSTRING ([Dt],FINDSTRING([Dt],"/",1)+1, FINDSTRING([Dt],"/",2) - FINDSTRING([Dt],"/",1)-1) + "/" + (FINDSTRING([Dt],"/",1)==2?"0":"") + SUBSTRING ([Dt],1,FINDSTRING([Dt],"/",1)-1) + "/" + SUBSTRING ([Dt],FINDSTRING([Dt],"/",2)+1,4)
SSIS functions can't be multi-line so the above is on 1 line, this is over multiples lines for ease of reading
((FINDSTRING([Dt],"/",2) - FINDSTRING([Dt],"/",1)-1)==1?"0":"") +
SUBSTRING ([Dt],FINDSTRING([Dt],"/",1)+1, FINDSTRING([Dt],"/",2) - FINDSTRING([Dt],"/",1)-1) + "/" +
(FINDSTRING([Dt],"/",1)==2?"0":"") +
SUBSTRING ([Dt],1,FINDSTRING([Dt],"/",1)-1) + "/" +
SUBSTRING ([Dt],FINDSTRING([Dt],"/",2)+1,4)
The 1st and 3rd lines are to handle if the day or month is single number, and adds a 0 of it is.
This stuff is always easier in c#. Assuming you have a string, then use script componentas follows:
var newCol = DateTime.ParseExact(row.ColumnName
, "MM/dd/yyyy hh:mm:ss tt"
, System.Globalization.CultureInfo.InvariantCulture)
.ToString("dd/MM/yyyy");
All that being said. It is not great practice to carry strings around that represent dates. Once properly loaded into a date, keep it as a date until the final presentation layer.
I have a column in a text-file with dates like this: 20131207.
I and trying to convert the date like this but everything I have tried has failed.
Here is my code: I have modified it many times to fix this problem.
(LEN(TRIM(MyDate)) < 8) || (TRIM(MyDate) == " ") ||
(ISNULL(MyDate)) ? NULL(DT_DBDATE) :
(DT_DBDATE)(SUBSTRING(TRIM(MyDate),1,4) + "-" +
SUBSTRING(TRIM(MyDate),5,2) + "-" + SUBSTRING(TRIM(MyDate),7,2))
Where am I going wrong?
I checked Retain Null Values in the Flat Source Task.
Update: Here is the error message:
The conditional operation failed.
The "Rename Columns (My_Table)" failed because error code 0xC0049063 occurred,
and the error row disposition on "Rename Columns (My_Table).Outputs[Derived
Column Output].Columns[MyDate]" specifies failure on error. An error occurred
on the specified object of the specified component. There may be error
messages posted before this with more information about the failure.
I need to convert the data in the text file to date, from 20131207 to 2013-12-07 but I want to insert null if the data in the text file is less then 8 in length or cannot be converted to date.
try this:
add a new flat file source then add a derived column then in expression do this
`LEFT(date,4) + "-" + SUBSTRING(date,5,2) + "-" + RIGHT(date,2)`
the package :
with output like this :
Trim(MyDate)==" " should be Trim(MyDate)==""
My expression is like following:
DATEDIFF("dd",(DT_DATE)(SUBSTRING(#[User::strExcelFileName],15,2) + "-" + SUBSTRING(#[User::strExcelFileName],18,2) + "-" + SUBSTRING(#[User::strExcelFileName],21,4)),GETDATE()) > #[User::intDaysCount]
My file : TempConfigPr_06172013.xlsx
The problem is it's saying error while converting from "DT_WSTR" to data type "DT_DATE"...Could any one please help, where I'm going wrong and how to solve this?
Note: I checked,this expression is working fine.
DATEDIFF("dd",(DT_DATE)("11-18-2010"),GETDATE()) > #[User::intDaysCount]
Editing expressions in SSIS can be painful. I find when they aren't working as expected, it's best to break them into multiple variables and have them feed off each other.
By doing so, I quickly determined that your root problem was this expression was not generating a valid date value. It generated 61-20-3.xl
SUBSTRING(#[User::strExcelFileName],15,2) + "-"
+ SUBSTRING(#[User::strExcelFileName],18,2) + "-"
+ SUBSTRING(#[User::strExcelFileName],21,4)
I created a variable strExcelFileDate of type string and I used the expression to create a string that is the 8 characters following the underscore.
SUBSTRING(#[User::strExcelFileName], FINDSTRING(#[User::strExcelFileName], "_", 1)+1, 8)
That string value, 06172013, cannot be directly cast to a date, which is a pity. I needed to slice and dice that string into something that can be cast to a date data type. A new Variable named dtExcelFileDate with a type of DateTime was created. I used an expression on that to transform the string into yyyy-mm-dd and then cast that entire thing to a DT_DATE data type.
(DT_DATE) (RIGHT(#[User::strExcelFileDate], 4) + "-"
+ SUBSTRING(#[User::strExcelFileDate], 1,2)
+ "-" + RIGHT(SUBSTRING(#[User::strExcelFileDate],1,4), 2))
I want to insert a value "5/27/2013 16:42:37.490000" (Read from a flat file (DT_STR)) into a column(datetime) of SQL Server table . If I try to cast it with (DT_DBDATE) or DT_DBTIMESTAMP in a derived column , it gives an error .
[Derived Column [130]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "Derived Column" (130)" failed because error code 0xC0049064 occurred, and the error row disposition on "output column "Derived Column 1" (155)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
How can I go about it ?
Thanks
The value is a datetime2 type .AFAIK SSIS doesn't support datetime2 .You need to store it in database as string and then update the column by converting it to datetime2.
Here is Microsoft Connect Issue
Update: Using DT_DBTIMESTAMP2 you can convert the string to proper datetime format
The below code works perfectly fine in Derived Column
(DT_DBTIMESTAMP2,7)"2013-5-27 16:42:37.490000"
7 is the precession here .The above code won't work if the format of datetime is different.For example MM/DD/YYYY HH:mm:ss.ffffff .
However you can use Script component and pass array of different datetime formats to Datetime.ParseExact function
Step1: Drag a Script component and create a new output column of DT_DBTIMESTAMP datatype and name it as NewDate.
Step2: Write the below C# code
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
string[] format = new string[] { #"M/dd/yyyy HH:mm:ss.ffffff",
#"MM/dd/yyyy HH:mm:ss",
#"M/d/yyyy HH:mm:ss" ,
#"M/dd/yyyy HH:mm:ss.ffffff",
#"MM/dd/yyyy HH:mm:ss.ffffff",
#"M/d/yyyy HH:mm:ss.ffffff"};
DateTime dt = DateTime.ParseExact(Row.Date,
format ,
CultureInfo.InvariantCulture,
DateTimeStyles.None);
Row.newDate = dt;
}
I won't say it's the perfect solution, but just a workaround.
Format the input and then insert. The formatted data after using the below expression works well with either datetime or datetime2.
SUBSTRING(TMS_CREAT,(FINDSTRING(TMS_CREAT,"/",2) + 1),4) + "-" + SUBSTRING(TMS_CREAT,1,(FINDSTRING(TMS_CREAT,"/",1) - 1)) + "-" + SUBSTRING(TMS_CREAT,(FINDSTRING(TMS_CREAT,"/",1) + 1),2) + SUBSTRING(TMS_CREAT,(FINDSTRING(TMS_CREAT,"/",2) + 5),16)
I got this error while debugging in SSIS:
Error: 0xC0049064 at Data Flow Task, Derived Column [70]: An error occurred while attempting to perform a type cast.
Error: 0xC0209029 at Data Flow Task, Derived Column [70]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "Derived Column" (70)" failed because error code 0xC0049064 occurred, and the error row disposition on "output column "EVENT_DT" (118)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Error: 0xC0047022 at Data Flow Task: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Derived Column" (70) failed with error code 0xC0209029. The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
Error: 0xC0047021 at Data Flow Task: SSIS Error Code DTS_E_THREADFAILED. Thread "WorkThread0" has exited with error code 0xC0209029. There may be error messages posted before this with more information on why the thread has exited.
Information: 0x40043008 at Data Flow Task, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x40043009 at Data Flow Task, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x4004300B at Data Flow Task, DTS.Pipeline: "component "DataReaderDest" (143)" wrote 0 rows.
Task failed: Data Flow Task
Warning: 0x80019002 at Package: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. The Execution method succeeded, but the number of errors raised (4) reached the maximum allowed (1); resulting in failure. This occurs when the number of errors reaches the number specified in MaximumErrorCount. Change the MaximumErrorCount or fix the errors.
SSIS package "Package.dtsx" finished: Failure.
My expression is:
(DT_DBTIMESTAMP)(SUBSTRING(EVENT_D,7,4) + "-" +
SUBSTRING(EVENT_D,4,2) + "-" +
SUBSTRING(EVENT_D,1,2) + EVENT_T)
My original data are in this sequence:
EVENT_D: DD/MM/YYYY
EVENT_T: HH:MM:SS
Any help are appreciated. I have try changing my expression numerous time but still fails.
I suspect that there are some date time values which are not in the correct format .So SSIS throws error while parsing them .
In order to find to incorrect date time value from your source table try to redirect the error rows from Derived Transformation and check the incorrect data using a data viewer
The problem with substring values are if the string data are not in the correct format then the Derived Transformation will throw error .
In your OLEDB source component try to write the sql and get only the correct datetime values
Ex:
Select col1,
case when isDate(EVENT_D) = 1 THEN EVENT_D ELSE NULL
END as [EVENT_D],
Col2,EVENT_T,other columns
from yourTable
in your derived transformations use your code to convert into DT_DBTIMESTAMP type .
Else Try to use a script component and parse the EVENT_D and EVENT_T values and convert to datetime values. No need to use Derived column with all those substring values
create a New Output column Valid_D with the datatype as DT_DBTIMESTAMP.Select the 2 input columns EVENT_D and EVENT_T in the available input Columns in Script Transformation Editor
VB.NET code
Dim result As DateTime
If DateTime.TryParseExact(Row.EVENT_D, "dd/MM/yyyy",
Nothing, Globalization.DateTimeStyles.None, result) Then
Row.ValidD = result.Add (TimeSpan .Parse (Row.EventT ) );
End If
C# code
DateTime result;
if (DateTime.TryParseExact(Row.EventD, "dd/MM/yyyy",
CultureInfo.InvariantCulture, DateTimeStyles.None,out result))
{
Row.ValidD = result.Add (TimeSpan .Parse (Row.EventT ) );
}
Now you can use Valid_D column which is of DateTime type in your subsequent transformations
Update : The problem with your syntax is you cannot add date+ time in the string format . You need to parse individually both EVENT_D and EVENT_T .
(DT_DBTIMESTAMP)(SUBSTRING(EVENT_D,7,4) + "-" +
SUBSTRING(EVENT_D,4,2) + "-" +
SUBSTRING(EVENT_D,1,2) + EVENT_T)
So your syntax is not valid.
The isDate function shows NULL for 30/04/2012 because as per MSDN
The return value of ISDATE depends on the settings set by
SET DATEFORMAT, SET LANGUAGE and Configure the default language
Server Configuration Option.
but it returns 1 for the value 01/05/2012 and 02/05/2012 because it takes as MM/dd/YYYY
so the 1st date is Jan 5th 2012 instead of May 1st 2012
So the soultion is use script component and parse these values into valid date and time and then use it in your subsequent transformations .
Please check my script transformation code above
Update 2:
I think your using SSIS 2005 .The code should be
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
Dim result As DateTime
If DateTime.TryParseExact(Row.EVENTD, "dd/MM/yyyy", Nothing, Globalization.DateTimeStyles.None, result) Then
Row.ValidD = result.Add(TimeSpan.Parse(Row.EVENTT))
End If
End Sub
after Script transformation ,you don't need to use Derived component .The result obtained in the column Valid_D contains the valid value which is of datetime format
If you wanted to do this without a script, you could have just changed your expression from
(DT_DBTIMESTAMP)(SUBSTRING(EVENT_D,7,4) + "-" +
SUBSTRING(EVENT_D,4,2) + "-" +
SUBSTRING(EVENT_D,1,2) + EVENT_T)
to
(DT_DBTIMESTAMP)(SUBSTRING(EVENT_D,7,4) + "-" +
SUBSTRING(EVENT_D,4,2) + "-" +
SUBSTRING(EVENT_D,1,2) + " " + EVENT_T)
Not having the space between the date and the time portions is what was most likely causing your original error. You were getting
2012-04-3012:25:37
instead of
2012-04-30 12:25:37
I faced the same problem while converting a string containing date to datetimestamp field. It is all due to spaces in the field.
So we need to check for space and then convert it to datetimestamp.
Below is the expression for the same.
LEN(TRIM([Field1])) == 1 ? NULL(DT_DBTIMESTAMP) : (DT_DBTIMESTAMP)(SUBSTRING([Field1],1,4) + "-" + SUBSTRING([Field1],5,2) + "-" + SUBSTRING([Field1],7,2) + " " + SUBSTRING([Field1],10,2) + ":" + SUBSTRING([Field1],12,2))