Derived Column Editor - ssis

I need to assign a formatted date to a column in a data flow. I have added a Derived shape and entered the following expression for a NEW column - Derived Column = "add as new column":
"BBD" + SUBSTRING((DT_WSTR,4)DATEADD("Day",30,GETDATE()),1,4) +
SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),6,2) +
SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),9,2)
The problem is that the Derived Column Transformation Editor automatically assigns a Data Type of Unicode string[DT_WSTR] and a length of "7". Howver, the length of a string is 11, therefore the following exception is thrown each time:
[Best Before Date [112]] Error: The "component "Best Before Date" (112)" failed
because truncation occurred, and the truncation row disposition on "output column
"Comments" (132)" specifies failure on truncation. A truncation error occurred
on the specified object of the specified component.
Does anyone know why the edit is insisting on a length of 7? I don't seem to be able to change this.
Many thanks,
Rob.

I can't understand why SSIS is measuring that column as only resulting in a seven character field - but to force it to provide an 11 character column for it, modify your expression slightly to this:
(DT_WSTR, 11)("BBD" + SUBSTRING((DT_WSTR,4)DATEADD("Day",30,GETDATE()),1,4) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),6,2) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),9,2))

What you want is:
"BBD" + (DT_WSTR,4)YEAR(DATEADD("Day",30,GETDATE()))
+ RIGHT("0" + (DT_WSTR,2)MONTH(DATEADD("Day",30,GETDATE())),2)
+ RIGHT("0" + (DT_WSTR,2)DAY(DATEADD("Day",30,GETDATE())),2)
The issue is in how you are converting your dates to a string. The calls to DATEADD return a full date & time. Next, you then have either (DT_WSTR,4) or (DT_WSTR,2) to convert that date into either a 4 or 2 character string. On my system, converting a datetime to a string defaults to "Aug 24 2011 4:18PM". So the first 4 characters gets you "Aug " and the first 2 characters gets you "Au". Then, you are extracting substrings using SUBSTRING. For your last two calls to SUBSTRING, you are starting the substring past the end of the 2 character string you converted the date into. This is why SSIS displays 7 characters:
"BBD" + "Aug " + "" + ""
3 + 4 + 0 + 0 = 7
It is better to use the built in functions to extract the Year, Month and Day from a datetime rather than converting to a string and then grabbing substrings. If you really wanted to use substrings, you would need to add a call to CONVERT to get the datetime to a specific string format, otherwise you will get whatever the default is for your locale setting in Windows. This could be different on each PC.

What release and service pack of SQL are you using?
I just tried this on my machine and had no problems changing the result size from 7 to 11. Is it possible that you have not installed all the service packs?

Are you replacing your existing field, and is that field possibly 7 chars long? The thing with the Derived Column Transform is that you can't change the field types (including length) of the existing fields.
Try to add a new field instead.
If that's not working, try adding an explicit cast around the whole expression.
(DT_WSTR,11)("BBD" + SUBSTRING((DT_WSTR,4)DATEADD("Day",30,GETDATE()),1,4) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),6,2) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),9,2))

Right click on "Derived Column" open "Show Advanced Editor" Select "Input and output Properties" tab.
Got to "Derived column output" => "Output Columns" => "Derived Column 1" (added by you)
In right side panel go to "Data type Properties" section=> DataType=>
Select "String [DT_STR]
click OKImage showing steps
This will solve your problem.

Related

Importing sensitive data from SQL server to a flat file

I am new to SSIS and I want to import data from the SQL database to a flat-file.
I have a column name named SecurityNumber which contains sensitive data, so using a derived column I want to use a symbol * to hide some numbers.
For example, if the column has a value 1983121802 and
if substring 3 plus substring 4 (in this case 8+3) is less than 13 display the column as **83****** or else display the whole value (1983121802)
how can I do that?
and the column SecurityNumber has a string data type, How can I change it to another data type so it's possible to make the comparison?
Thank you for your help!
Try to use the following expression:
((DT_I4)SUBSTRING([SecurityNumber],3,1) + (DT_I4)SUBSTRING([SecurityNumber],4,1)) > 13 ? [SecurityNumber] : "**" + SUBSTRING([SecurityNumber],3,2) + "******"

SSIS Derived Column System Variable Length with expression

I have a source column, KEY2 that I need to perform some string manipulation on within the Derived Column transformation within SSIS. It's configured as the following
KEY2
Replace 'KEY2'
(FINDSTRING(KEY,",",1) - 2) > 0 ?
SUBSTRING(KEY,FINDSTRING(KEY,",",1) + 2,LEN(KEY) - FINDSTRING(KEY,",",1) - 1) : ""
string [DT_STR]
1
1252 (ANSI - Latin I)
The length is 1. I need to change it to 100. I have tried this:
(DT_STR,100)(FINDSTRING(KEY,",",1) - 2) > 0 ?
SUBSTRING(KEY,FINDSTRING(KEY,",",1) + 2,LEN(KEY) - FINDSTRING(KEY,",",1) - 1) : ""
However, I am prompted with a parse error. What is the problem?
Update
I have followed markTheLiars' answer. The expression now looks like this:
KEY2 Replace 'KEY2' (DT_STR,100,1252)((FINDSTRING(KEY,",",1) - 2) > 0 ?
SUBSTRING(KEY,FINDSTRING(KEY,",",1) + 2,LEN(KEY) - FINDSTRING(KEY,",",1) - 1) : "") string [DT_STR] 1 1252 (ANSI - Latin I)
The expression compiles and runs, however I am prompted with the same error as the length does not change even though there is a cast.
Your cast is missing the "Code_page" parameter:
(DT_STR, 100, 1252) ((FINDSTRING(KEY,",",1) - 2) > 0 ? SUBSTRING(KEY,FINDSTRING(KEY,",",1) + 2,LEN(KEY) - FINDSTRING(KEY,",",1) - 1) : "")
1252 is the default value. See this answer for a much better explanation than I could give as to why it's important. See here for more info about casting/conversions.
It appears that the meta-data for that column is still set to be 1 character long. Right-click on the derived column transformation, select Show Advanced Editor, select the "Input and Output Properties". Expand "Derived Column Output" -> "Output Columns" -> Your column (Key2 in this instance I believe). Under "Data Type Properties" edit Length to be 100.
As was eventually discovered, the problem was not in this component but a previous component that was truncating the data before it ever reached this date flow component. In this case, the easiest way to determine which component is causing the truncation is to use the meta-data viewer, available by double-clicking on the flow pathway or right-clicking and selecting "Edit":
This will quickly give you info at a glance about the variables and their source components. Unfortunately it will not tell you exactly where the truncation occurred, but if your data flow component isn't too complicated you should still be able to find the problem area relatively quickly.

SSIS converting Time Stamp format using derived column

I have a timestamp in a text file that looks like this: 7/2/2013 17:40:22
I need to convert it to this: 2013-07-02 17:40:22.913
Using a derived column, I tried this: (DT_DBTIMESTAMP)TIME_STAMP
and
(DT_DBTIMESTAMP)(SUBSTRING(TIME_STAMP,1,4) + "-" +
SUBSTRING(TIME_STAMP,5,2) + "-" +
SUBSTRING(TIME_STAMP,7,2))
But I am getting an error: Conversion between types DT_STR and DT_DBTIMESTAMP is not supported.
Thanks!
If you must go the route of casts, let me know and I'll edit this answer to reflect it. Otherwise, simplify your life and fix your Flat File Connection Manager. By default, it's going to assume every column coming in is varchar(50). This gets tricky with dates and times because well known formats that should just convert require far too much expression garbage to make them work.
In your case, change the type to DT_DATE I know you'd think it's for dealing date only but that'd be DT_DBDATE.
By simply changing to that data type, I was able to import your value without issue.
using token will do ... assume your timestamp is in a string(WSTR) format
TOKEN([Order Date],"/",3) + "-" + TOKEN([Order Date],"/",1) + "-" + TOKEN([Order Date],"/",2)

How to convert string in format yyyyMMdd to date using SSIS expression?

I have a date 20130131 in csv which I'm trying to convert to 2013-13-01 using Derived Column in SSIS. I have used this expression but it does not seem to work.
(DT_DBTIMESTAMP)(SUBSTRING(DATE_DECISION_TO_REFER,1,2) + "-" + SUBSTRING(DATE_DECISION_TO_REFER,3,2) + "-" + SUBSTRING(DATE_DECISION_TO_REFER,5,2))
How do I rewrite this expression to produce the correct output with a value of data type DT_DBTIMESTAMP?
Fast Parse is a much more elegant solution
Add a Data Conversion Data Flow Component.
Right click, Show Advanced Editor...
Goto Input and Output Properties
Expand Data Conversion Output and click on the date column
Set FastParse to True in the Custom Properties
Make sure the data type is set to database date [DT_DBDATE]
Run the ssis package again the yyyymmdd dates should flow smoothly from nvarchar to date
As per Kyle Hale suggestion, below is my answer
SUBSTRING([DATE_DECISION_TO_REFER],1,4) + "-" +
SUBSTRING([DATE_DECISION_TO_REFER],5,2) + "-" + SUBSTRING([DATE_DECISION_TO_REFER],7,2)
To educate you so that you will never face this issue again, this is how expression works
Get 4 characters from DATE_DECISION_TO_REFER starting at position 1, add a dash,
get 2 characters from DATE_DECISION_TO_REFER starting at position 5,
add a dash then add 2 characters from DATE_DECISION_TO_REFER starting at position 7.
Hope this will help.
Expression:
You should use the expression to format the date to yyyy-MM-dd like shown below. This expression will format the incoming value 20130131 to 2013-01-31 and then will convert to appropriate data type. The expression uses the incoming column named Column0.
(DT_DBTIMESTAMP)(SUBSTRING(Column0,1,4) + "-" + SUBSTRING(Column0,5,2) + "-" + SUBSTRING(Column0,7,2))
Issue:
Your expression is taking the value 20130131 and then incorrectly converting it to the value 20-13-01. This is an invalid format for dates and hence the expression is failing to type cast it to DT_DBTIMESTAMP data type.

SSIS Derived Column Expression

Reading from a flat file that has a column containing the year and month in the following format --> "201212".
I need to insert this into a DATETIME2 column but first I must parse it with "/" and add a "01" as the day. Such that 201212 would become 2012/12/01
My expression in my Derived Column Transformation looks like this:
(DT_DBTIMESTAMP2,0)((DT_WSTR,4)SUBSTRING(RptMthDTM,1,4) + "/" + (DT_WSTR,2)SUBSTRING(RptMthDTM,5,2) + "/" + "01")
This seems like it should work and SSIS accepts this(as in it can parse the expression) but when running the package it throws a completely useless error "An error occurred while attempting to perform a type cast." along with the column it had the error on.
I didn't build this package, it was pawned off on me and I was told to get it working.
Thanks in advance interwebs family.
DT_DBTIMESTAMP2 can only be converted from a specific string format: yyyy-mm-dd hh:mm:ss[.fffffff]
You can use this expression instead:
(DT_DBTIMESTAMP2,0)((DT_WSTR,4)SUBSTRING(RptMthDTM,1,4) + "-" + (DT_WSTR,2)SUBSTRING(RptMthDTM,5,2) + "-" + "01 00:00:00")
More details:
[http://msdn.microsoft.com/en-us/library/ms141036.aspx][1]