I have a source column, KEY2 that I need to perform some string manipulation on within the Derived Column transformation within SSIS. It's configured as the following
KEY2
Replace 'KEY2'
(FINDSTRING(KEY,",",1) - 2) > 0 ?
SUBSTRING(KEY,FINDSTRING(KEY,",",1) + 2,LEN(KEY) - FINDSTRING(KEY,",",1) - 1) : ""
string [DT_STR]
1
1252 (ANSI - Latin I)
The length is 1. I need to change it to 100. I have tried this:
(DT_STR,100)(FINDSTRING(KEY,",",1) - 2) > 0 ?
SUBSTRING(KEY,FINDSTRING(KEY,",",1) + 2,LEN(KEY) - FINDSTRING(KEY,",",1) - 1) : ""
However, I am prompted with a parse error. What is the problem?
Update
I have followed markTheLiars' answer. The expression now looks like this:
KEY2 Replace 'KEY2' (DT_STR,100,1252)((FINDSTRING(KEY,",",1) - 2) > 0 ?
SUBSTRING(KEY,FINDSTRING(KEY,",",1) + 2,LEN(KEY) - FINDSTRING(KEY,",",1) - 1) : "") string [DT_STR] 1 1252 (ANSI - Latin I)
The expression compiles and runs, however I am prompted with the same error as the length does not change even though there is a cast.
Your cast is missing the "Code_page" parameter:
(DT_STR, 100, 1252) ((FINDSTRING(KEY,",",1) - 2) > 0 ? SUBSTRING(KEY,FINDSTRING(KEY,",",1) + 2,LEN(KEY) - FINDSTRING(KEY,",",1) - 1) : "")
1252 is the default value. See this answer for a much better explanation than I could give as to why it's important. See here for more info about casting/conversions.
It appears that the meta-data for that column is still set to be 1 character long. Right-click on the derived column transformation, select Show Advanced Editor, select the "Input and Output Properties". Expand "Derived Column Output" -> "Output Columns" -> Your column (Key2 in this instance I believe). Under "Data Type Properties" edit Length to be 100.
As was eventually discovered, the problem was not in this component but a previous component that was truncating the data before it ever reached this date flow component. In this case, the easiest way to determine which component is causing the truncation is to use the meta-data viewer, available by double-clicking on the flow pathway or right-clicking and selecting "Edit":
This will quickly give you info at a glance about the variables and their source components. Unfortunately it will not tell you exactly where the truncation occurred, but if your data flow component isn't too complicated you should still be able to find the problem area relatively quickly.
Related
I have a database(Source) which has a column named "country name" and few cells in it are empty & when I am transferring its data to another database (destination) it is also empty
I have tried to use derived column in which I had used REPLACE() but is didn't work I thought it would have been any "", " " or "\t" but it was neither of these then I thought may be it is NULL & I used ISNULL() but this also failed.
I assume you are trying to work out what the "empty" data in the source column actually is? If thats the case then just convert the value to its ascii value and look that value up in an ascii table to find out what it is.
If the empty data is actually more than one character you may need to extract each character individually using, for example, the MID function.
You did not specify which DB is in the source and which is in the target, and it has meaning.
Anyway, add this code to the Derived Column component:
(DT_WSTR,100)country_name == "" || ISNULL(country_name) == TRUE ? "new_value" : country_name
Results:
is there a way of replicating the below expression in SSIS Derived Column?
SELECT CASE
WHEN LEN(column1) > 8
THEN
column1
ELSE
REPLICATE('0', 18 - LEN(column1)) + column1
END AS column1
FROM myTable
I want to pad the value of column1 with 0 if the lenght of the value is less than 8 character
The SSIS expression language supports the ternary operator ? :
(LEN([column1]) > 8) ? column1 : replicate("0", (18 - LEN([column1]))) + [column1]
That expression ought to work but it doesn't because the REPLICATE call is going to provide metadata back stating it's 4k nvarchar characters (the limit). If you're deadset on getting it to work that way, comment and I'll hack the expression to size the output of replicate before concatenating with column1
An easier approach is to just always add 8 leading zeros to the expression and then slice it off from the right.
RIGHT(REPLICATE("0",8) + column1,8)
You might need 18 on the second as your example seemed to use 18 but the concept will be the same.
This is an old question, but I had the same question today and the below Derived Column Expression is what worked for me. It doesn't specifically "replicate" the SQL example in the original question (by using "REPLICATE") but I found the below to be a little simpler for padding a value using an SSIS Derived Column in a Data Flow, and it's a good example of a simple "If / Else" statement.
I am padding with 10 zeros because that's what the SQL does in the original question.
(LEN(column1) > 8) ? column1 : ("0000000000" + (column1))
Solution is here :
I have taken group as a column from column drop down and then derived these condition .
If group is "Cash Capex" then "Capex" AND where "Internal Opex" OR "External Opex" then "Opex" ELSE "Other".
[`Group] == "Cash Capex" ? "Capex" : ([Group] =="Internal Opex" ? "Opex" :([Group] =="External Opex" ? "Opex":"Other"))`
I am using Execute Process task in SSIS 2008 R2. I have a variable idVar which is of data type Int32. I need to pass this variable to property Arguments of the task so the process executable can take this variable as argument. I use expression to assign #idVar to Arguments.
Now the system says I need to convert Int to String so I used the following expression in the expression builder
(DT_STR, 10, 1252) #[User::IdVar]
It gives the following error:
Expression cannot be evaluated.
Additional information:
The expression "(DT_STR, 10, 1252) #[User:IdVar]" has a result type of "DT_STR",
which cannot be converted to a supported type.
(Microsoft.DataTransformationServices.Controls)
What is the correct way to type cast the number to string?
Cause of the issue:
Arguments property in Execute Process Task available on the Control Flow tab is expecting a value of data type DT_WSTR and not DT_STR.
SSIS 2008 R2 package illustrating the issue and fix:
Create an SSIS package in Business Intelligence Development Studio (BIDS) 2008 R2 and name it as SO_13177007.dtsx. Create a package variable with the following information.
Name Scope Data Type Value
------ ------------ ---------- -----
IdVar SO_13177007 Int32 123
Drag and drop an Execute Process Task onto the Control Flow tab and name it as Pass arguments
Double-click the Execute Process Task to open the Execute Process Task Editor. Click Expressions page and then click the Ellipsis button against the Expressions property to view the Property Expression Editor.
On the Property Expression Editor, select the property Arguments and click the Ellipsis button against the property to open the Expression Builder.
On the Expression Builder, enter the following expression and click Evaluate Expression. This expression tries to convert the integer value in the variable IdVar to string data type.
(DT_STR, 10, 1252) #[User::IdVar]
Clicking Evaluate Expression will display the following error message because the Arguments property on Execute Process Task expects a value of data type DT_WSTR.
To fix the issue, update the expression as shown below to convert the integer value to data type DT_WSTR. Clicking Evaluate Expression will display the value in the Evaluated value text area.
(DT_WSTR, 10) #[User::IdVar]
References:
To understand the differences between the data types DT_STR and DT_WSTR in SSIS, read the documentation Integration Services Data Types on MSDN. Here are the quotes from the documentation about these two string data types.
DT_STR
A null-terminated ANSI/MBCS character string with a maximum length of 8000 characters. (If a column value contains additional null terminators, the string will be truncated at the occurrence of the first null.)
DT_WSTR
A null-terminated Unicode character string with a maximum length of 4000 characters. (If a column value contains additional null terminators, the string will be truncated at the occurrence of the first null.)
Expression:
"Total Count: " + (DT_WSTR, 11)#[User::int32Value]
For reference, Int 32's range is (-2,147,483,648 to 2,147,483,647). Including the - sign for negatives, that makes up to 11 characters.
Expression:
"Total Count: " + (DT_WSTR, 5)#[User::Cnt]
I need to assign a formatted date to a column in a data flow. I have added a Derived shape and entered the following expression for a NEW column - Derived Column = "add as new column":
"BBD" + SUBSTRING((DT_WSTR,4)DATEADD("Day",30,GETDATE()),1,4) +
SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),6,2) +
SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),9,2)
The problem is that the Derived Column Transformation Editor automatically assigns a Data Type of Unicode string[DT_WSTR] and a length of "7". Howver, the length of a string is 11, therefore the following exception is thrown each time:
[Best Before Date [112]] Error: The "component "Best Before Date" (112)" failed
because truncation occurred, and the truncation row disposition on "output column
"Comments" (132)" specifies failure on truncation. A truncation error occurred
on the specified object of the specified component.
Does anyone know why the edit is insisting on a length of 7? I don't seem to be able to change this.
Many thanks,
Rob.
I can't understand why SSIS is measuring that column as only resulting in a seven character field - but to force it to provide an 11 character column for it, modify your expression slightly to this:
(DT_WSTR, 11)("BBD" + SUBSTRING((DT_WSTR,4)DATEADD("Day",30,GETDATE()),1,4) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),6,2) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),9,2))
What you want is:
"BBD" + (DT_WSTR,4)YEAR(DATEADD("Day",30,GETDATE()))
+ RIGHT("0" + (DT_WSTR,2)MONTH(DATEADD("Day",30,GETDATE())),2)
+ RIGHT("0" + (DT_WSTR,2)DAY(DATEADD("Day",30,GETDATE())),2)
The issue is in how you are converting your dates to a string. The calls to DATEADD return a full date & time. Next, you then have either (DT_WSTR,4) or (DT_WSTR,2) to convert that date into either a 4 or 2 character string. On my system, converting a datetime to a string defaults to "Aug 24 2011 4:18PM". So the first 4 characters gets you "Aug " and the first 2 characters gets you "Au". Then, you are extracting substrings using SUBSTRING. For your last two calls to SUBSTRING, you are starting the substring past the end of the 2 character string you converted the date into. This is why SSIS displays 7 characters:
"BBD" + "Aug " + "" + ""
3 + 4 + 0 + 0 = 7
It is better to use the built in functions to extract the Year, Month and Day from a datetime rather than converting to a string and then grabbing substrings. If you really wanted to use substrings, you would need to add a call to CONVERT to get the datetime to a specific string format, otherwise you will get whatever the default is for your locale setting in Windows. This could be different on each PC.
What release and service pack of SQL are you using?
I just tried this on my machine and had no problems changing the result size from 7 to 11. Is it possible that you have not installed all the service packs?
Are you replacing your existing field, and is that field possibly 7 chars long? The thing with the Derived Column Transform is that you can't change the field types (including length) of the existing fields.
Try to add a new field instead.
If that's not working, try adding an explicit cast around the whole expression.
(DT_WSTR,11)("BBD" + SUBSTRING((DT_WSTR,4)DATEADD("Day",30,GETDATE()),1,4) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),6,2) + SUBSTRING((DT_WSTR,2)DATEADD("Day",30,GETDATE()),9,2))
Right click on "Derived Column" open "Show Advanced Editor" Select "Input and output Properties" tab.
Got to "Derived column output" => "Output Columns" => "Derived Column 1" (added by you)
In right side panel go to "Data type Properties" section=> DataType=>
Select "String [DT_STR]
click OKImage showing steps
This will solve your problem.
I have an SSIS package with a Data Flow that takes an ADO.NET data source (just a small table), executes a select * query, and outputs the query results to a flat file (I've also tried just pulling the whole table and not using a SQL select).
The problem is that the data source pulls a column that is a Money datatype, and if the value is not zero, it comes into the text flat file just fine (like '123.45'), but when the value is zero, it shows up in the destination flat file as '.00'. I need to know how to get the leading zero back into the flat file.
I've tried various datatypes for the output (in the Flat File Connection Manager), including currency and string, but this seems to have no effect.
I've tried a case statement in my select, like this:
CASE WHEN columnValue = 0 THEN
'0.00'
ELSE
columnValue
END
(still results in '.00')
I've tried variations on that like this:
CASE WHEN columnValue = 0 THEN
convert(decimal(12,2), '0.00')
ELSE
convert(decimal(12,2), columnValue)
END
(Still results in '.00')
and:
CASE WHEN columnValue = 0 THEN
convert(money, '0.00')
ELSE
convert(money, columnValue)
END
(results in '.0000000000000000000')
This silly little issue is killin' me. Can anybody tell me how to get a zero Money datatype database value into a flat file as '0.00'?
I was having the exact same issue, and soo's answer worked for me. I sent my data into a derived column transform (in the Data Flow Transform toolbox). I added the derived column as a new column of data type Unicode String ([DT_WSTR]), and used the following expression:
Price < 1 ? "0" + (DT_WSTR,6)Price : (DT_WSTR,6)Price
I hope that helps!
Could you use a Derived Column to change the format of the value? Did you try that?
I used the advanced editor to change the column from double-precision float to decimal and then set the Scale to 2:
Since you are exporting to text file, just export data preformatted.
You can do it in the query or create a derived column, whatever you are more comfortable with.
I chose to make the column 15 characters wide. If you import into a system that expects numbers those zeros should be ignored...so why not just standardize the field length?
A simple solution in SQL is as follows:
select
cast(0.00 as money) as col1
,cast(0.00 as numeric(18,2)) as col2
,right('000000000000000' + cast( 0.00 as varchar(10)), 15) as col3
go
col1 col2 col3
--------------------- -------------------- ---------------
.0000 .00 000000000000.00
Simply replace '0.00' with your column name and don't forget to add the FROM table_name, etc..
It is good to use derived column and need to check the condition as well
pricecheck <=0 ? "0" + (DT_WSTR,10)pricecheck : (DT_WSTR,10)pricecheck
or alternative way is to use vb script
Ultimately what I ended up doing was using the FORMAT() function.
CAST(FORMAT(balance, '0000000000.0000') AS varchar(30)) AS "balance"
This does have some significant CPU performance impact (often at least an order of magnitude) due to the way SQL Server implements that function, but nothing worked easier, more correctly, or more consistently for me. I was working with less than 100,000 rows and the package executes no more than once an hour. Going from 100ms to 1000ms just wasn't a big deal in my situation.
The FORMAT() function returns an nvarchar(4000) by default, so I also cast it back to a varchar of appropriate size since my output file needed to be in Windows-1252 encoding. Transcoding text is much more obnoxious in SSIS than it has any right to be.