So I am currently working on a migration from an old Advantage database server to SQL 2005 using SSIS 2008. One of the columns in the old Advantage database is a MEMO type. By default this translates to a DT_TEXT column. Well in the new database I do not need this large of field, but can limit it to something such as VARCHAR(50). I successfully set up a derived column transformation to convert this with the following expression:
(DT_STR,50,1252)[ColumnName]
Now I want to go a step further and replace all NULL values with an empty string. This would seem easy enough using an ISNULL([ColumnName])?"":(DT_STR,50,1252)[ColumnName] expression, but the problem is that the OLE DB Destination contains the following error
Cannot convert between unicode and non-unicode strings...
So apparently the whole ISNULL expression converts the data type to Unicode string [DT-WSTR]. I have tried a variety of casts upon the whole expression or different parts, but I cannot get the data type to match what I need it.
First, is it possible to convert the DT_TEXT type directly to unicode? From what I can tell, the casts don't work that way. If not, is there a way to get an expression to work so that NULL values get converted to empty strings?
Thank you for all your help!
Give this a try in your derived column.
(DT_STR,50,1252) (ISNULL(ColumnName) ? "" : (DT_STR,50,1252) ColumnName)
It includes an additional type cast with the Conditional (?:) in parentheses to ensure the desired processing sequence. I think your original expression was implicitly casting to DT_WSTR because the "" defaults to DT_WSTR. With this new version, you force the cast to DT_STR after the expression is evaluated.
I figured something out that works. It may not be the best solution, but it will work for my situation.
From my OLE DB source I first did a Derived Column. This I used the ISNULL which ended up converting it to a DT_WSTR unicode type. although I could not get any casts to get it back to the type required, I then added a Data Conversion transformation in-between the Derived Column and the OLE DB Destination. This would take the input string and convert it back to a DT_STR. This all feels a little annoying converting so many times, but the column does not contain any funky information that I should have to worry about, so I suppose it will work.
Thanks for all those who pondered the solution, and if you find some awesome way to tackle it, I would be more than interested.
Related
I'm using SSIS to separate good data from unusable date. In order to do that I used derived columns, script task and conditional split where I assigned certain conditions. One of the conditions I need to apply is that none of the numbers in one column cannot be negative. I'm guessing that the best way to solve this would be using conditional split, but I cannot get it to work. I'm new to SSIS, so any help would be appreciated.
You'd have an Expression like
[MyCaseSensitiveColumnName] < 0
and then name the output path something like BadData_NegativeValue
From the comments
that is what I did before, but I'm getting an error saying that The data types "DT_WSTR" and "DT_I4" are incompatible for binary operator ">"
That error message indicates that you are attempting to compare a unicode string (DT_WSTR) and an integer (DT_I4) and that the expression language does not allow it.
To resolve this type incompatibility, you would need to first convert the value of MyCaseSensitiveColumnName from DT_WSTR to an integer.
I'd likely add a Derived Column Component to my data flow and create a new column called MyCaseSensitiveColumnNameAsInteger with an expression like
(DT_I4) [MyCaseSensitiveColumnName]
Now, that may be perilous depending on the quality of your source data. I don't know why you are pulling numeric data in as a string. If there could be non whole numbers in the data set, then we will need to check before making the cast. If there are NULLs in that dataset, those too may cause issues.
That would result in our conditional split check becoming
[MyCaseSensitiveColumnNameAsInteger] < 0
I have a table where the date column name is defined as proddate and defined as DATE.
Here is the code I write the value into table:
cmd.Parameters.Add("#Mydate", MySqlDbType.DateTime).Value = Convert.ToDateTime(MyArray[4]).ToString();
This gives a result of 0000-00-00
When I change it to
cmd.Parameters.Add("#Mydate", MySqlDbType.DateTime).Value = Convert.ToDateTime(MyArray[4]);
The result is correct. eg: 2013-11-14
However few lines down I have this code
cmd1.Parameters.Add("#date", MySqlDbType.DateTime).Value = Convert.ToDateTime(MyArray[4].ToString());
This gives no error. I get the correct result in table
And few lines after I have this code in the same method:
cmd3.Parameters.Add("#Mydate", MySqlDbType.DateTime).Value = MyArray[4].ToString();
This gives no error too
The last two lines I did the mistake for testing. But the columns of 2 last exemples are defined as DATE format
Any idea ? Or Am I welcome in the mystery world of Mysql ?
PS: I use MYSQL CONNECTOR 6.7.4.0 .NET3.5 and C# 2008 Express. In my server MYSQL is 5.0
In the first version, you're converting your value to a DateTime and then to a string, using the default format for a DateTime value. MySQL is then trying to parse that, and presumably failing.
In the second version, you're parsing your value as a DateTime, and supplying it to MySQL directly. This is the best approach, although you need to be careful about the input format you're using.
The third version is like the second, just with an explicit ToString call. If MyArray is of type string[], that's basically a no-op.
The fourth version is providing the string input directly to MySQL, which is presumably able to parse it itself. I would suggest avoiding that form though - parse it in the .NET code, where you have much more control.
trying to format number (decimal), but it shows me #Error on production server:
=FormatNumber(First(Fields!SumWithMailDelivery.Value, "document"), 0)
but on developers server it works fine.
Maybe someone know, how can i resolve this?
p.s. without formatting it works on production server fine too.
As #IanPreston says, it is most likely a type conversion error. I imagine your production data has some invalid characters for that column or some Null columns that make the numeric conversion fail.
You can use the Val function to do the conversion. Val differs from other numeric conversion functions in that it won't error when the string to be converted isn't numeric - it just does the best job it can.
So try an expression like this for the Value property:
=IIF(Fields!SumWithMailDelivery.Value Is Nothing,
Nothing,
IIF(IsNumeric(Fields!SumWithMailDelivery.Value),
Val(Fields!SumWithMailDelivery.Value),
Fields!SumWithMailDelivery.Value)
)
then use N0 as the Format property to format it as numeric if possible.
This formula will:
Leave the cell as Nothing if the field is Null
Convert to numeric and use the appropriate format if possible
Otherwise just output whatever is in the field
I have numeric data with have value 0.546, 0.456 in the database. When I try to export these values to flat file using flat file manager the zero value is truncated and only .546,.456 is shown in the flat file.
The value other than zero doesn't have this problem.
I tried using data conversion but of no use
Try Data Conversion transform - Data Type: decimal [DT-DECIMAL) : Scale 3.
Good find, Prakash!
I'm afraid you'll have to first convert you numeric data to DT_WSTR. It has to be Derived Column transformation, not Data Conversion, because you'd get the same result. In expression you need to prepend converted number with 0. Don't know about negative numbers, but maybe they will have correct format (you need to test it).
floor(decimal_column)==0 ? "0" + (DT_WSTR,10)decimal_column : (DT_WSTR,10)decimal_column
I know that's not what you expected, but I had the same problem the other day and was unable to finde better solution :).
I have a source flat file with values such as 24.209991, but they need to load to SQL Server as type money. In the DTS (which I am converting from), that value comes across as 24.21. How do I convert that field in SSIS?
Right now, I am just changing the type from DT_STR to DT_CY, and it gives a run error of 'Data conversion failed. The data conversion for column "Col003" returned status value 2 and status text "The value could not be converted because of a potential loss of data.".'
Do I use a Data Conversion task? And then what?
I've also tried setting the source output column to DT_NUMERIC, and then convert that to DT_CY, with the same result.
I've also tried using Derived Columns, casting the DT_STR field Col003 to (DT_NUMERIC,10,2)Col003 and then casting that to (DT_CY)Col003_Numeric. That's getting a cast error.
The flat file defaults to all fields being DT_STR. Use the Advanced option on editing the connection to have the numeric field as float (DT_R4). Then, in the advanced editing of the Flat File Source (on the Data Flow tab), set that output column to money (DT_CY).
Then, the field will convert without any additional conversions. The issue was leaving the source file definition as DT_STR.
If you don't have any null value use Data Conversion, and make sure you don't have any funny character (e.g. US$200 produce error)
If you have null or empty fields in your field and you are using Flat file source, make sure that you tick "Return null value from source.."
Another trick I have used is something like: (taxvalue != "" ? taxvalue : NULL(DT_WSTR,50)). in Derived Column transformation (you can just replace the field)
Generally SSIS doesn't convert empty strings to money properly.
For some reason in my scenario, the OLE DB Destination actually was configured to accept a DT_CY. However, casting to this format (no matter the length of the input and destination data, and no matter wether or not the data was NULL when it arrived) always caused the same issue.
After adding data viewers, I can conclude that this has something to do with the locale. Here in Denmark, we use comma (,) as decimal delimiters and dots (.) as thousands-delimiters, instead of the opposite.
This means that a huge number like 382,939,291,293.38 would (after the conversion to DT_CY) look like 382.939.291.293,38. Even though I highly doubted that it could be the issue, I decided to do the opposite of what I originally had intended.
I decided to go to the advanced settings of my OLE DB Destination and change the DT_CY column's type to DT_STR instead. Then, I added a Derived Column transformation, and entered the following expression to transform the column before the data would arrive at the destination.
REPLACE(SUBSTRING(Price, 2, 18), ",", ".") where Price was the column's name.
To my big surprise, this solved the problem, since I figured out that my OLE DB Destination was now sending the data as a string, which the SQL Server understood perfectly fine.
I am certain that this is a bug! I was using SQL Server 2008, so it might have been solved in later editions. However, I find it quite critical that such an essential thing is not working correctly!