Consider the following statement in a Derived Column Transformation:
Derived Column Name:
EFFECTIVE_DATE
Expression:
TRIM([EFFECTIVE DATE]) == "" ? (DT_WSTR,255)NULL(DT_WSTR,255) : [EFFECTIVE DATE]
I read this as:
"If Trim of Effective Date is an empty string then assign EFFECTIVE_DATE NULL converted to a Unicode string of length 255. Otherise, assign EFFECTIVE DATE its current value."
I assume that this is what the code is doing, but I am confused about the syntax of the following:
(DT_WSTR,255)NULL(DT_WSTR,255)
I only expected:
(DT_WSTR,255)NULL
I thought that the leading (DT_WSTR,255) was a cast of the value that immediately follows, a NULL value. Why is there another (DT_WSTR,255) immeditely after. What I am missing?
That expression, per se, looks a bit redundant, as:
NULL(DT_WSTR,255)
...means "Generate a NULL value of type DT_WSTR with length 255".
So it looks like:
(DT_WSTR,255)NULL(DT_WSTR,255)
...means "Cast a NULL value of type DT_WSTR with length 255 to type DT_WSTR with length 255".
So it therefores seems at face value as if the type cast actually does nothing. However, if you leave it off, you may run into a problem as described here, which is that you'll experience this error at runtime:
For operands of the conditional operator, the data type DT_STR is supported only for input columns and cast operations. The expression "FINDSTRING([string-col],"",1) == 0 ? [string-col] : NULL(DT_STR,255,1252)" has a DT_STR operand that is not an input column or the result of a cast, and cannot be used with the conditional operation. To perform this operation, the operand needs to be explicitly cast with a cast operator.
This seems like an oddly arbitrary limitation, but it exists, so you need the cast.
Related
I am running a query on a db table which is returning one record when I expect it to return no records.
SELECT yeargroupID FROM tbl_yeargroup WHERE yeargroup='S' AND schoolID=2.
The yeargroup field is a tinyint field. Thefore the WHERE clause is looking the letter 'S' in the numeric field, so should not find anything. Yet it returns the record with the yeargroup = 0, and yeargroupID=17 (the bottom record in the table)
I'm confused as to why it is returning this record and how to avoid it.
Thanks
This logic, as you have pointed out, is comparing a number and a string:
WHERE yeargroup = 'S'
Handling such situations is an important part of most SQL compilers, and it is well documented. The solution is to implicitly convert values to "conforming" types. This is sad. My preference would be for the compiler to generate an error and force the user to use correct types. I find that implicit conversion creates more problems than it solves.
In any case, the rules in this case are pretty simple. The string is converted to an integer. But, how is a string with no digits converted? Well, the rule in MySQL is that the leading digits are converted to a number. And if there are none, the value is 0. So, this turns into:
where yeargroup = 0
You can see the results more clearly if you run:
select 'S', 'S' + 0
Note that most databases would return an error in this case (a type conversion error). But even those would accept the string if it looked like a number, so this would be allowed:
where yeargroup = '5'
What is the proper solution? Never mix types. Do not construct queries by munging constant values. Instead, queries from an application should always be using parameters.
I have created an SSIS package where two columns of type varchar(1) have to be mapped to columns of Integer. I have this working using a Derived Column and giving both fields a type cast of (DT_I4). However, I discovered in the complete data set there are records with no value in these two fields and so I have to Type Cast AND add a condition in expression to default to "0" if null.
So far I have tried the following but are not valid
(IsNull[Notes Taken])?(DT_I4)"0":[Notes Taken]
(DT_I4)(IsNull[Notes Taken])?"0":[Notes Taken]
How do I create this expression properly
The most simple solution is to use REPLACENULL function like:
REPLACENULL([Notes Taken], "0")
And then - cast it to DT_I4. This function replaces the logic you are devising with conditional operator.
Your both formulas have errors. The most prominent - ISNULL is a function and needs parenthesis around its arguments, ISNULL([Notes Taken]), brackets only define a dataflow column. See MS Docs.
Then, your first expression
(IsNull[Notes Taken])?(DT_I4)"0":[Notes Taken]
Possibly the field [Notes Taken] is not matching data type of the DT_I4 which is the datatype of the first argument of ? : operator.
Your second expression
(DT_I4)(IsNull[Notes Taken])?"0":[Notes Taken]
Applies the data cast to the logical function ISNULL, not to the complete expression. You should put the parenthesis around the complete ? : operator like:
(DT_I4)(IsNull([Notes Taken])?"0":[Notes Taken])
I'm having a little trouble with an SSIS expression where, in a Derived Column Transformation Data Flow Task, I am attempting to grab a 6 character substring from a string input, casting the derived columns value to NULL if it doesn't exist. This is the code I am using, with line breaks and indentation added for readability:
KeyValueLength == -2 ?
NULL(DT_STR,6,65001) :
(
KeyValueLength == -1 ?
(DT_STR,6,65001)RTRIM(SUBSTRING(StringInput,KeyValueStart,999)) :
(DT_STR,6,65001)SUBSTRING(StringInput,KeyValueStart,KeyValueLength)
)
(For reference, when KeyValueLength is -2 the key value is not found, when it is -1 then it is found at the end of StringInput, any other number and it is found in the middle of StringInput. This code works for other key values I'm getting that are casting to DT_I4 and DT_DECIMAL)
Individually, the following three expressions do not generate an error:
NULL(DT_STR,6,65001)
(DT_STR,6,65001)RTRIM(SUBSTRING(StringInput,KeyValueStart,999))
(DT_STR,6,65001)SUBSTRING(StringInput,KeyValueStart,KeyValueLength)
But when put together in that nested conditional above, I get the following error when trying to save the window:
For operands of the conditional operator, the data type DT_STR is
supported only for input columns and cast operations. The expression
"KeyValueLength == -2 ? NULL(DT_STR,6,65001) : (KeyValueLength == -1 ?
(DT_STR,6,65001)RTRIM(SUBSTRING(StringInput,KeyValueStart,999)) :
(DT_STR,6,65001)SUBSTRING(StringInput,KeyValueStart,KeyValueLength))"
has a DT_STR operand that is not an input column or the result of a
cast, and cannot be used with the conditional operation. To perform
this operation, the operand needs to be explicitly cast with a cast
operator.
I'm having a little trouble figuring out exactly what the issue is here. That error message suggests it's to do with the use of conditionals, but I'm not seeing the problem.
So, in the infinite wisdom of Microsoft, this is null as a DT_STR and perfectly valid as a direct value assignment:
NULL(DT_STR,6,65001)
But if you want to assign that value in a conditional where all eventual conditions must be the same type you have to do this:
(DT_STR,6,65001)NULL(DT_STR,6,65001)
The same does not apply for other types, where something like NULL(DT_I4) is valid irrespective of whether it is directly assigned or assigned via condition. SMH
Can anyone explain why query:
select rma.id, history_transactions.reference
from history_transactions
join rma on history_transactions.reference = rma.id
Returns:
id | reference
100144 | 100144
102299 | 102299a
100316 | 100316AFEN1
Can't get it to show only 100% matched, so only first row. If someone can explain why it happens it would be great.
Evidently rma.id column is numeric (integer), while the reference field is textual, since it contains text as well.
As MySQL documentation on Type Conversion in Expression Evaluation describes, if you compare text with number, the comparison is one as floating point numbers, meaning that the reference field is converted to a number.
MySQL converts a string to number by evaluating its characters left to right, as long as the charters can be interpreted as a number. If it encounters a character that cannot be evaluated as a number, then MySQL stops the evaluation and returns the previous characters as the numeric value.
In case of the 2nd record, the letter a is the 1st character that cannot be evaluated as number, therefore the numeric value of '102299a' string is 102299. The same logic applies to the 3rd record.
To force MySQL to return exact matches only, explicitly convert rma.id to string using cast() or convert() functions in the query. This way the comparison would be done as strings, not as floating point numbers.
The reason is most probably implicit type conversion. My guess is that id field is of integer type, whereas reference field is of varchar type. Hence, when comparing MySQL converts varchar to a number. So, e.g. value '10299a' is converted to 10299 and is then compared to the corresponding value of id field.
Live demo of the issue
Implicit datatype conversion.
I suspect the id column is declared as numeric datatype. Most likely INT.
The reference column is declared as character type. Most likely VARCHAR.
To do the equality comparison, MySQL can't compare a "string" with a "number".
So MySQL is implicitly doing a conversion of the string value into a number, and then doing the comparison of the numeric values.
Other databases would throw an error given a string that isn't a valid number.
But MySQL allows the conversion (without error or warning.)
As a demonstration, these expressions cause MySQL to do implicit conversion of a string to a number:
SELECT '123ABC' + 0
, '4D5E6F' + 0
, 'G7H8I9' + 0
Given your values, for example
SELECT '100316AFEN1' + 0
We see that MySQL returns a numeric value of 100316 which it uses to compare.
When SUM is used in query on field of type VARCHAR in MySql database, does SUM automatically convert it into number ?
I tried this by using
SELECT SUM(parametervalue) FROM table
and it reveals that MySql returns the sum although I expected to throw it an error as "parametervalue" field is of VARCHAR type
MySQL does silent conversion for a string in a numeric context. Because it expects a number for the sum(), MySQL simply does the conversion using the leading "numbers" from a string. Note that this include decimal points, minus sign, and even e representing scientific notation. So, '1e6' is interpreted as a number.
In code, I personally would make the conversion explicit by adding 0:
SELECT SUM(parametervalue + 0) FROM table
Ironically, the cast() might return an error if the string is not in a numeric format, but this doesn't return an error in that case.