SSIS The expression for variable 'Variable' failed evaluation. There was an error in the expression - ssis

So here we have an error I keep getting in my SSIS package but I can't see what is wrong with the statement. I have even tried another sql statement from a project that works and it still raises the error.
The system is VS 2005 running 64 bit debugger, on XP machine. The project has amongst other things a script task then a sql task, the script task outputs the month value to a variable (Dts.Variables("monthName").Value = month), which I then use to create dynamic table name in SQL statement. I haven't got to the excel sheet bit yet as I am trying to get the sql task stage working.
So i have a variable at package level called SQLTableCreate, and in that I have the properties set to:
Evaluate as Expression = true
Expression = "Create Table "+ #[user::monthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
And when I build the package I get:
Nonfatal errors occurred while saving the package:
Error at Package: The variable "user::monthName" was not found in the Variables collection. The variable might not exist in the correct scope.
Error at Package: Attempt to parse the expression ""Create Table "+ #[user::MonthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
" failed and returned error code 0xC00470A6. The expression cannot be parsed. It might contain invalid elements or it might not be well-formed. There may also be an out-of-memory error.Error at Package: The expression for variable "SQLTableCreate" failed evaluation. There was an error in the expression.
There is also a default SQL statement for the variable SQLTableCreate, which uses the current excel connection manager table name. When I put my dynamic statement in the expression section of properties it fills the value and valuetype property of the SQLTableCreate variable with the message:
The expression for variable "SQLTableCreate" failed evaluation. There was an error in the expression.

It's exactly as the error says
The variable "user::monthName" was not found in the Variables collection
Things in SSIS are case sensitive and Variables are one of those things. Make your expression
"Create Table "+ #[User::monthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
Also, I hope this table design is just a sample and not real. Lack of column names and strong data types is technical debt you don't need to incur at this stage.

Related

Airflow MySQL operator trying to execute script path string as SQL, rather than using template

I've got a confusing issue on Airflow which I don't understand.
I have a SQL scripts folder at DML/analytics/my_script.sql. The MySQL operator works perfectly in normal circumstances, but does not when I try to call it from a Python operator as follows. This is necessitated by needing to pass in XCOM values from another task:
def insert_func(**kwargs):
run_update = MySqlOperator(
sql='DML/analytics/my_script.sql',
task_id='insert_func',
mysql_conn_id="bi_mysql",
params={
"table_name": table_name,
'ts': kwargs['task_instance'].xcom_pull(key='return_value',task_ids='get_existing_data')
},
)
run_update.execute(context=kwargs['task_instance'])
with DAG("my_dag", **dag_params) as dag:
with TaskGroup(group_id='insert') as insert:
get_existing_data = PythonOperator(
task_id='get_existing_data',
python_callable=MySQLGetRecord,
op_kwargs={
'target_db_conn_id':'bi_mysql',
'target_db':'analytics',
'sql': f'SELECT invoice_date FROM analytics.{table_name} ORDER BY 1 DESC'
}
),
insert = PythonOperator(
task_id='insert',
python_callable=insert_func
)
get_existing_data >> insert_func
The error I get is: MySQLdb._exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DML/analytics/my_script.sql' at line 1")
Clearly it is trying to run the literal string passed in the sql parameter rather than using it as a file location. Why is this happening? Again, this works if I move the run_update task to the my_dag with clause, but I need to do it this way to get the XCOM value from get_existing_data, correct...?
When you are using operator as normal (e.g to be used by Airflow) then Airflow is responsible for the whole task lifecycle. This means Airflow handles the templating, executing pre_execute(), executing execute(), executing on_faulure/retries etc...
What you did is using operator inside operator -> PythonOperator that contains MySqlOperator. In this case the inner operator (MySqlOperator) is just a regular Python class. While it's called Operator - it's is not a "real" Operator.
You are not enjoying any of the lifecycle steps as you might expect.
You might have already realised it as by your own example you specifically triggered the execute():
run_update.execute(context=kwargs['task_instance'])
Notice you didn't need to do this for the PythonOperaor.
You can see in the code base that Airflow invokes render_templates before it invokes pre_execute() and before it invokes execute().
This means that if you want the MySqlOperator to be templated you need to call the function that does the templating before you invoke the execute()
That said - I strongly encourage you - Do not use operator inside operator.
From your code I don't see reason why you can't just use MySqlOperator directly without the PythonOperaor but should there be a reason the proper way to handle it is to create a CustomMySqlOperator that handles the logic you seek. By doing so you will not have problems with using .sql files.

Data Flow Task - Set two User Date Variables as Parameters

I am creating an SSIS package that will run each month. This particular stored procedure needs to run for one week at a time since the data returned is very large.
I have set up my stored procedure to with two parameters: #StartDT and #EndDT. I created two SSIS variables: StartDT and Wk1EndDT (I'll create the other start and end dates for the weeks once I get this one working).
StartDT has this expression:
(DT_DATE)((DT_WSTR, 4)YEAR(DATEADD("mm", -1, GETDATE())) + "-" +RIGHT("0" + (DT_WSTR,2)MONTH(DATEADD("mm", -1, GETDATE())),2)+"-01")
Wk1EndDT has this expression:
DATEADD("DD",7, #[User::StartDT])
I'm using a DataFlow task with a SQL command text of:
EXECUTE dbo.uspUploadWk1 ?,?
When I go to preview the results, I receive the following error message:
There was an error displaying the preview.
No value given for one or more required parameters. (Microsoft SQL Server Native Client 11.0)
I have the parameters set like this:
I am not sure why this isn't working. I've searched all over and have not found an answer. I am using Visual Studio 2015.
Assuming an OLE DB Connection Manager, the Mappings tab should be using a zero based ordinal system on the Parameters column. Yes, it defaults to naming them as Parameter0, Parameter1, etc but for an OLE DB connection manager, you'll use the ordinal position of the question marks, ?, starting at zero.
For ODBC, it becomes a 1 based counting but still uses ? as the parameter place holder.
ADO.NET uses named parameters so we'd match EXECUTE dbo.uspUploadWk1 #Parameter0, #Parameter1 but the ADO.NET source component doesn't support parameterization
Reference on parameters and mapping for Execute SQL Task but the syntax remains the same for Data Flow Task components

SSIS error -1073548535 DTS_E_SQLTASK_ERRORASSIGINGVALUETOVAR

I have implemented a SSIS package using BIDS 2008 which has an Execute SQL task to get Max of record from a table. I am getting an error -1073548535 which I believe equates to DTS_E_SQLTASK_ERRORASSIGINGVALUETOVAR.
I have checked the datatypes and not sure what the problem is
Following is the query
select max(PICS8Id) as PICS8Id from [dbo].[Industries]
PICS8Id is float in the database
I have mapped it to the following variable in Result Set
The maxPICS8Id is of type int32. I have tried Uint32 but that doesnt work either

SSIS - Use Derived Column to Cast String to Float

I'm having a problem getting data from a .CSV into a column of datatype FLOAT. I've tried to link it directly and also use the Data Conversion Task, but (in both cases) it kept telling me that it couldn't convert:
Error: 0xC02020C5 at DC_Weekly_Cost_Target csv to FatzWklyCst_Target, Data Conversion [156]: Data conversion failed while converting column "Target" (22) to column "Copy of Target" (163). The conversion returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
My research led me to using the Derived Column Transformation Editor. I found a few websites that walked me through how properly use the "Expression" portion:
Above is how I'm attempting to transform the strings (Target and Waste) into datatype Float. I'm not receiving an error message when using the Editor (i.e. It will let me clik OK without an error), however, I am receiving an error when I attempt to run the package:
Error: 0xC0049064 at DC_Weekly_Cost_Target csv to FatzWklyCst_Target, Map Target in correct datatype 1 1 [222]: An error occurred while attempting to perform a type cast.
Error: 0xC0209029 at DC_Weekly_Cost_Target csv to FatzWklyCst_Target, Map Target in correct datatype 1 1 [222]: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "component "Map Target in correct datatype 1 1" (222)" failed because error code 0xC0049064 occurred, and the error row disposition on "output column "Target_Float" (227)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
Error: 0xC0047022 at DC_Weekly_Cost_Target csv to FatzWklyCst_Target, SSIS.Pipeline: SSIS Error Code DTS_E_PROCESSINPUTFAILED. The ProcessInput method on component "Map Target in correct datatype 1 1" (222) failed with error code 0xC0209029 while processing input "Derived Column Input" (223). The identified component returned an error from the ProcessInput method. The error is specific to the component, but the error is fatal and will cause the Data Flow task to stop running. There may be error messages posted before this with more information about the failure.
This is my first time using the Derived Column Transformation Editor. Does anyone see what I'm doing incorrectly? Or, do you have any suggestions as to what may be the best approach to getting data from a .csv file into a column of datatype float? I appreciate any help that anyone can give me.
You have tried a reasonable approach but something in the data is blowing it up - possibly "invalid" characters e.g. $ or ,
I would replace the Derived Column transformation with a Script Task. There you can leverage the .NET Framework e.g. Try ... Catch, TryParse, Regex. You can debug your code line-by-line to inspect the rows with errors. You can also use Reflection to factor your conversion code as a function that you call for each column passed into the Script Task.
PS: your destination is irrelevant.

SSIS (2008R2) import from mssql to mysql failing due to a date column

I have an oledb connection to mssql and an ado.net destination (with odbc driver used) to mysql. The tables are exectly the same and all the columns are working bar one.
The error message received is:
[ADO NET Destination [325]] Error: An exception has occurred during data insertion, the message returned from the provider is: Unable to cast object of type 'System.DateTime' to type 'System.Char[]'.
I've seen similar questions on other data types but the resolution of changing to string does not work here. If I convert to string (has to be length 29 otherwise the conversion step fails) I get the following error message:
[ADO NET Destination [325]] Error: An exception has occurred during data insertion, the message returned from the provider is: ERROR [HY000] [MySQL][ODBC 5.1 Driver][mysqld-5.5.15]Incorrect datetime value: '2011-03-21 11:23:48.573000000' for column 'LastModificationDate' at row 1
Other potentially relevant details:
connection driver- {MySQL ODBC 5.1 Driver}
script run before dataflow - set sql_mode='STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ANSI_QUOTES'
Other datetime columns are working
This column has a reasonably high proportion of nulls
mssql spec: [LastModificationDate] [datetime] NULL
mysql spec: LastModificationDate datetime NULL
Has anyone had experience with this issue and could provide some advice on resolving it?
Can you try converting it to string on sql server side in your query using:
convert(char(10),LastModificationDate,111)+' '+convert(char(8),LastModificationDate,108)
This works for me all the time.
I got the same big headache this week. I tried many ways. Thanks God, finnally, one of them worked. Hope it could help you a little bit.
For some columns with the data type of Int, datetime, decimal....,here, I identified as ColumnA, and I used it as datetime type.
1.in Data Flow Source, use SQL Command to retrieve data. Sth like select isnull(ColumnA,'1800-01-01') as ColumnA, C1, C2, ... Cn from Table
Make sure to use Isnull function for all columns with the datatype mentioned before.
2.Excute the SSIS pkg. It should work.
3.Go back to Control Flow, under the data flow task, add SQL Task control to replace the data back. I mean, update the ColumnA from '1800-01-01' to null again.
That works for me. In my situation, I cannot use ignore failure option. Because if I do, I will lose thousands rows of data.