I am trying to run some hive functinos in Dbeaver tool but it always giving the below error
What I am trying to run is for example :
SELECT from_unixtime(unix_timestamp('17-Sep-21', 'dd-MMM-yy'),'dd-MMM-yy');
error I am getting :
SQL Error [30328] [50000]: TEIID30328 Remote
org.teiid.api.exception.query.ExpressionEvaluationException
TEIID30328 Unable to evaluate timestampadd(SQL_TSI_SECOND,
unix_timestamp('17-Sep-21', 'dd-MMM-yy'), {ts'1969-12-31 19:00:00.0'})
TEIID30341 Function impala.unix_timestamp is marked in the function
metadata as a function that must be evaluated at the source.
The full stack trace should contain more information as to why it can't evaluate. Most of the later versions of Teiid even for pushdown functions will pre-evaluate them if the surrounding expression cannot be pushed.
Related
I've got a confusing issue on Airflow which I don't understand.
I have a SQL scripts folder at DML/analytics/my_script.sql. The MySQL operator works perfectly in normal circumstances, but does not when I try to call it from a Python operator as follows. This is necessitated by needing to pass in XCOM values from another task:
def insert_func(**kwargs):
run_update = MySqlOperator(
sql='DML/analytics/my_script.sql',
task_id='insert_func',
mysql_conn_id="bi_mysql",
params={
"table_name": table_name,
'ts': kwargs['task_instance'].xcom_pull(key='return_value',task_ids='get_existing_data')
},
)
run_update.execute(context=kwargs['task_instance'])
with DAG("my_dag", **dag_params) as dag:
with TaskGroup(group_id='insert') as insert:
get_existing_data = PythonOperator(
task_id='get_existing_data',
python_callable=MySQLGetRecord,
op_kwargs={
'target_db_conn_id':'bi_mysql',
'target_db':'analytics',
'sql': f'SELECT invoice_date FROM analytics.{table_name} ORDER BY 1 DESC'
}
),
insert = PythonOperator(
task_id='insert',
python_callable=insert_func
)
get_existing_data >> insert_func
The error I get is: MySQLdb._exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DML/analytics/my_script.sql' at line 1")
Clearly it is trying to run the literal string passed in the sql parameter rather than using it as a file location. Why is this happening? Again, this works if I move the run_update task to the my_dag with clause, but I need to do it this way to get the XCOM value from get_existing_data, correct...?
When you are using operator as normal (e.g to be used by Airflow) then Airflow is responsible for the whole task lifecycle. This means Airflow handles the templating, executing pre_execute(), executing execute(), executing on_faulure/retries etc...
What you did is using operator inside operator -> PythonOperator that contains MySqlOperator. In this case the inner operator (MySqlOperator) is just a regular Python class. While it's called Operator - it's is not a "real" Operator.
You are not enjoying any of the lifecycle steps as you might expect.
You might have already realised it as by your own example you specifically triggered the execute():
run_update.execute(context=kwargs['task_instance'])
Notice you didn't need to do this for the PythonOperaor.
You can see in the code base that Airflow invokes render_templates before it invokes pre_execute() and before it invokes execute().
This means that if you want the MySqlOperator to be templated you need to call the function that does the templating before you invoke the execute()
That said - I strongly encourage you - Do not use operator inside operator.
From your code I don't see reason why you can't just use MySqlOperator directly without the PythonOperaor but should there be a reason the proper way to handle it is to create a CustomMySqlOperator that handles the logic you seek. By doing so you will not have problems with using .sql files.
I'm getting the following error message when trying to reflect any of my SQL views:
sqlalchemy/dialects/mysql/reflection.py", line 306, in _describe_to_create
buffer.append(" ".join(line))
TypeError: sequence item 2: expected str instance, bytes found
I have tried using both the autoload_with and autoload=True options in my select query constructor to no avail.
I have the appropriate permissions on my view. My query is pretty simple:
company_country = Table('company_country', metadata, autoload_with=engine)
query = select(company_country.c.country)
return query
I've tried the inspect utility and it does not list my SQL view, nor does the reflecting all tables described below the views section on this page: https://docs.sqlalchemy.org/en/14/core/reflection.html#reflecting-views
I'm using version SQLAlchemy->1.4.32, Python 3.x and mySQL 8.0.28 on Mac if that's any help
I should add that I can query my SQL views using the text() constructor but it would be far more preferable to use select() if possible.
Any tips appreciated
I was using the mysql-connector client for interop with other code, but after switching to the mysqlclient, I was able to reflect the views.
https://docs.sqlalchemy.org/en/14/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb
I'm using Simba ODBC to create a connection with Google Big Query and using SSIS (Visual Studio 2019) to read and write information on Big Query. The connection works fine and when I use the ODBC Source with the query option, I'm able to get data from Big Query and used it inside SSIS. But when I use the list of tables, I get an error as below:
Exception of HRESULT: 0xC0014020
Error in Data Flow Task[ODBC Source [100]]: SQLSTATE: 42000, Message: [Simba][BigQuery] (70) Invalid query: Invalid dataset ID ""TEST"". Dataset IDs must be alphanumeric (plus underscores and dashes) and must be at most 1024 characters long.
I believe that this happens because the list of tables appear between ("), instead of (`).
Print of table list
The same happens when I use the ODBC Destination. Is there a way to change the format in which the table list appears ?
Obs.: On the Visual Studio 2015 this table list comes with (`) and I can connect with big query just fine.
I can see that the tool is sending "TEST" as the dataset, however, depending on if Visual Studio is using StandardSQL or LegacySQL, the dataset should be specified as:
# LegacySQL
FROM [myproject:TEST.TABLE_TEST]
# StandardSQL
FROM `myproject:TEST.TABLE_TEST`
I was wondering if Visual Studio accepts a custom query or can be parameterized to remove the quotes. If this doesn't help, could you please share the query that cause the error? I understand that there is a query option (I'm not familiar with Visual Studio) and it is not clear for me the exact moment when the tool returns the error, screenshot without sensitive information would be appreciated.
UPDATE:
You can review the following checkpoints that could help to verify that the Simba driver is correctly set up and it is not the cause of the reported error:
Installation. Check that you are using the last version of the driver. The last version usually contains improvements on the driver.
ODBC Configuration. For example, the Step 13 of the link you will be able to see a drop-down list with the datasets available and select one as the default. If you don't have issues is this step, then the issue could be in the tool that uses the ODBC connection.
Language Dialect. In here you case change between StandardSQL or LegacySQL as needed, for example, you can force your tool to use LegacySQL and use the characters [ and ] that I explained above.
Connection String. If your tool allows to use a string with the connection, you might want to use it and explicitly indicating the default Dataset (among other driver options).
According to MSDN the syntax to convert a number to decimal in VBA is
CDec(expression)
I often use conversion function in access SQL as well. eg Clng, Cint etc...
However when I use CDec I sometime get this error
Compile error:Wrong number of arguments or invalid property assignment
Consider:
In the immediate window
? cdec(round(0.00023,4))
0.0002
? CDec(Round(0.12345678+0.00000001,6))
0.123457
? CDec(Round(0.12345679,6))
0.123457
? CDec(Round(0.12345679,6),10)
This gives the above error (ie as expected as no parameter is allowed)
In a SQL query column
CDec(Round(0.12345678,6))
This gives the above error.
CDec(Round(0.12345678,6),2)
This WORKS and give the answer 0. What does the parameter do!?
CDec(Round(0.12345678,6),2,1)
This gives the above error.
I guess when a function is called from access SQL it is using different code to the one used by VBA. However, I'm stuck and don't understand.
Help!
I'm using MS Access 2013: Build: 15.0.4727.1003 32bit
Harvey
Dennis Wilmar of MS tells, that this is a confirmed bug from Access 2003 - not up to be removed:
Error message when you use the CDec() function in an Access query
The suggested work-around is to wrap CDec in a custom function:
Function NewCDec(MyVal)
NewCDec = CDec(MyVal)
End Function
So here we have an error I keep getting in my SSIS package but I can't see what is wrong with the statement. I have even tried another sql statement from a project that works and it still raises the error.
The system is VS 2005 running 64 bit debugger, on XP machine. The project has amongst other things a script task then a sql task, the script task outputs the month value to a variable (Dts.Variables("monthName").Value = month), which I then use to create dynamic table name in SQL statement. I haven't got to the excel sheet bit yet as I am trying to get the sql task stage working.
So i have a variable at package level called SQLTableCreate, and in that I have the properties set to:
Evaluate as Expression = true
Expression = "Create Table "+ #[user::monthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
And when I build the package I get:
Nonfatal errors occurred while saving the package:
Error at Package: The variable "user::monthName" was not found in the Variables collection. The variable might not exist in the correct scope.
Error at Package: Attempt to parse the expression ""Create Table "+ #[user::MonthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
" failed and returned error code 0xC00470A6. The expression cannot be parsed. It might contain invalid elements or it might not be well-formed. There may also be an out-of-memory error.Error at Package: The expression for variable "SQLTableCreate" failed evaluation. There was an error in the expression.
There is also a default SQL statement for the variable SQLTableCreate, which uses the current excel connection manager table name. When I put my dynamic statement in the expression section of properties it fills the value and valuetype property of the SQLTableCreate variable with the message:
The expression for variable "SQLTableCreate" failed evaluation. There was an error in the expression.
It's exactly as the error says
The variable "user::monthName" was not found in the Variables collection
Things in SSIS are case sensitive and Variables are one of those things. Make your expression
"Create Table "+ #[User::monthName]+"(Column1 DATETIME,Column2 NVARCHAR(255),Column3 NVARCHAR(255),Column4 NVARCHAR(255),Column5 NVARCHAR(255),Column6 NVARCHAR(255),Column7 NVARCHAR(255),Column8 NVARCHAR(255),Column9 NVARCHAR(255),Column10 NVARCHAR(255))"
Also, I hope this table design is just a sample and not real. Lack of column names and strong data types is technical debt you don't need to incur at this stage.