I'm developing a webservice using eclipseLink 2.4.1, JPA 2.0.1 and other supporting tools. I have created a table which has a column of date type. I want to get records of a particular date. I'm using date_format of mysql and getting following error:
java.lang.IllegalArgumentException: An exception occurred while creating a query in EntityManager:
Exception Description: Syntax error parsing[select o from Clients o where FUNCTION('date_format',{o.effectiveStartDate},{d,'dd-mm-yyyy'})='27-05-2014']
[90, 111] The JDBC escape format does not start with either 'd', 't' or 'ts'.
[110, 110] The JDBC escape format is missing the close quote.
Even I tried to call date_format directly as :
Select o from Clients o where date_format(o.effectiveStartDate '%d-%m-%Y'})='27-05-2014'
Then I get syntax error parsing. invalid token [(].
I tried with namedQuery also but didn't get success.
How to call such function JPQL.
What is "{o.effectiveStartDate}" trying to do ? Remove the curly brackets or it will try to interpret that as a JDBC escape syntax date (which it isn't).
JPA 2.1 'FUNCTION' syntax would be
FUNCTION("date_format", o.effectiveStartDate, "the_date_format")
where "the_date_format" matches what MySQL accepts for its second arg as per http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_date-format
That of course requires you to use a JPA2.1 compliant implementation; your implementation may have some other vendor-specific FUNC or otherwise which is not portable and I've no time for when there's a standard variant that should do what you need
Related
I've got a confusing issue on Airflow which I don't understand.
I have a SQL scripts folder at DML/analytics/my_script.sql. The MySQL operator works perfectly in normal circumstances, but does not when I try to call it from a Python operator as follows. This is necessitated by needing to pass in XCOM values from another task:
def insert_func(**kwargs):
run_update = MySqlOperator(
sql='DML/analytics/my_script.sql',
task_id='insert_func',
mysql_conn_id="bi_mysql",
params={
"table_name": table_name,
'ts': kwargs['task_instance'].xcom_pull(key='return_value',task_ids='get_existing_data')
},
)
run_update.execute(context=kwargs['task_instance'])
with DAG("my_dag", **dag_params) as dag:
with TaskGroup(group_id='insert') as insert:
get_existing_data = PythonOperator(
task_id='get_existing_data',
python_callable=MySQLGetRecord,
op_kwargs={
'target_db_conn_id':'bi_mysql',
'target_db':'analytics',
'sql': f'SELECT invoice_date FROM analytics.{table_name} ORDER BY 1 DESC'
}
),
insert = PythonOperator(
task_id='insert',
python_callable=insert_func
)
get_existing_data >> insert_func
The error I get is: MySQLdb._exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DML/analytics/my_script.sql' at line 1")
Clearly it is trying to run the literal string passed in the sql parameter rather than using it as a file location. Why is this happening? Again, this works if I move the run_update task to the my_dag with clause, but I need to do it this way to get the XCOM value from get_existing_data, correct...?
When you are using operator as normal (e.g to be used by Airflow) then Airflow is responsible for the whole task lifecycle. This means Airflow handles the templating, executing pre_execute(), executing execute(), executing on_faulure/retries etc...
What you did is using operator inside operator -> PythonOperator that contains MySqlOperator. In this case the inner operator (MySqlOperator) is just a regular Python class. While it's called Operator - it's is not a "real" Operator.
You are not enjoying any of the lifecycle steps as you might expect.
You might have already realised it as by your own example you specifically triggered the execute():
run_update.execute(context=kwargs['task_instance'])
Notice you didn't need to do this for the PythonOperaor.
You can see in the code base that Airflow invokes render_templates before it invokes pre_execute() and before it invokes execute().
This means that if you want the MySqlOperator to be templated you need to call the function that does the templating before you invoke the execute()
That said - I strongly encourage you - Do not use operator inside operator.
From your code I don't see reason why you can't just use MySqlOperator directly without the PythonOperaor but should there be a reason the proper way to handle it is to create a CustomMySqlOperator that handles the logic you seek. By doing so you will not have problems with using .sql files.
I have the following query that works in MYSQL
$query->whereRaw('
ST_Distance_Sphere(
point(address->>"$.longitude", address->>"$.latitude"),
point(?, ?)
) * .000621371192 < ?
', [$longitude, $latitude, $distance]);
When i deployed to heroku im getting this error
SQLSTATE[42703]: Undefined column: 7 ERROR: column "$.longitude" does not exist
LINE 3: point(address->>"$.longitude", address->>"$.latitu...
Im using postgres on heroku and i know the issue is something to do with the json operator.
What is the $.<json key> syntax called? and how do i convert this raw sql query to postgres?
In the worse case i leaning on switching my database to mysql if i cant solve this by Wednesday as it wont be a big deal
Postgres uses double quotes as delimiter for columns, so you must use single quotes to indicate strings. Laravel will take care of it
st_distance_sphere(
st_point(address->>'$.longitude',address->>'$.latitude'),st_point(?, ?)
)
I'm using mysql 5.7 in two enviroments (staging and prod)
both with this same version
but when I try this query:
SELECT * FROM `EVENT_ONDEMAND_MODULE_CONTENT`
WHERE JSON_EXTRACT(EXTRA_INFO, '$.UPLOAD_ID') = 'Jh24ACh59sJ6mSSpQKXORJVscmy2iD3Dy1B0115L01hbI';
In staging that works, but in production I got this error:
SQLSTATE[22032]: <>: 3141 Invalid JSON text in argument 1 to function json_extract: "Invalid escape character in string."
{"UPLOAD_ID":"Jh24ACh59sJ6mSSpQKXORJVscmy2iD3Dy1B0115L01hbI","POLICY":"signed","FILE_NAME":"VIDEO 10s.mp4","FILE_MIME_TYPE":"video\/mp4","READY":false}
I copy the json from prod to staging and the query worked too.
Anybody knows if have something I must do to this function Json_extract work in prod??
Thanks in advance!
I'm trying to do a simple query as prepared statement but have no success. Here is the code:
package sqltest;
use DBI;
DBI->trace(2);
my $dbh = DBI->connect('dbi:mysql:database=test;host=***;port=3306','the_username', '****');
my $prep = 'SELECT me.id, me.session_data, me.expires FROM sys_session me WHERE me.id = ?';
$dbh->{RaiseError} = 1;
my $sth = $dbh->prepare($prep);
$sth->bind_param(1, 'session:06b6d2138df949524092eefc066ee5ab3598bf96');
$sth->execute;
DBI::dump_results($sth);
The MySQL server responds with a syntax error near '''.
The output of the DBI-trace shows
-> bind_param for DBD::mysql::st (DBI::st=HASH(0x21e35cc)~0x21e34f4 1 'session:06b6d2138df949524092eefc066ee5ab3598bf96') thr#3ccdb4
Called: dbd_bind_ph
<- bind_param= ( 1 ) [1 items] at perl_test_dbi_params.pl line 10
[...]
>parse_params statement SELECT me.id, me.session_data, me.expires FROM sys_session me WHERE me.id = ?
Binding parameters: SELECT me.id, me.session_data, me.expires FROM sys_session me WHERE me.id = '
[...]
DBD::mysql::st execute failed: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''' at line 1
So to me, it looks like the statement does not get prepared as it should.
When I send the query without the parameter it works as expected.
What do I miss here?
DBI version is DBI 1.637-ithread, MySQL version is 5.5.57-0+deb8u1
Tested with Windows perl 5, version 26, subversion 1 (v5.26.1) built for MSWin32-x86-multi-thread-64int
and Ubuntu perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-linux-gnu-thread-multi
Edit1:
for context: I noticed the problem while using Catalyst with Catalyst::Plugin::Session::Store::DBIC. Here, the id-column is a Varchar(72) type, that holds a session-id.
Edit2:
DBD::mysql version is 4.043
Binding via $sth->execute('session:foo'); results in the same problem
Binding via $sth->bind_param('session:foo', SQL_VARCHAR); results in the same problem
Binding a numeric field does work, but only with explicit type definition $sth->bind_param(1, 1512407082, SQL_INTEGER);
Edit3:
I found the time to do some more testing, but not with no satisfying results:
I was able to test with an older server and it worked. The versions of DBI and DBD::mysql are the same, but I found the server using MySQL 5.5 client, reported in the DBI-trace as MYSQL_VERSION_ID 50557, whereas both my original test servers using MySQL 5.7 MYSQL_VERSION_ID 50720 and MYSQL_VERSION_ID 50716
with $dbh->{mysql_server_prepare} = 1;it works! Maybe this helps someone who finds this q., but I would rather now the real cause of the problem
From your trace log can be seen that question mark placeholder (?) was replaced by DBD::mysql by one apostrophe ('). So it is pure DBD::mysql bug. At the first glance it does not make sense at all... as placeholder is replaced by parameter which is put into two apostrophes.
Relevant code which is doing this placeholder replacement can be found there: https://metacpan.org/source/MICHIELB/DBD-mysql-4.043/dbdimp.c#L784-786
*ptr++ = '\'';
ptr += mysql_real_escape_string(sock, ptr, valbuf, vallen);
*ptr++ = '\'';
So question is, can above C code result in just one apostrophe in *ptr buffer? And answer is yes, when mysql_real_escape_string() returns integer of same value as pointer size minus one -- to simulate numeric operation of decrement by one, when both apostrophes are written to same position in *ptr buffer.
And can this happen? Yes, it can, because Oracle changed API of mysql_real_escape_string() C function in MySQL 5.7.6 client library:
https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-6.html#mysqld-5-7-6-feature
Incompatible Change: A new C API function, mysql_real_escape_string_quote(), has been implemented as a replacement for mysql_real_escape_string() because the latter function can fail to properly encode characters when the NO_BACKSLASH_ESCAPES SQL mode is enabled. In this case, mysql_real_escape_string() cannot escape quote characters except by doubling them, and to do this properly, it must know more information about the quoting context than is available. mysql_real_escape_string_quote() takes an extra argument for specifying the quoting context. For usage details, see mysql_real_escape_string_quote().
And documentation for mysql_real_escape_string() from MySQL 5.7.6 version says:
https://dev.mysql.com/doc/refman/5.7/en/mysql-real-escape-string.html
Return Values: The length of the encoded string that is placed into the to argument, not including the terminating null byte, or -1 if an error occurs.
Therefore if you enable NO_BACKSLASH_ESCAPES SQL mode on you MySQL server, then mysql_real_escape_string() from MySQL 5.7.6 client cannot work and return error, therefore -1 casted to unsigned long. unsigned long is on both 32bit and 64bit x86 platform of same size as pointer, therefore above C code from DBD::mysql drivers results in one apostrophe character.
Now I fixed this problem for DBD::MariaDB driver (fork of DBD::mysql) in following pull request: https://github.com/gooddata/DBD-MariaDB/pull/77
So DBD::MariaDB would be compatible also when compiled with MySQL 5.7 client library.
After some testing, I came to the conclusion that this seems to be a compatibility problem between DBD::mysql and MySQL client 5.7 (and/or MySQL server 5.5).
At least, I found a solution for Ubuntu 16 (xenial), so for others, that may run into the same issue:
downgrade to MySQL 5.6 like described here. For me, installing libmysqlclient-dev without the server/client was sufficient
reinstall DBD::MySQL sudo cpanm --reinstall DBD::mysql, so that it gets build with the now installed MySQL 5.6
I will file an issue at the DBD::mysql GitHub and will update this answer, if there are any news regarding this problem.
An alternative solution, that also worked for me:
let the server prepare your statement $dbh->{mysql_server_prepare} = 1;
I have a JSONB column which holds arrays of strings, eg: ["foo", "bar"]
I want to write the jOOQ equivalent of:
SELECT * FROM sometable WHERE somecolumn ?| <mylist>
...where should be bound to a java List of string tag names.
There doesn't appear to be any direct support for ?| in jOOQ 3.8. I have looked at binding to raw sql in a condition but I'm not quite sure the syntax; it gets even worse if trying to use the ? postgres operator which conflicts with the binding expression.
UPDATE: the stacktrace with 3.8.3
I stripped this down to a minimal test. When adding a condition like this using jOOQ 3.8.3:
query.addConditions(DSL.condition("sometable.tags ?| array['sometag']"));
Produces a stacktrace like this:
Caused by: org.postgresql.util.PSQLException: No value specified for parameter 1.
at org.postgresql.core.v3.SimpleParameterList.checkAllParametersSet(SimpleParameterList.java:228)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:163)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:622)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:472)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:465)
at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:194)
at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:269)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:348)
... 36 more
An issue related to jOOQ parsing ?| and similar operators as bind variables has been addressed in jOOQ 3.8.3: https://github.com/jOOQ/jOOQ/issues/5307
JDBC limitation
Note, in addition to the above, there is also a JDBC limitation that I have documented in a separate question. In jOOQ, you can work around this JDBC limitation by specifying:
Settings settings = new Settings().withStatementType(StatementType.STATIC_STATEMENT);
See also: http://www.jooq.org/doc/latest/manual/sql-execution/statement-type
Or, alternatively, by falling back to using the jsonb_exists_any() function instead of the ?| operator.