I'm trying to run a native query through JPA that uses a ':' character. The particular instance is using a MySQL user variable in the query:
SELECT foo, bar, baz,
#rownum:= if (#id = foo, #rownum+1, 1) as rownum,
#id := foo as rep_id
FROM
foo_table
ORDER BY
foo,
bar desc
The JPA code:
Query q = getEntityManager().createNativeQuery(query, SomeClass.class);
return q.getResultList();
However, this gives me an exception about not being allowed to follow a ':' with a space. I've tried escaping them with backslashes, I've tried escaping them by doubling them up. Is there any way to actually do this, or am I SOL?
I faced similar experience when using postgresql json function in native JPA query.
select * from component where data ::json ->> ?1 = ?2
JPA will throw error that i have not set the named parameter :json.
The solution:
"select * from component where data \\:\\:json ->> ?1 = ?2"
I'm not aware of a standard way to escape a colon character in a query that is obviously interpreted as a named parameter prefix, and thus confuses the query parser.
My suggestion would be to create and use SQL functions if possible. Depending on your provider, there might be other options (like using another character and substituting the chosen character by a : in an interceptor) but at least the previous suggestion would keep your JPA code portable across providers.
PS: if you're using Hibernate, there is a very old patch attached to HHH-1237.
Update: There is an "interesting" paragraph in the JPA 1.0 spec about named parameters and native queries:
3.6.3 Named Parameters
A named parameter is an identifier
that is prefixed by the ":" symbol.
Named parameters are case-sensitive.
Named parameters follow the rules for
identifiers defined in Section 4.4.1.
The use of named parameters applies to the Java Persistence query
language, and is not defined for
native queries. Only positional
parameter binding may be portably used
for native queries.
The parameter names passed to the
setParameter methods of the Query
API do not include the ":" prefix.
This won't really help you but your case is a strong hint that the ":" in native queries shouldn't even be considered (at least not without a way to escape it or disable it detection).
Try this:
String query =
"SELECT foo, bar, baz,
#rownum \\\\:= if (#id = foo, #rownum+1, 1) as rownum,
#id \\\\:= foo as rep_id
FROM
foo_table
ORDER BY
foo,
bar desc -- escape='\' ";
Query q = getEntityManager().createNativeQuery(query, SomeClass.class);
return q.getResultList();
Related
I need to variably group data points according to a user-defined stride and I would like to minimize the amount of raw SQL in my code that is interfacing with django. The question "Can I create view with parameter in MySQL?", seems like a great way to create a parameterized view, but it requires that I somehow set a variable before the main query is executed.
Thus, how would I set a parameter value in the mysql connection before any query is executed on the queryset? If I have the model Foo and I do the very basic:
Foo.objects.all()
How could I inject the following as a preamble to any code that is being emitted by the django ORM to SQL compiler ?
set #my_param := 5;
I think I have a solution which works for my current MySQL backend. I believe it may depend on unspecified order of evaluation and could break in the future version changes:
Foo.objects.extra(where=('5 = (select #my_param := 5)',)).all()[0]
The SQL that django then produces is:
SELECT `srvr_foo`.`id`
FROM `srvr_foo`
WHERE (5 = (select #my_param := 5))
LIMIT 1;
Imagine you have the following SQL query:
SELECT *
FROM foo
WHERE type = ?
AND subtype IN (?)
And you have the following possible data (we imagine that a user interface can set these data):
var Type int
var SubTypes []int
In the case of SubTypes, we are talking about a multiple choice selection.
Now, the following code won't work:
rows, err := sqldb.Query(`SELECT *
FROM foo
WHERE type = ?
AND subtype IN (?)`, Type, SubTypes)
Because the driver (at least the mysql driver used in this example) doesn't recognise a []slice. Typing to explode it (SubTypes...) doesn't work either, because A) you cannot have more than one exploded parameter and B) even if you could, your SQL only supports a single item ((?)).
However, there is a solution. First of all, since we can only have a single exploding parameter and no others, we should first put together our parameters in a single []slice:
var params []interface{}
params = append(params, Type)
for _, subtype := range SubTypes {
params = append(params, SubTypes)
}
Since the SQL will not expand on its own, let's expand that loop:
var params []interface{}
params = append(params, Type)
inCondition := ""
for _, subtype := range SubTypes {
params = append(params, SubTypes)
if inCondition != "" {
inCondition += ", "
}
inCondition += "?"
}
Assuming SubTypes contains []int{1,2,3}, inCondition should now contain ?, ?, ?.
We then combine that to our SQL statement and explode the argument:
sqlstr := fmt.Sprintf(`SELECT *
FROM foo
WHERE type = ?
AND subtype IN (%s)`, inCodition)
rows, err := sqldb.Query(sqlstr, params...)
Of course, it would be pretty cool, if you could simply pass []slices to your prepared statements, and the automatically expanded. But that might give some unexpected results if you are dealing with more 'unknown' data.
Prepared statements do not work that way, at least not in major DBMS I know. I mean, in Go, the support for prepared statements implemented by database/sql drivers is supposed to use the corresponding facility provided by the underlying DBMS (a driver might opt to simulate such support if it's not provided by the DB engine it interfaces with).
Now in all the DBMS-s I'm familiar with, the whole idea of prepared statement is that it's processed once by the DB engine and cached; "processed" here means syntax checking, compiling into some DB-specific internal representation and its execution plan figured out. As follows from the term "compiled", the statement's text is processed exactly once, and then each call to the prepared statement just essentially tells the server "here is the ID of that prepared statement I supplied you earlier, and here's the list of actual parameters to use for placeholders it contained". It's like compiling a Go program and then calling it several times in a row with different command-line flags.
So the solution you have come up with is correct: if you want to mess with the statement text between invocation then by all means use client-side text manipulations1 but do not attempt to use the result of it as a prepared statement unless you really intend to execute the resulting text more than once.
And to be may be more clear: your initial attempt to prepare something like
SELECT a, b FROM foo WHERE a IN (?)
supposedly fails at your attempt to supply a set of values for that IN (?) placeholder because commas which would be required there to specify several values are syntax, not parts of the value.
I think it should still be fine to prepare something like
SELECT a, b FROM foo WHERE a IN (?, ?, ?)
because it does not break that rule. Not that it's a solution for you…
See also this and this — studying the latter would allow you to play with prepared statements directly in the MySQL client.
1 Some engines provide for server-side SQL generation with subsequent execution of the generated text.
In my Django app, I need to generate a MySQL query like this:
SELECT * FROM player WHERE (myapp_player.sport_id = 4 AND (myapp_player.last_name LIKE 'smi%'))
UNION
SELECT * FROM player WHERE (myapp_player.sport_id = 4 AND (myapp_player.first_name LIKE 'smi%'));
I can't use Q objects to OR together the __istartswith filters because the query generated by the Django ORM does not use UNION and it runs at least 40 times slower than the UNION query above. For my application, this performance is unacceptable.
So I'm trying stuff like this:
Player.objects.raw("SELECT * FROM myapp_player WHERE (sport_id = %%s AND (last_name LIKE '%%s%')) UNION SELECT * FROM sports_player WHERE (sport_id = %%s AND (first_name LIKE '%%s%'))", (sport.id, qword, sport.id, qword))
I apologize for the long one-liner, but I wanted to avoid using a triple-quoted string while trying to debug this type of issue.
When I execute or repr this queryset object, I get exceptions like this:
*** ValueError: unsupported format character ''' (0x27) at index 133
That's a single-quote in single quotes, not a triple-quote. If I get rid of the single-quotes around the LIKE clauses, then I get a similar exception about the close-paren ) character that follows the LIKE clause.
Apparently Django and MySQL disagree on the correct syntax for this query, but is there a syntax that will work for both?
Finally, I'm not sure that my %%s syntax for string interpolation is correct, either. The Django docs suggest that I should be able to use the regular %s syntax in the arguments for raw(), but several online resources suggest using %%s or ? as the placeholder for string interpolation in raw SQL.
My sincere thanks for just a little bit of clarity on this issue!
I got it to work like this:
qword = word + '%'
Player.objects.raw("SELECT * FROM myapp_player WHERE (sport_id = %s AND (last_name LIKE %s)) UNION SELECT * FROM myapp_player WHERE (sport_id = %s AND (first_name LIKE %s))", (sport.id, qword, sport.id, qword))
Besides the fact that %s seems to be the correct way to parameterize the raw query, the key here was to add the % wildcard to the LIKE clause before calling raw() and to exclude the single quotes from around the LIKE clause. Even though there are no quotes around the LIKE clause, quotes appear in the query ultimately sent to the MySQL sever.
I just asked an SQL related question, and the first answer was: "This is a situation where dynamic SQL is the way to go."
As I had never heard of dynamic SQL before, I immediately searched this site and the web for what it was. Wikipedia has no article with this title. The first Google results all point to user forums where people ask more or less related questions.
However, I didn't find a clear definition of what a 'dynamic SQL' is. Is it something vendor specific? I work with MySQL and I didn't find a reference in the MySQL handbook (only questions, mostly unanswered, in the MySQL user forums).
On the other hand, I found many references to stored procedures. I have a slightly better grasp of what stored procedures are, although I have never used any. How are the two concepts related? Are they the same thing or does one uses the other?
Basically, what is needed is a simple introduction to dynamic SQL for someone who is new to the concept.
P.S.: If you feel like it, you may have a go at answering my previous question that prompted this one: SQL: How can we make a table1 JOIN table2 ON a table given in a field in table1?
Dynamic SQL is merely where the query has been built on the fly - with some vendors, you can build up the text of the dynamic query within one stored procedure, and then execute the generated SQL. In other cases, the term merely refers to a decision made by code on the client (this is at least vendor neutral)
Other answers have defined what dynamic SQL is, but I didn't see any other answers that attempted to describe why we sometimes need to use it. (My experience is SQL Server, but I think other products are generally similar in this respect.)
Dynamic SQL is useful when you are replacing parts of a query that can't be replaced using other methods.
For example, every time you call a query like:
SELECT OrderID, OrderDate, TotalPrice FROM Orders WHERE CustomerID = ??
you will be passing in a different value for CustomerID. This is the simplest case, and one that can by solved using a parameterized query, or a stored procedure that accepts a parameter, etc.
Generally speaking, dynamic SQL should be avoided in favor of parameterized queries, for performance and security reasons. (Although the performance difference probably varies quite a bit between vendors, and perhaps even between product versions, or even server configuration).
Other queries are possible to do using parameters, but might be simpler as dynamic SQL:
SELECT OrderID, OrderDate, TotalPrice FROM Orders
WHERE CustomerID IN (??,??,??)
If you always had 3 values, this is as easy as the first one. But what if this is a variable-length list? Its possible to do with parameters, but can be very difficult. How about:
SELECT OrderID, OrderDate, TotalPrice FROM Orders WHERE CustomerID = ??
ORDER BY ??
This can't be substituted directly, you can do it with a huge complicated CASE statement in the ORDER BY explicitly listing all possible fields, which may or may not be practical, depending on the number of fields available to sort by.
Finally, some queries simply CAN'T be done using any other method.
Let's say you have a bunch of Orders tables (not saying this is great design), but you might find yourself hoping you can do something like:
SELECT OrderID, OrderDate, TotalPrice FROM ?? WHERE CustomerID = ??
This can't be done using any other methods. In my environment, I frequently encounter queries like:
SELECT (programatically built list of fields)
FROM table1 INNER JOIN table2
(Optional INNER JOIN to table3)
WHERE (condition1)
AND (long list of other optional WHERE clauses)
Again, not saying that this is necessarily great design, but dynamic SQL is pretty much required for these types of queries.
Hope this helps.
Dynamic SQL is simply a SQL statement that is composed on the fly before being executed. For example, the following C# (using a parameterized query):
var command = new SqlCommand("select * from myTable where id = #someId");
command.Parameters.Add(new SqlParameter("#someId", idValue));
Could be re-written using dynamic sql as:
var command = new SqlCommand("select * from myTable where id = " + idValue);
Keep in mind, though, that Dynamic SQL is dangerous since it readily allows for SQL Injection attacks.
Dynamic SQL is a SQL built from strings at runtime. It is useful to dynamically set filters or other stuff.
An example:
declare #sql_clause varchar(1000)
declare #sql varchar(5000)
set #sql_clause = ' and '
set #sql = ' insert into #tmp
select
*
from Table
where propA = 1 '
if #param1 <> ''
begin
set #sql = #sql + #sql_clause + ' prop1 in (' + #param1 + ')'
end
if #param2 <> ''
begin
set #sql = #sql + #sql_clause + ' prop2 in (' + #param2 + ')'
end
exec(#sql)
It is exactly what Rowland mentioned. To elaborate on that a bit, take the following SQL:
Select * from table1 where id = 1
I am not sure which language you are using to connect to the database, but if I were to use C#, an example of a dynamic SQL query would be something like this:
string sqlCmd = "Select * from table1 where id = " + userid;
You want to avoid using dynamic SQL, because it becomes a bit cumbersome to keep integrity of the code if the query get too big. Also, very important, dynamic SQL is susceptible to SQL injection attacks.
A better way of writing the above statement would be to use parameters, if you are using SQL Server.
Rowland is correct, and as an addendum, unless you're properly using parameters (versus just concatonating parameter values inline from provided text, etc.) it can also be a security risk. It's also a bear to debug, etc.
Lastly, whenever you use dynamic SQL unwisely, things are unleashed and children are eaten.
To most databases, every SQL query is "dynamic" meaning that it is a program that is interpreted by the query optimiser given the input SQL string and possibly the parameter bindings ("bind variables").
Static SQL
However, most of the time, that SQL string is not constructed dynamically but statically, either in procedural languages like PL/SQL:
FOR rec IN (SELECT * FROM foo WHERE x = 1) LOOP
-- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ "static SQL"
..
END LOOP;
Or in client / host languages like Java, using JDBC:
try (ResultSet rs = stmt.executeQuery("SELECT * FROM foo WHERE x = 1")) {
// "static SQL" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
..
}
In both cases, the SQL string is "static" in the language that embeds it. Technically, it will still be "dynamic" to the SQL engine, which doesn't know how the SQL string is constructed, nor that it was a static SQL string.
Dynamic SQL
Sometimes, the SQL string needs to be constructed dynamically, given some input parameters. E.g. the above query might not need any predicate at all in some cases.
You might then choose to proceed to constructing the string dynamically, e.g. in PL/SQL:
DECLARE
TYPE foo_c IS REF CURSOR;
v_foo_c foo_c;
v_foo foo%ROWTYPE;
sql VARCHAR2(1000);
BEGIN
sql := 'SELECT * FROM foo';
IF something THEN
sql := sql || ' WHERE x = 1'; -- Beware of syntax errors and SQL injection!
END IF;
OPEN v_foo_c FOR sql;
LOOP
FETCH v_foo_c INTO v_foo;
EXIT WHEN v_foo_c%NOTFOUND;
END LOOP;
END;
Or in Java / JDBC:
String sql = "SELECT * FROM foo";
if (something)
sql += " WHERE x = 1"; // Beware of syntax errors and SQL injection!
try (ResultSet rs = stmt.executeQuery(sql)) {
..
}
Or in Java using a SQL builder like jOOQ
// No syntax error / SQL injection risk here
Condition condition = something ? FOO.X.eq(1) : DSL.trueCondition();
for (FooRecord foo : DSL.using(configuration)
.selectFrom(FOO)
.where(condition)) {
..
}
Many languages have query builder libraries like the above, which shine most when doing dynamic SQL.
(Disclaimer: I work for the company behind jOOQ)
Is it something vendor specific?
The SQL-92 Standard has a whole chapter on dynamic SQL (chapter 17) but it only applies to FULL SQL-92 and I know of no vendor that has implemented it.
I think what's meant is that you should build the query dynamically before executing it. For your other questions this means that you should select the table name you need first and the use your programming language to build a second query for doing what you want (what you want to do in the other question isn't possible directly like you want).
The following doesn't work, but something like this is what I'm looking for.
select *
from Products
where Description like (#SearchedDescription + %)
SSRS uses the # operator in-front of a parameter to simulate an 'in', and I'm not finding a way to match up a string to a list of strings.
There are a few options on how to use a LIKE operator with a parameter.
OPTION 1
If you add the % to the parameter value, then you can customize how the LIKE filter will be processed. For instance, your query could be:
SELECT name
FROM master.dbo.sysobjects
WHERE name LIKE #ReportParameter1
For the data set to use the LIKE statement properly, then you could use a parameter value like sysa%. When I tested a sample report in SSRS 2008 using this code, I returned the following four tables:
sysallocunits
sysaudacts
sysasymkeys
sysaltfiles
OPTION 2
Another way to do this that doesn't require the user to add any '%' symbol is to generate a variable that has the code and exceute the variable.
DECLARE #DynamicSQL NVARCHAR(MAX)
SET #DynamicSQL =
'SELECT name, id, xtype
FROM dbo.sysobjects
WHERE name LIKE ''' + #ReportParameter1 + '%''
'
EXEC (#DynamicSQL)
This will give you finer controller over how the LIKE statement will be used. If you don't want users to inject any additional operators, then you can always add code to strip out non alpha-numeric characters before merging it into the final query.
OPTION 3
You can create a stored procedure that controls this functionality. I generally prefer to use stored procedures as data sources for SSRS and never allow dynamically generated SQL, but that's just a preference of mine. This helps with discoverability when performing dependency analysis checks and also allows you to ensure optimal query performance.
OPTION 4
Create a .NET code assembly that helps dynamically generate the SQL code. I think this is overkill and a poor choice at best, but it could work conceivably.
Have you tried to do:
select * from Products where Description like (#SearchedDescription + '%')
(Putting single quotes around the % sign?)
Dano, which version of SSRS are you using? If it's RS2000, the multi-parameter list is
not officially supported, but there is a workaround....
put like this:
select *
from tsStudent
where studentName like #SName+'%'
I know this is super old, but this came up in my search to solve the same problem, and I wound up using a solution not described here. I'm adding a new potential solution to help whomever else might follow.
As written, this solution only works in SQL Server 2016 and later, but can be adapted for older versions by writing a custom string_split UDF, and by using a subquery instead of a CTE.
First, map your #SearchedDescription into your Dataset as a single string using JOIN:
=JOIN(#SearchedDedscription, ",")
Then use STRING_SPLIT to map your "A,B,C,D" kind of string into a tabular structure.
;with
SearchTerms as (
select distinct
Value
from
string_split(#SearchedDescription, ',')
)
select distinct
*
from
Products
inner join SearchTerms on
Products.Description like SearchTerms.Value + '%'
If someone adds the same search term multiple times, this would duplicate rows in the result set. Similarly, a single product could match multiple search terms. I've added distinct to both the SearchTerms CTE and the main query to try to suppress this inappropriate row duplication.
If your query is more complex (including results from other joins) then this could become an increasingly big problem. Just be aware of it, it's the main drawback of this method.