Jpa Criteria equals Predicate for a table's String property (with newLine character '\n') doesn't work when in MySQL Workbench it works - mysql

as the title suggests I am having a problem trying to use JPA Criteria with Spring Boot for a specific case.
Generally everything works but trying to search for stored data with a String property having newLine character embedded ( \n ) doesn't seem to work.
I am able to Get the data, edit them, save them through my front end, create new with multiple lines etc. but trying to search for them when for example a column is equals with 'hello\nworld' it wont work even though running this query in MySQL Workbench works returning the desired data :
select * from kerkinidb.ct_thhlastika_press_threats where description_en = 'hello\nworld';
To clarify, the way I do the search is by waiting in a Get request an argument called search which has all the properties that the user filtered. I am matching it with a Regex (which also has inside the Java 8.0 new Regex \\\\R for matching with multilines (and it works) ) then I am giving the the Service layer the Search Criteria that I matched which then passes to the Jpa Criteria Repository to parse them and generating the Predicates (matching again with Regex and \\\\R to create a final Predicate with OR and ANDs for the filtering) then triggering the query, then making another query called count to implement Pagination and finally mapping to a custom object and returning it.
I debugged every step and the final Predicate does generate the query I want, but the db doesn't return the expected data. So I am really confused since as I said the query does work in MySQL Workbench.
This is an example of the logging (I turned Spring Boot logging for MySQL logs on) generated when the Request is being triggered (in this case the stored data I have in my Table ct_thhlastika_press_threats in column description_en is a\ns\ndd so I am searching for this one as you can see instead of the example I said earlier hello\nworld :
2019-02-12 16:01:01.929 DEBUG 18368 --- [nio-8080-exec-2] org.hibernate.SQL : select ctthhlasti0_.id as col_0_0_, ctthhlasti0_.act_code as col_1_0_, ctthhlasti0_.description_en as col_2_0_, ctthhlasti0_.remarks as col_3_0_ from ct_thhlastika_press_threats ctthhlasti0_ where 1=1 and ctthhlasti0_.description_en=? order by ctthhlasti0_.id asc limit ?
2019-02-12 16:01:01.933 TRACE 18368 --- [nio-8080-exec-2] o.h.type.descriptor.sql.BasicBinder : binding parameter [1] as [VARCHAR] - [a\ns\ndd]
2019-02-12 16:01:01.944 DEBUG 18368 --- [nio-8080-exec-2] org.hibernate.SQL : select count(ctthhlasti0_.id) as col_0_0_ from ct_thhlastika_press_threats ctthhlasti0_ where 1=1 and ctthhlasti0_.description_en=?
2019-02-12 16:01:01.944 TRACE 18368 --- [nio-8080-exec-2] o.h.type.descriptor.sql.BasicBinder : binding parameter [1] as [VARCHAR] - [a\ns\ndd]
2019-02-12 16:01:01.946 TRACE 18368 --- [nio-8080-exec-2] o.h.type.descriptor.sql.BasicExtractor : extracted value ([col_0_0_] : [BIGINT]) - [0]
For anyone interested to look farther into the code you could find it Github repo. The project is for my University thesis. I made comments for adding to the Regexes (both of them) for ctThhlastikaPressThreats Controller and Repository. The db (the standard as well as the secondary for testing) need to be generated (can be generated by changing the auto-dll in application.properties).
Update
I tried using System.lineSeparator() (as suggested by #Hermann Steidel in the answer bellow) replacing the text newLine \n. But even though in the logs we can see now clearly that the value for the equals Predicate has line separations it still doesn't return the data from the DB.
Farther Explanation of my implementation
A correct Get request for the dynamic searching is like this :
http://localhost:8080/v1/ctThhlastikaPressThreats/search?search=descriptionEn~hello\nworld;#&size=10&page=0&sort=Asc
As you can see I am using a path variable called search (which has all the properties filtering requested from the user) along with 3 more for the size, page and sort.
For the search variable I am using 3 distinct characters to implement OR and AND Predicates for the final query. Those are the ~ which is to know that the property before it needs to use an equal predicate, the ; which is to be able to have multiple values for each property requested by the user and finally the # which triggers the end of this property's filtering.
Those 3 characters are being Regexed in two places. In the Controller and in the SearchRepository of (for example since we specifically started with this one -> the CtThhlastasikaPressThreats)
Finally in the SearchRepository you can see that I am triggering 2 queries, one is to get the filtered data from the db and another is to get the count of the data for Pagination purposes while also in the end mapping to a custom DTO.
Steps to reproduce :
After generating the db, change for the 2 Regexes of CtThhlastikaPressThreats. For the Controller to be (\w+?)(~|<|>)([(!-/.\\\\R 0-9\p{L});]+)?# and for the SearchRepository to be ([(!-/.\\\\R 0-9\p{L})]+).
Then you can use the example of the request I have above when having in the db saved for the specific table and for column descriptionEn with value of hello\nworld for example or whatever value you put (also change it into the request).
Final thing I tried but wasn't the solution :
Put in the CtThhlastikaPressThreatsSearchRepository in the method search (it is after line 61) above the :
predicate = builder.equal(root.get(param.getKey()), match.toString());
Make it as :
match = match.toString().replace("\\n", System.lineSeparator());
predicate = builder.equal(root.get(param.getKey()), match.toString());
This will basically change the value from being hellow\nworld to become hello\r\nworld so I guess it still isn't the desired solution. Thinking that in the db it is stored as \n for the lineSeparators.
Now in the logs you can see that when you trigger the Get Request again the VARCHAR value of descriptionEn is indeed now with line separations instead of the \n (which still should be recognized by MySQL) text.
Final thoughts
I believe since even this
select * from kerkinidb.ct_thhlastika_press_threats where description_en = 'hello\nworld';
works in MySQL Workbench, that something in between might be ruining the request when trying to also include newLine char or lineSeparators.
If there is any idea of why it doesn't work as intended please share to try it out.
Thank you for your time

I got it to work. Probably not the answer you wanted but, if you replace your incoming "\n"s with the system line separator, it will get you want you want.
search = search.replace("\\n", System.getProperty("line.separator"));
try {
return ctThhlastikaPressThreatsService.searchCtThhlastikaPressThreats(producedSearchCriterias(search), size, page, sort);
If you look at the params value logging, you'll see the param is now
extracted value ([col_2_0_] : [VARCHAR]) - [hello
world]
instead of
extracted value ([col_2_0_] : [VARCHAR]) - [hello\nworld]
It's like I suspected, somewhere in search pipeline, it's escaping the "\n". I'm not suggesting you do what I suggest right there on the controller, I just wanted to show you that this is what needs to happen (somewhere) for it to work.

Related

Spring data Couchbase #n1ql.fields query

I'm trying to make a N1QL based query on Spring Data Couchbase. The documentation says
#n1ql.fields will be replaced by the list of fields (eg. for a SELECT clause) necessary to reconstruct the entity.
My repository implementation is this one:
#Query("#{#n1ql.fields} WHERE #{#n1ql.filter}")
List<User> findAllByFields(String fields);
And I'm calling this query as follows:
this.userRepository.findAllByFields("SELECT firstName FROM default");
I'm getting this error:
Caused by: org.springframework.data.couchbase.core.CouchbaseQueryExecutionException: Unable to execute query due to the following n1ql errors:
{"msg":"syntax error - at AS","code":3000}
After a little bit of researching, I also tryed:
#Query("SELECT #{#n1ql.fields} FROM #{#n1ql.bucket} WHERE #{#n1ql.filter}")
With this query, I don't get an error, I get all the documents stored but only the ID the other fields are set to null, when my query tries to get the firstName field.
this.userRepository.findAllByFields("firstName");
Anyone knows how to do such a query?
Thank you in advance.
You're misunderstanding the concept, I encourage you to give the documentation more time and see more examples. I'm not sure what exactly you're trying to achieve but I'll throw some examples.
Find all users (with all of their stored data)
#Query("#{#n1ql.selectEntity} WHERE #{#n1ql.filter}")
List<User> findAllUsers();
This will basically generate SELECT meta().id,_cas,* FROM bucket WHERE type='com.example.User'
Notice findAllUsers() does not take any parameters because there are no param placeholders defined in the #Query above.
Find all users where firstName like
#Query("#{#n1ql.selectEntity} WHERE #{#n1ql.filter} AND firstName like $1")
List<User> findByFirstNameLike(String keyword);
This will generate something like the above query but with an extra where condition firstName like
Notice this method takes a keyword because there is a param placeholder defined $1.
Notice in the documentation it says
#{#n1ql.selectEntity} WHERE #{#n1ql.filter} AND test = $1
is equivalent to
SELECT #{#n1ql.fields} FROM #{#n1ql.bucket} WHERE
#{#n1ql.filter} AND test = $1
Now if you don't want to fetch all the data for user(s), you'll need to specify the fields being selected, read following links for more info
How to fetch a field from document using n1ql with spring-data-couchbase
https://docs.spring.io/spring-data/couchbase/docs/2.2.4.RELEASE/reference/html/#_dto_projections
I think you should try below query, that should resolve the issue to get fields based parameter you have sent as arguments.
Please refer blow query.
#Query("SELECT $1 FROM #{#n1q1.bucket} WHERE #{#n1ql.filter}")
List findByFirstName(String fieldName);
Here, bucket name resolve to the User entity and and n1ql.filter would be a default filter.

Camel Blueprint specify parameter for prepared sql statement

I have a poll enrich which enriches a POJO with the result of an SQL query (from a MySQL database). It currently gets the brand from the POJO and then gets the name from the order matching the brand. I had to add quotes around the ${body.getBrand}, else the query would look for a column with the brand name instead of using the value. Currently it looks like this:
<pollEnrich id="_enrich1" strategyRef="merge" timeout="5000">
<simple>sql:SELECT name FROM orders WHERE brand= '${body.getBrand}'</simple>
</pollEnrich>
I want to change it because I'll probably need to create more sql queries and the current version does not work if the value contains quotes and thus is vulnerable to sql injection.
I thought prepared statements would do the trick and wanted to use a named parameter but I do not seem to be able to set the value of the parameter.
I have tried many different things like for example setting a header and change the query to have a named parameter:
<setHeader headerName="brand" id="brand">
<simple>${body.getBrand}</simple>
</setHeader>
<pollEnrich id="_enrich1" strategyRef="merge" timeout="5000">
<simple>sql:SELECT name FROM orders WHERE brand= :#brand</simple>
</pollEnrich>
but I keep getting
PreparedStatementCallback; bad SQL grammar [SELECT name FROM orders WHERE brand= ?]; nested exception is java.sql.SQLException: No value specified for parameter 1
I have also tried setting the useMessageBodyForSql option to true (since this seemed like something that might help?) but nothing I have tried seemed to work.
I have seen a lot of examples/solutions for people setting the routes with java, but I assume there must also be a solution for the blueprint xml?
If anyone got any suggestion or example that would be great.
In Camel version < 2.16, pollEnrich doesn't have access to the original exchange and therefore cannot read your header, hence the exception. This is documented here: http://camel.apache.org/content-enricher.html
Guessing from your example, a normal enrich should work too and it has access to the original exchange. Try changing 'pollEnrich' to 'enrich'.

JSON Queries - Failed to execute

So, I am trying to execute a query using ArcGIS API, but it should match any Json queries. I am kind of new to this query format, so I am pretty sure I must be missing something, but I can't figure out what it is.
This page allows for testing queries on the database before I actually implement them in my code. Features in this database have several fields, including OBJECTID and Identificatie. I would like to, for example, select the feature where Identificatie = 1. If I enter this in the Where field though (Identificatie = 1) an error Failed to execute appears. This happens for every field, except for OBJECTID. Querying where OBJECTID = 1 returns the correct results. I am obviously doing something wrong, but I don't get it why OBJECTID does work here. A brief explanation (or a link to a page documenting queries for JSON, which I haven't found), would be appreciated!
Identificatie, along with most other fields in the service you're using, is a string field. Therefore, you need to use single quotes in your WHERE clause:
Identificatie = '1'
Or to get one that actually exists:
Identificatie = '1714100000729432'
OBJECTID = 1 works without quotes because it's a numeric field.
Here's a link to the correct query. And here's a link to the query with all output fields included.

Multiple, unknown number of fields passed into a query

Is it possible to create a generic query that would work for different types of documents? For example I have "cases" and "factories",
They have different set of fields. e.g:
{
id: 'case_o1',
name: 'Case numero uno',
amount: 40
}
{
id: 'factory_002',
location: 'Venezuela',
workers: 200,
operating: true
}
Is it possible to create a generic query where I would pass the type of an entity (case or factory) and additional parameters and it would filter results based on those?
I could of course use javascript view, but it doesn't allow me to filter by multiple fields. Let's say I want to fetch all factories located in Venezuela, with number of workers between 20 and 55.
I started with this, but then I got stuck:
select * from `mybucket` as entity
where position(meta(entity).id, $entity_type) == 0
How do I pass multiple predicates and have the query to recognize them?
I can of course list fields like this:
where position(meta(entity).id, $entity_type) == 0
and entity.location == 'Venezuela'
and entity.workers > $workers_min
and entity.workers < $workers_max
but then
I'm gonna have to create a separate query for each entity
And even then it won't solve my problem - I have no idea how to ignore predicates, what if next time $workers_min and $workers_max are not passed, does it mean I have to create a query for every single predicate (column)?
For security reasons I cannot generate free-form queries and pass them to Couchbase server, all the queries are already stored in the database, our api just picks them up out of a document and executes them
I think it's possible to create a query that would be "short-circuiting" for args that's undefined (e.g. WHERE $location IS MISSING OR entity.location == $location or something like that)
Is it possible at all to create a query that would be able to effectively filter and order a dataset based on arbitrary parameters? Or there's no way?
#Agzam. Sorry. I were writting my comment when you said it. But anyway. What you are asking for is possible by using coalesces in a not too complex expressions, but it is a REALLY bad idea because this will drastically throw down most of internal database optimizations. Including the use of any existing index. So, except if you are dealing with a relatively small database (and you are sure it will remain being approximately the same size), I suggest you to better try distinct approach… This is, in fact, the reason I implmented sqlapi.
If you need to have all querys previously stored in database, it probably could be much better to sort given arguments by its name and precalculate and store precalculated querys for each possible combination.
You can do it by assigning a default value to the variable when is not used. For instance if $location is not used you can set it to -1 as default value.
Then the where condition would be:
WHERE ($location=-1 OR entity.location = $location)

Get Redmine custom field value to a file

I'm trying to create a text file that contains the value of a custom field I added on redmine. I tried to get it from an SQL query in the create method of the project_controller.rb (at line 80 on redmine 1.2.0) as follows :
sql = Mysql.new('localhost','root','pass','bitnami_redmine')
rq = sql.query("SELECT value
FROM custom_values
INNER JOIN projects
ON custom_values.customized_id=projects.id
WHERE custom_values.custom_field_id=7
AND projects.name='#{#project.name}'")
rq.each_hash { |h|
File.open('pleasework.txt', 'w') { |myfile|
myfile.write(h['value'])
}
}
sql.close
This works fine if I test it in a separate file (with an existing project name instead of #project.name) so it may be a syntax issue but I can't find what it is. I'd also be glad to hear any other solution to get that value.
Thanks !
(there's a very similar post here but none of the solutions actually worked)
First, you could use Project.connection.query instead of your own Mysql instance. Second, I would try to log the SQL RAILS_DEFAULT_LOGGER.info "SELECT ..." and check if it's ok... And the third, I would use identifier instead of name.
I ended up simply using params["project"]["custom_field_values"]["x"] where x is the custom field's id. I still don't know why the sql query didn't work but well, this is much simpler and faster.