So, I am trying to execute a query using ArcGIS API, but it should match any Json queries. I am kind of new to this query format, so I am pretty sure I must be missing something, but I can't figure out what it is.
This page allows for testing queries on the database before I actually implement them in my code. Features in this database have several fields, including OBJECTID and Identificatie. I would like to, for example, select the feature where Identificatie = 1. If I enter this in the Where field though (Identificatie = 1) an error Failed to execute appears. This happens for every field, except for OBJECTID. Querying where OBJECTID = 1 returns the correct results. I am obviously doing something wrong, but I don't get it why OBJECTID does work here. A brief explanation (or a link to a page documenting queries for JSON, which I haven't found), would be appreciated!
Identificatie, along with most other fields in the service you're using, is a string field. Therefore, you need to use single quotes in your WHERE clause:
Identificatie = '1'
Or to get one that actually exists:
Identificatie = '1714100000729432'
OBJECTID = 1 works without quotes because it's a numeric field.
Here's a link to the correct query. And here's a link to the query with all output fields included.
Related
as the title suggests I am having a problem trying to use JPA Criteria with Spring Boot for a specific case.
Generally everything works but trying to search for stored data with a String property having newLine character embedded ( \n ) doesn't seem to work.
I am able to Get the data, edit them, save them through my front end, create new with multiple lines etc. but trying to search for them when for example a column is equals with 'hello\nworld' it wont work even though running this query in MySQL Workbench works returning the desired data :
select * from kerkinidb.ct_thhlastika_press_threats where description_en = 'hello\nworld';
To clarify, the way I do the search is by waiting in a Get request an argument called search which has all the properties that the user filtered. I am matching it with a Regex (which also has inside the Java 8.0 new Regex \\\\R for matching with multilines (and it works) ) then I am giving the the Service layer the Search Criteria that I matched which then passes to the Jpa Criteria Repository to parse them and generating the Predicates (matching again with Regex and \\\\R to create a final Predicate with OR and ANDs for the filtering) then triggering the query, then making another query called count to implement Pagination and finally mapping to a custom object and returning it.
I debugged every step and the final Predicate does generate the query I want, but the db doesn't return the expected data. So I am really confused since as I said the query does work in MySQL Workbench.
This is an example of the logging (I turned Spring Boot logging for MySQL logs on) generated when the Request is being triggered (in this case the stored data I have in my Table ct_thhlastika_press_threats in column description_en is a\ns\ndd so I am searching for this one as you can see instead of the example I said earlier hello\nworld :
2019-02-12 16:01:01.929 DEBUG 18368 --- [nio-8080-exec-2] org.hibernate.SQL : select ctthhlasti0_.id as col_0_0_, ctthhlasti0_.act_code as col_1_0_, ctthhlasti0_.description_en as col_2_0_, ctthhlasti0_.remarks as col_3_0_ from ct_thhlastika_press_threats ctthhlasti0_ where 1=1 and ctthhlasti0_.description_en=? order by ctthhlasti0_.id asc limit ?
2019-02-12 16:01:01.933 TRACE 18368 --- [nio-8080-exec-2] o.h.type.descriptor.sql.BasicBinder : binding parameter [1] as [VARCHAR] - [a\ns\ndd]
2019-02-12 16:01:01.944 DEBUG 18368 --- [nio-8080-exec-2] org.hibernate.SQL : select count(ctthhlasti0_.id) as col_0_0_ from ct_thhlastika_press_threats ctthhlasti0_ where 1=1 and ctthhlasti0_.description_en=?
2019-02-12 16:01:01.944 TRACE 18368 --- [nio-8080-exec-2] o.h.type.descriptor.sql.BasicBinder : binding parameter [1] as [VARCHAR] - [a\ns\ndd]
2019-02-12 16:01:01.946 TRACE 18368 --- [nio-8080-exec-2] o.h.type.descriptor.sql.BasicExtractor : extracted value ([col_0_0_] : [BIGINT]) - [0]
For anyone interested to look farther into the code you could find it Github repo. The project is for my University thesis. I made comments for adding to the Regexes (both of them) for ctThhlastikaPressThreats Controller and Repository. The db (the standard as well as the secondary for testing) need to be generated (can be generated by changing the auto-dll in application.properties).
Update
I tried using System.lineSeparator() (as suggested by #Hermann Steidel in the answer bellow) replacing the text newLine \n. But even though in the logs we can see now clearly that the value for the equals Predicate has line separations it still doesn't return the data from the DB.
Farther Explanation of my implementation
A correct Get request for the dynamic searching is like this :
http://localhost:8080/v1/ctThhlastikaPressThreats/search?search=descriptionEn~hello\nworld;#&size=10&page=0&sort=Asc
As you can see I am using a path variable called search (which has all the properties filtering requested from the user) along with 3 more for the size, page and sort.
For the search variable I am using 3 distinct characters to implement OR and AND Predicates for the final query. Those are the ~ which is to know that the property before it needs to use an equal predicate, the ; which is to be able to have multiple values for each property requested by the user and finally the # which triggers the end of this property's filtering.
Those 3 characters are being Regexed in two places. In the Controller and in the SearchRepository of (for example since we specifically started with this one -> the CtThhlastasikaPressThreats)
Finally in the SearchRepository you can see that I am triggering 2 queries, one is to get the filtered data from the db and another is to get the count of the data for Pagination purposes while also in the end mapping to a custom DTO.
Steps to reproduce :
After generating the db, change for the 2 Regexes of CtThhlastikaPressThreats. For the Controller to be (\w+?)(~|<|>)([(!-/.\\\\R 0-9\p{L});]+)?# and for the SearchRepository to be ([(!-/.\\\\R 0-9\p{L})]+).
Then you can use the example of the request I have above when having in the db saved for the specific table and for column descriptionEn with value of hello\nworld for example or whatever value you put (also change it into the request).
Final thing I tried but wasn't the solution :
Put in the CtThhlastikaPressThreatsSearchRepository in the method search (it is after line 61) above the :
predicate = builder.equal(root.get(param.getKey()), match.toString());
Make it as :
match = match.toString().replace("\\n", System.lineSeparator());
predicate = builder.equal(root.get(param.getKey()), match.toString());
This will basically change the value from being hellow\nworld to become hello\r\nworld so I guess it still isn't the desired solution. Thinking that in the db it is stored as \n for the lineSeparators.
Now in the logs you can see that when you trigger the Get Request again the VARCHAR value of descriptionEn is indeed now with line separations instead of the \n (which still should be recognized by MySQL) text.
Final thoughts
I believe since even this
select * from kerkinidb.ct_thhlastika_press_threats where description_en = 'hello\nworld';
works in MySQL Workbench, that something in between might be ruining the request when trying to also include newLine char or lineSeparators.
If there is any idea of why it doesn't work as intended please share to try it out.
Thank you for your time
I got it to work. Probably not the answer you wanted but, if you replace your incoming "\n"s with the system line separator, it will get you want you want.
search = search.replace("\\n", System.getProperty("line.separator"));
try {
return ctThhlastikaPressThreatsService.searchCtThhlastikaPressThreats(producedSearchCriterias(search), size, page, sort);
If you look at the params value logging, you'll see the param is now
extracted value ([col_2_0_] : [VARCHAR]) - [hello
world]
instead of
extracted value ([col_2_0_] : [VARCHAR]) - [hello\nworld]
It's like I suspected, somewhere in search pipeline, it's escaping the "\n". I'm not suggesting you do what I suggest right there on the controller, I just wanted to show you that this is what needs to happen (somewhere) for it to work.
I'm trying to make a N1QL based query on Spring Data Couchbase. The documentation says
#n1ql.fields will be replaced by the list of fields (eg. for a SELECT clause) necessary to reconstruct the entity.
My repository implementation is this one:
#Query("#{#n1ql.fields} WHERE #{#n1ql.filter}")
List<User> findAllByFields(String fields);
And I'm calling this query as follows:
this.userRepository.findAllByFields("SELECT firstName FROM default");
I'm getting this error:
Caused by: org.springframework.data.couchbase.core.CouchbaseQueryExecutionException: Unable to execute query due to the following n1ql errors:
{"msg":"syntax error - at AS","code":3000}
After a little bit of researching, I also tryed:
#Query("SELECT #{#n1ql.fields} FROM #{#n1ql.bucket} WHERE #{#n1ql.filter}")
With this query, I don't get an error, I get all the documents stored but only the ID the other fields are set to null, when my query tries to get the firstName field.
this.userRepository.findAllByFields("firstName");
Anyone knows how to do such a query?
Thank you in advance.
You're misunderstanding the concept, I encourage you to give the documentation more time and see more examples. I'm not sure what exactly you're trying to achieve but I'll throw some examples.
Find all users (with all of their stored data)
#Query("#{#n1ql.selectEntity} WHERE #{#n1ql.filter}")
List<User> findAllUsers();
This will basically generate SELECT meta().id,_cas,* FROM bucket WHERE type='com.example.User'
Notice findAllUsers() does not take any parameters because there are no param placeholders defined in the #Query above.
Find all users where firstName like
#Query("#{#n1ql.selectEntity} WHERE #{#n1ql.filter} AND firstName like $1")
List<User> findByFirstNameLike(String keyword);
This will generate something like the above query but with an extra where condition firstName like
Notice this method takes a keyword because there is a param placeholder defined $1.
Notice in the documentation it says
#{#n1ql.selectEntity} WHERE #{#n1ql.filter} AND test = $1
is equivalent to
SELECT #{#n1ql.fields} FROM #{#n1ql.bucket} WHERE
#{#n1ql.filter} AND test = $1
Now if you don't want to fetch all the data for user(s), you'll need to specify the fields being selected, read following links for more info
How to fetch a field from document using n1ql with spring-data-couchbase
https://docs.spring.io/spring-data/couchbase/docs/2.2.4.RELEASE/reference/html/#_dto_projections
I think you should try below query, that should resolve the issue to get fields based parameter you have sent as arguments.
Please refer blow query.
#Query("SELECT $1 FROM #{#n1q1.bucket} WHERE #{#n1ql.filter}")
List findByFirstName(String fieldName);
Here, bucket name resolve to the User entity and and n1ql.filter would be a default filter.
I have a poll enrich which enriches a POJO with the result of an SQL query (from a MySQL database). It currently gets the brand from the POJO and then gets the name from the order matching the brand. I had to add quotes around the ${body.getBrand}, else the query would look for a column with the brand name instead of using the value. Currently it looks like this:
<pollEnrich id="_enrich1" strategyRef="merge" timeout="5000">
<simple>sql:SELECT name FROM orders WHERE brand= '${body.getBrand}'</simple>
</pollEnrich>
I want to change it because I'll probably need to create more sql queries and the current version does not work if the value contains quotes and thus is vulnerable to sql injection.
I thought prepared statements would do the trick and wanted to use a named parameter but I do not seem to be able to set the value of the parameter.
I have tried many different things like for example setting a header and change the query to have a named parameter:
<setHeader headerName="brand" id="brand">
<simple>${body.getBrand}</simple>
</setHeader>
<pollEnrich id="_enrich1" strategyRef="merge" timeout="5000">
<simple>sql:SELECT name FROM orders WHERE brand= :#brand</simple>
</pollEnrich>
but I keep getting
PreparedStatementCallback; bad SQL grammar [SELECT name FROM orders WHERE brand= ?]; nested exception is java.sql.SQLException: No value specified for parameter 1
I have also tried setting the useMessageBodyForSql option to true (since this seemed like something that might help?) but nothing I have tried seemed to work.
I have seen a lot of examples/solutions for people setting the routes with java, but I assume there must also be a solution for the blueprint xml?
If anyone got any suggestion or example that would be great.
In Camel version < 2.16, pollEnrich doesn't have access to the original exchange and therefore cannot read your header, hence the exception. This is documented here: http://camel.apache.org/content-enricher.html
Guessing from your example, a normal enrich should work too and it has access to the original exchange. Try changing 'pollEnrich' to 'enrich'.
Here's the story. I'm testing doing some security testing (using zaproxy) of a Laravel (PHP framework) application running with a MySQL database as the primary store for data.
Zaproxy is reporting a possible SQL injection for a POST request URL with the following payload:
id[]=3-2&enabled[]=on
Basically, it's an AJAX request to turn on/turn off a particular feature in a list. Zaproxy is fuzzing the request: where the id value is 3-2, there should be an integer - the id of the item to update.
The problem is that this request is working. It should fail, but the code is actually updating the item where id = 3.
I'm doing things the way I'm supposed to: the model is retrieved using Eloquent's Model::find($id) method, passing in the id value from the request (which, after a bit of investigation, was determined to be the string "3-2"). AFAIK, the Eloquent library should be executing the query by binding the ID value to a parameter.
I tried executing the query using Laravel's DB class with the following code:
$result = DB::select("SELECT * FROM table WHERE id=?;", array("3-2"));
and got the row for id = 3.
Then I tried executing the following query against my MySQL database:
SELECT * FROM table WHERE id='3-2';
and it did retrieve the row where id = 3. I also tried it with another value: "3abc". It looks like any value prefixed with a number will retrieve a row.
So ultimately, this appears to be a problem with MySQL. As far as I'm concerned, if I ask for a row where id = '3-2' and there is no row with that exact ID value, then I want it to return an empty set of results.
I have two questions:
Is there a way to change this behaviour? It appears to be at the level of the database server, so is there anything in the database server configuration to prevent this kind of thing?
This looks like a serious security issue to me. Zaproxy is able to inject some arbitrary value and make changes to my database. Admittedly, this is a fairly minor issue for my application, and the (probably) only values that would work will be values prefixed with a number, but still...
SELECT * FROM table WHERE id= ? AND ? REGEXP "^[0-9]$";
This will be faster than what I suggested in the comments above.
Edit: Ah, I see you can't change the query. Then it is confirmed, you must sanitize the inputs in code. Another very poor and dirty option, if you are in an odd situation where you can't change query but can change database, is to change the id field to [VAR]CHAR.
I believe this is due to MySQL automatically converting your strings into numbers when comparing against a numeric data type.
https://dev.mysql.com/doc/refman/5.1/en/type-conversion.html
mysql> SELECT 1 > '6x';
-> 0
mysql> SELECT 7 > '6x';
-> 1
mysql> SELECT 0 > 'x6';
-> 0
mysql> SELECT 0 = 'x6';
-> 1
You want to really just put armor around MySQL to prevent such a string from being compared. Maybe switch to a different SQL server.
Without re-writing a bunch of code then in all honesty the correct answer is
This is a non-issue
Zaproxy even states that it's possibly a SQL injection attack, meaning that it does not know! It never said "umm yeah we deleted tables by passing x-y-and-z to your query"
// if this is legal and returns results
$result = DB::select("SELECT * FROM table WHERE id=?;", array("3"));
// then why is it an issue for this
$result = DB::select("SELECT * FROM table WHERE id=?;", array("3-2"));
// to be interpreted as
$result = DB::select("SELECT * FROM table WHERE id=?;", array("3"));
You are parameterizing your queries so Zaproxy is off it's rocker.
Here's what I wound up doing:
First, I suspect that my expectations were a little unreasonable. I was expecting that if I used parameterized queries, I wouldn't need to sanitize my inputs. This is clearly not the case. While parameterized queries eliminate some of the most pernicious SQL injection attacks, this example shows that there is still a need to examine your inputs and make sure you're getting the right stuff from the user.
So, with that said... I decided to write some code to make checking ID values easier. I added the following trait to my application:
trait IDValidationTrait
{
/**
* Check the ID value to see if it's valid
*
* This is an abstract function because it will be defined differently
* for different models. Some models have IDs which are strings,
* others have integer IDs
*/
abstract public static function isValidID($id);
/**
* Check the ID value & fail (throw an exception) if it is not valid
*/
public static function validIDOrFail($id)
{
...
}
/**
* Find a model only if the ID matches EXACTLY
*/
public static function findExactID($id)
{
...
}
/**
* Find a model only if the ID matches EXACTLY or throw an exception
*/
public static function findExactIDOrFail($id)
{
...
}
}
Thus, whenever I would normally use the find() method on my model class to retrieve a model, instead I use either findExactID() or findExactIDOrFail(), depending on how I want to handle the error.
Thank you to everyone who commented - you helped me to focus my thinking and to understand better what was going on.
I use the following sql statement:
Keyword.where("id IN (#{params[:keyword_ids]})").order("find_in_set(id, '#{params[:keyword_ids]}')")
The Problem at this statement is that if "keyword id" hold the same id more than ones, the call returns it only ones.
But I need the same number(not fulfilled) as well as the same order(which is fulfilled with this statement) which occurs in the array, independent if the same id occurs more than ones.
How should I change that statement to fix.
Thanks, dot
Well, that's not a bug, it's a feature ;)
My first recommendation would be to sanitize your input. Passing params[:keyword_ids] directly to the database, despite the help that the Rails framework does, is prone to lead to some kind of vulnerability sooner or later.
Secondly, the easiest solution is probably to keep the query as is, convert the results to a map and map the input params to the result.
Sth like
keywords = Keyword.where('id IN (?)', checked_keyword_ids)
keyword_map = Hash[keywords.map { |kw| [kw.id, kw] }]
checked_keyword_ids.map { |id| keyword_map[id] }