Couchbase N1QL Java SDK Pagination and Sorting issue - couchbase

I have started using couchbase recently. I am using Spring-Data couchbase for inserting Java POJOs in Couchbase. Since, Pagination and Sorting is not supported by Spring Data Couchbase project, I have tried to use couchbase java client 2.2.0-dp2.
I have inserted 8 users where ids ranging from 1 to 8.
I have wrote following code to apply pagination and sorting.
public void test() {
int offset = 5 * (1 - 1);
Statement statement = select("*").from("test").where(x("_class").eq(s("com.test.rest.entity.User"))).orderBy(Sort.asc("id")).limit(5).offset(offset);
log.info(statement.toString());
Iterator<QueryRow> result = bucket.query(Query.simple(statement)).rows();
while(result.hasNext()) {
QueryRow doc = result.next();
log.info("Document:: " + doc.value());
}
}
However, I am seeing result as below. It should be test1 to test5, though users being selected randomly. Can someone help me with that?
Document:: {“test":{"createdAt":1.443420400374E12,"firstname":"test5","_class":"com.test.rest.entity.User","type":"User","lastname":"test5"}}
Document:: {“test":{"createdAt":1.443420708495E12,"firstname":"test8","_class":"com.test.rest.entity.User","type":"User","lastname":"test8"}}
Document:: {“test:{"createdAt":1.443420386638E12,"firstname":"test2","_class":"com.test.rest.entity.User","type":"User","lastname":"test2"}}
Document:: {“test":{"createdAt":1.443420704104E12,"firstname":"test7","_class":"com.test.rest.entity.User","type":"User","lastname":"test7"}}
Document:: {“test":{"createdAt":1.443420379712E12,"firstname":"test1","_class":"com.test.rest.entity.User","type":"User","lastname":"test1"}}

It should be test1 to test5, though users being selected randomly
Looks like you are expecting to sort by first name, since id is generated by CB.
Try replacing "id" with "firstname", e.g.:
Statement statement = select("*").from("test").where(x("_class")
.eq(s("com.test.rest.entity.User")))
.orderBy(Sort.asc("firstname")).limit(5).offset(offset);
NOTE: I suspect that field name "id" collides with the metadata's id (hence not returned in your json result). Try naming it differently like "_id", "id#" etc.

After going through Spring Data Couchbase codebase, I figure out the way they are querying the n1ql. By default select(*) doesn't select id because id is not part of document. So,
N1QL Statement:
SELECT META(`test`).id AS _ID, META(`test`).cas AS _CAS, `test`.* FROM `test` WHERE `_class` = "com.test.rest.entity.User";
Couchbase Java Client code:
Statement statement = select("META(`test`).id AS _ID, META(`test`).cas AS _CAS, `test`.*").from("test").where(x("_class").eq(s("com.test.rest.entity.User"))).orderBy(Sort.asc("_ID")).limit(5).offset(offset);
NOTE: orderBy(Sort.asc("_ID")) is not required. I just kept it for sample.

Related

Spring data Couchbase #n1ql.fields query

I'm trying to make a N1QL based query on Spring Data Couchbase. The documentation says
#n1ql.fields will be replaced by the list of fields (eg. for a SELECT clause) necessary to reconstruct the entity.
My repository implementation is this one:
#Query("#{#n1ql.fields} WHERE #{#n1ql.filter}")
List<User> findAllByFields(String fields);
And I'm calling this query as follows:
this.userRepository.findAllByFields("SELECT firstName FROM default");
I'm getting this error:
Caused by: org.springframework.data.couchbase.core.CouchbaseQueryExecutionException: Unable to execute query due to the following n1ql errors:
{"msg":"syntax error - at AS","code":3000}
After a little bit of researching, I also tryed:
#Query("SELECT #{#n1ql.fields} FROM #{#n1ql.bucket} WHERE #{#n1ql.filter}")
With this query, I don't get an error, I get all the documents stored but only the ID the other fields are set to null, when my query tries to get the firstName field.
this.userRepository.findAllByFields("firstName");
Anyone knows how to do such a query?
Thank you in advance.
You're misunderstanding the concept, I encourage you to give the documentation more time and see more examples. I'm not sure what exactly you're trying to achieve but I'll throw some examples.
Find all users (with all of their stored data)
#Query("#{#n1ql.selectEntity} WHERE #{#n1ql.filter}")
List<User> findAllUsers();
This will basically generate SELECT meta().id,_cas,* FROM bucket WHERE type='com.example.User'
Notice findAllUsers() does not take any parameters because there are no param placeholders defined in the #Query above.
Find all users where firstName like
#Query("#{#n1ql.selectEntity} WHERE #{#n1ql.filter} AND firstName like $1")
List<User> findByFirstNameLike(String keyword);
This will generate something like the above query but with an extra where condition firstName like
Notice this method takes a keyword because there is a param placeholder defined $1.
Notice in the documentation it says
#{#n1ql.selectEntity} WHERE #{#n1ql.filter} AND test = $1
is equivalent to
SELECT #{#n1ql.fields} FROM #{#n1ql.bucket} WHERE
#{#n1ql.filter} AND test = $1
Now if you don't want to fetch all the data for user(s), you'll need to specify the fields being selected, read following links for more info
How to fetch a field from document using n1ql with spring-data-couchbase
https://docs.spring.io/spring-data/couchbase/docs/2.2.4.RELEASE/reference/html/#_dto_projections
I think you should try below query, that should resolve the issue to get fields based parameter you have sent as arguments.
Please refer blow query.
#Query("SELECT $1 FROM #{#n1q1.bucket} WHERE #{#n1ql.filter}")
List findByFirstName(String fieldName);
Here, bucket name resolve to the User entity and and n1ql.filter would be a default filter.

What is the Equvalent N1ql query for inserting the document in the couchbase

The following query is for fetching the documents from the couchbase while working with Java_client
N1qlQueryResult result = bucket.query(N1qlQuery.simple(
"SELECT * FROM `default;"));
How to write the statement to insert the docs into Couchbase using N1qlQuery like above?
Since 4.1, N1QL also have INSERT and UPSERT statements that you can use to create documents. See the reference (INSERT)
You can insert a document using the insert(...) method from the Bucket class. For example:
JSONDocument resultDoc = bucket.insert(JSONDocument.create(myId, myJSONDocument));
where both myId and myJSONDocument are strings.
For future reference, you can always refer to the Couchbase JAVA SDK.

Couchbase N1QL: How to get complete Document including ID and CAS with SELECT

I am using Couchbase 4.0 beta with java-client 2.1.3.
The Bucket.get(id) returns JsonDocument from which I can get id and cas as well as the content. Now I want to query on secondary indices using N1QL (select *). However, the QueryResult only returns rows of JsonObjects which are just the document contents. Is there anyway I can get the meta data (id and cas) too?
You can do the following:
SELECT b, meta(b) AS meta FROM my_bucket b;

Why does MySQL permit non-exact matches in SELECT queries?

Here's the story. I'm testing doing some security testing (using zaproxy) of a Laravel (PHP framework) application running with a MySQL database as the primary store for data.
Zaproxy is reporting a possible SQL injection for a POST request URL with the following payload:
id[]=3-2&enabled[]=on
Basically, it's an AJAX request to turn on/turn off a particular feature in a list. Zaproxy is fuzzing the request: where the id value is 3-2, there should be an integer - the id of the item to update.
The problem is that this request is working. It should fail, but the code is actually updating the item where id = 3.
I'm doing things the way I'm supposed to: the model is retrieved using Eloquent's Model::find($id) method, passing in the id value from the request (which, after a bit of investigation, was determined to be the string "3-2"). AFAIK, the Eloquent library should be executing the query by binding the ID value to a parameter.
I tried executing the query using Laravel's DB class with the following code:
$result = DB::select("SELECT * FROM table WHERE id=?;", array("3-2"));
and got the row for id = 3.
Then I tried executing the following query against my MySQL database:
SELECT * FROM table WHERE id='3-2';
and it did retrieve the row where id = 3. I also tried it with another value: "3abc". It looks like any value prefixed with a number will retrieve a row.
So ultimately, this appears to be a problem with MySQL. As far as I'm concerned, if I ask for a row where id = '3-2' and there is no row with that exact ID value, then I want it to return an empty set of results.
I have two questions:
Is there a way to change this behaviour? It appears to be at the level of the database server, so is there anything in the database server configuration to prevent this kind of thing?
This looks like a serious security issue to me. Zaproxy is able to inject some arbitrary value and make changes to my database. Admittedly, this is a fairly minor issue for my application, and the (probably) only values that would work will be values prefixed with a number, but still...
SELECT * FROM table WHERE id= ? AND ? REGEXP "^[0-9]$";
This will be faster than what I suggested in the comments above.
Edit: Ah, I see you can't change the query. Then it is confirmed, you must sanitize the inputs in code. Another very poor and dirty option, if you are in an odd situation where you can't change query but can change database, is to change the id field to [VAR]CHAR.
I believe this is due to MySQL automatically converting your strings into numbers when comparing against a numeric data type.
https://dev.mysql.com/doc/refman/5.1/en/type-conversion.html
mysql> SELECT 1 > '6x';
-> 0
mysql> SELECT 7 > '6x';
-> 1
mysql> SELECT 0 > 'x6';
-> 0
mysql> SELECT 0 = 'x6';
-> 1
You want to really just put armor around MySQL to prevent such a string from being compared. Maybe switch to a different SQL server.
Without re-writing a bunch of code then in all honesty the correct answer is
This is a non-issue
Zaproxy even states that it's possibly a SQL injection attack, meaning that it does not know! It never said "umm yeah we deleted tables by passing x-y-and-z to your query"
// if this is legal and returns results
$result = DB::select("SELECT * FROM table WHERE id=?;", array("3"));
// then why is it an issue for this
$result = DB::select("SELECT * FROM table WHERE id=?;", array("3-2"));
// to be interpreted as
$result = DB::select("SELECT * FROM table WHERE id=?;", array("3"));
You are parameterizing your queries so Zaproxy is off it's rocker.
Here's what I wound up doing:
First, I suspect that my expectations were a little unreasonable. I was expecting that if I used parameterized queries, I wouldn't need to sanitize my inputs. This is clearly not the case. While parameterized queries eliminate some of the most pernicious SQL injection attacks, this example shows that there is still a need to examine your inputs and make sure you're getting the right stuff from the user.
So, with that said... I decided to write some code to make checking ID values easier. I added the following trait to my application:
trait IDValidationTrait
{
/**
* Check the ID value to see if it's valid
*
* This is an abstract function because it will be defined differently
* for different models. Some models have IDs which are strings,
* others have integer IDs
*/
abstract public static function isValidID($id);
/**
* Check the ID value & fail (throw an exception) if it is not valid
*/
public static function validIDOrFail($id)
{
...
}
/**
* Find a model only if the ID matches EXACTLY
*/
public static function findExactID($id)
{
...
}
/**
* Find a model only if the ID matches EXACTLY or throw an exception
*/
public static function findExactIDOrFail($id)
{
...
}
}
Thus, whenever I would normally use the find() method on my model class to retrieve a model, instead I use either findExactID() or findExactIDOrFail(), depending on how I want to handle the error.
Thank you to everyone who commented - you helped me to focus my thinking and to understand better what was going on.

How best to retrieve result of SELECT COUNT(*) from SQL query in Java/JDBC - Long? BigInteger?

I'm using Hibernate but doing a simple SQLQuery, so I think this boils down to a basic JDBC question. My production app runs on MySQL but my test cases use an in memory HSQLDB. I find that a SELECT COUNT operation returns BigInteger from MySQL but Long from HSQLDB.
MySQL 5.5.22
HSQLDB 2.2.5
The code I've come up with is:
SQLQuery tq = session.createSQLQuery(
"SELECT COUNT(*) AS count FROM calendar_month WHERE date = :date");
tq.setDate("date", eachDate);
Object countobj = tq.list().get(0);
int count = (countobj instanceof BigInteger) ?
((BigInteger)countobj).intValue() : ((Long)countobj).intValue();
This problem of the return type negates answers to other SO questions such as getting count(*) using createSQLQuery in hibernate? where the advice is to use setResultTransformer to map the return value into a bean. The bean must have a type of either BigInteger or Long, and fails if the type is not correct.
I'm reluctant to use a cast operator on the 'COUNT(*) AS count' portion of my SQL for fear of database interoperability. I realise I'm already using createSQLQuery so I'm already stepping outside the bounds of Hibernates attempts at database neutrality, but having had trouble before with the differences between MySQL and HSQLDB in terms of database constraints
Any advice?
I don't known a clear solution for this problem, but I will suggest you to use H2 database for your tests.
H2 database has a feature that you can connect using a compatibility mode to several different databases.
For example to use MySQL mode you connect to the database using this jdbc:h2:~/test;MODE=MySQL URL.
You can downcast to Number and then call the intValue() method. E.g.
SQLQuery tq = session.createSQLQuery("SELECT COUNT(*) AS count FROM calendar_month WHERE date = :date");
tq.setDate("date", eachDate);
Object countobj = tq.list().get(0);
int count = ((Number) countobj).intValue();
Two ideas:
You can get result value as String and then parse it to Long or BigInteger
Do not use COUNT(*) AS count FROM ..., better use something like COUNT(*) AS cnt ... but in your example code you do not use name of result column but it index, so you can use simply COUNT(*) FROM ...