I am currently working on a ecommerce application where I have to show a list of available products using search functionality.
As with every search, I have to implement Pagination here.
I am using mybatis as my ORM tool and mysql as an underlying database.
Googling around I found following ways to accomplish this task :
Client Side paging
: Here I will have to fetch all the results from the database matching the search criteria in one stroke and handle the pagination at my code level (Possibly frond end code ).
Server Side Paging :
With mysql I can use the Limit and the offset of the resultset to construct a query like :
SELECT * FROM sampletable WHERE condition1>1 AND condition2>2 LIMIT 0,20
Here, I have to pass the offset and limit count everytime the user selects a new page while navigating in search results.
Can anyone tell,
which will be better way to implement paging ?
Do mybatis supports a better way to implement paging than just relying on above SQL queries ( like the hibernate criteria APIs).
Any inputs is highly appreaciated.
Thanks .
I myself use your second opion with LIMIT in sql query.
But there is range of methods that support pagination using RowBounds class.
This is well described in mybatis documentation here
Pay attention to correct result set type to use.
If you're using Mappers (much easier than using raw SqlSessions), the easiest way to apply a limit is by adding a RowBounds parameter to the mapping function's argument list, e.g:
// without limit
List<Foo> selectFooByExample(FooExample ex);
// with limit
List<Foo> selectFooByExample(FooExample ex, RowBounds rb);
This is mentioned almost as an afterthought in the link Volodymyr posted, under the Using Mappers heading, and could use some more emphasis:
You can also pass a RowBounds instance to the method to limit query results.
Note that support for RowBounds may vary by database. The Mybatis documentation implies that Mybatis will take care of using the appropriate query. However, for Oracle at least, this gets handled by very inefficient repeat calls to the database.
pagination has two types, physical and logical
logical means to retrieve all the data first then sort them in memory
physical means database level subset select
the default mybatis pagination is logical... thus when you select a massive database e.g 100GB of blobs, the rowbound method will still be very slow
the solution is to use the physical pagination
you can do your own way through the mybatis interceptor
or using plugins pre made by someone else
If you are using Spring MyBatis, you can achieve pagination manually using 2 MyBatis queries and the useful Spring Page and Pageable interfaces.
You create a higher level DAO interface e.g. UploadDao
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
public interface UploadDao {
Page<Upload> search(UploadSearch uploadSearch, Pageable pageable);
}
... where Upload maps to an upload table and UploadSearch is a parameter POJO e.g.
#Data // lombok
public class UploadSearch {
private Long userId;
private Long projectId;
...
}
An implementation of UploadDao (which injects a MyBatis UploadMapper mapper) is as follows:
public class DefaultUploadDao implements UploadDao {
#Autowired
private UploadMapper uploadMapper;
public Page<Upload> searchUploads(UploadSearch uploadSearch, Pageable pageable) {
List<Upload> content = uploadMapper.searchUploads(uploadSearch, pageable);
Long total = uploadMapper.countUploads(uploadSearch);
return new PageImpl<>(content, pageable, total);
}
}
The DAO implementation calls 2 methods of UploadMapper. These are:
UploadMapper.searchUploads - returns a page of results based on search param (UploadSearch) and Pageable param (contains offset / limit etc).
UploadMapper.countUploads - returns total count, again based on search param UploadSearch. NOTE - Pageable param is not required here as we're simply determining the total rows the search parameter filters to and don't care about page number / offset etc.
The injected UploadMapper interface looks like ...
#Mapper
public interface UploadMapper {
List<Upload> searchUploads(
#Param("search") UploadSearch search,
#Param("pageable") Pageable pageable);
long countUploads(
#Param("search") UploadSearch search);
}
... and the mapper XML file containing the dynamic SQL e.g. upload_mapper.xml contains ...
<mapper namespace="com.yourproduct.UploadMapper">
<select id="searchUploads" resultType="com.yourproduct.Upload">
select u.*
from upload u
<include refid="queryAndCountWhereStatement"/>
<if test="pageable.sort.sorted">
<trim prefix="order by">
<foreach item="order" index="i" collection="pageable.sort" separator=", ">
<if test="order.property == 'id'">id ${order.direction}</if>
<if test="order.property == 'projectId'">project_id ${order.direction}</if>
</foreach>
</trim>
</if>
<if test="pageable.paged">
limit #{pageable.offset}, #{pageable.pageSize}
</if>
<!-- NOTE: PostgreSQL has a slightly different syntax to MySQL i.e.
limit #{pageable.pageSize} offset #{pageable.offset}
-->
</select>
<select id="countUploads" resultType="long">
select count(1)
from upload u
<include refid="queryAndCountWhereStatement"/>
</select>
<sql id="queryAndCountWhereStatement">
<where>
<if test="search != null">
<if test="search.userId != null"> and u.user_id = #{search.userId}</if>
<if test="search.productId != null"> and u.product_id = #{search.productId}</if>
...
</if>
</where>
</sql>
</mapper>
NOTE - <sql> blocks (along with <include refid=" ... " >) are very useful here to ensure your count and select queries are aligned. Also, when sorting we are using conditions e.g. <if test="order.property == 'projectId'">project_id ${order.direction}</if> to map to a column (and stop SQL injection). The ${order.direction} is safe as the Spring Direction class is an enum.
The UploadDao could then be injected and used from e.g. a Spring controller:
#RestController("/upload")
public UploadController {
#Autowired
private UploadDao uploadDao; // Likely you'll have a service instead (which injects DAO) - here for brevity
#GetMapping
public Page<Upload>search (#RequestBody UploadSearch search, Pageable pageable) {
return uploadDao.search(search, pageable);
}
}
If you are using the MyBatis Generator, you may want to try the Row Bounds plugin from the official site: org.mybatis.generator.plugins.RowBoundsPlugin. This plugin will add a new version of the
selectByExample method that accepts a RowBounds parameter.
Related
I have a server running Spring boot + JPA + Hibernate. I am using MySQL database (Using InnoDb engine by default).
The implementation draws inspiration from many articles I had search on Internet.
I have implemented REST API to facilitate building a website dynamically.
I wanted to log all the API requests into a log (audit log). So when the API is called,
I store the request method name and few parameters into auditlog table in MySql.
Just before I return from the API, I store the response as well by updating the same record.
I was reviewing the code logs of Hibernate when I make API requests using the web application client as well as Postman.
I noticed that for every API, it takes on an average 150ms - 200ms for inserts and updates.
This is proving to be costly for APIs which fetch very less information.
So I want to know how I can speed up the inserts so that my inserts/updates take less than 10 -20 ms.
My Auditlog entity is
#Entity
#Table(name="auditlog")
public class AuditLog{
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
#Column(nullable = false, updatable = false)
#Temporal(TemporalType.TIMESTAMP)
#CreatedDate
private Date created_at;
#Column(nullable = false)
#Temporal(TemporalType.TIMESTAMP)
#LastModifiedDate
private Date updated_at;
#NotBlank
private String methodName;
private String param1;
// Text field with private information like password masked
#Column(length = 65535, columnDefinition = "text")
private String request;
// Text field
#Column(length = 65535, columnDefinition = "text")
private String response;
private Integer result;
... // getters and setters
}
My AuditLogRepository is :
public interface AuditLogRepository extends JpaRepository<AuditLog, Long>{
}
In my REST API controller I am doing the following
...
AuditLog logEntry = new AuditLog();
// set all the values except generated ones like id, created_at and updated_at
logEntry.setMethodName(...);
logEntry.setParam1(...);
logEntry.setRequest(...);
// Save into the table using autowired repoitory
auditLogRepoitory.saveAndFlush(logEntry);
// ... do the operation of the API
// Update the logEntry
logEntry.setResult(...);
logEntry.setResponse(...);
auditLogRepoitory.saveAndFlush(logEntry);
...
Please help me in improving the insert and updates to the table.
Or please help in improving the code so that I can make APIs response faster.
Thanks,
Sri Prad
First tips
if you want to speed up insert/update don't user JpaRepository.save method (notice that saveAndFlush() internally calls save method).
Because JpaRepository.save internal select the entity in order to know if the entity is new or if it exists in database.
Here is the default implementation of jpaRepository.save :
#Transactional
public <S extends T> S save(S entity) {
Assert.notNull(entity, "Entity must not be null.");
if (this.entityInformation.isNew(entity)) {
this.em.persist(entity);
return entity;
} else {
return this.em.merge(entity);
}
}
I think using jdbcTemplate is the best option.
Second tips
when thinking about optimizing the inserts, it is probably useful to think about doing bulk inserts. According to mysql documentation website , The time required for inserting a row is determined by the following factors, where the numbers indicate approximate proportions:
Connecting: (3)
Sending query to server: (2)
Parsing query: (2)
Inserting row: (1 × size of row)
Inserting indexes: (1 × number of indexes)
Closing: (1)
So you can easily see how bulk insert can help you improve insert speed.
Third tips
You probably need to tune your mysql instance settings as explained in this stackeroverflow anwser
Others options
Make sur you have selected the right ID generation strategy as explained here https://dzone.com/articles/spring-boot-boost-jpa-bulk-insert-performance-by-100x
If your framework allows for it, do
START TRANSACTION
at the beginning of building the page and storing the auditing. And
COMMIT
at the end.
I am doing a spring application and am kind stack. Iam running a query as shown below
#Autowired
EntityManagerFactory entityManagerFactory;
public List countTransactionsGroupByProvider(){
EntityManager em = entityManagerFactory.createEntityManager();
String query = "SELECT t.order_name,count(t.order_name) as number_of_transactions from transactions_view t where "
+ "t.transaction_date between '2014-07-24' and '2014-10-27' group by t.order_name";
List result = em.createNativeQuery(query).getResultList();
return result;
}
Now,This is working fine.it returns the data below:
[["Airtel",148], ["Expresso",8], ["Glo",49],
["MTN",110],["Select network",1],["Surfline",88],
["Tigo",35],["Vodafone",136],["Vouchers",30]]
My problem is I want this to return in the below format:
[{"order_name":"Airtel","number_of_transactions":148},
{"order_name":"Expresso","number_of_transactions":8},
{"order_name":"MTN","number_of_transactions":110},etc]
Then I can feed this into morris.js to plot a graph.
Any suggestion as to how to go about this.Thank much
You should probably just write some supporting code to transform the data into the format you want. Not sure you're going to get much traction trying to get JPA to produce the data in the format you want although it's arguably not out of the question.
Java 1.7, WildFly 8, MyBatis 3.2.6, PostgreSQL 9.4.
I have prototype that is simple system to lend books. I use MyBatis as ORM. All was fine and dandy until I tried filters.
There is table of books. Getting list of books is defined in mapper like this:
<select id="selectBooks" resultType="org.bookman.json.library.JsonBook">
select id, created, inclusion_date as inclusionDate, title, author
from bok_books
<where>
<if test="filter('title') != null">
title ilike #{filter('title')}||'%'
</if>
</where>
order by ${orderCols}
</select>
There are two methods used in EL - filter(String) and getOrderCols(). Select is called in this way:
TableReqHnd bookHnd = new TableReqHnd();
... // among other things sets filter data
List<JsonBook> books = sqlSession.selectList("selectBooks", bookHnd, rowBounds);
This is how TableReqHnd class looks:
public class TableReqHnd
{
private Map<String, String> filters = new HashMap<>();
...
public String getOrderCols()
{
...
}
...
public String filter(String fieldId)
{
if (!filters.containsKey(fieldId)) return null;
return filters.get(fieldId);
}
}
Why I did it like that? It allows me to reuse TableReqHnd object for any table, just with defined new values for ordering, filters etc. I don't have to create separate class for each table with its own setters and getters for each filter.
Now problem itself. Method getOrderCols() works fine, but filter(String) does not. Here is what exception is thrown: http://pastebin.com/raw.php?i=9jAjBhtd
It throws essentially java.lang.NoSuchMethodException: filter(java.lang.String). I find it strange. This method certainly exists in TableReqHnd: public String filter(String fieldId). ${orderCols} works fine (it is just simple getter public String getOrderCols()).
According to OGNL specs, call
filter('title')
should be correct. Exception in fact confirms it should be possible, it just cannot find method for some reason. Anyone knows if it can be fixed somehow?
Have you try
<select id="selectBooks" paramType="org.bookman.json.library.JsonBook">
select id, created, inclusion_date as inclusionDate, title, author
from bok_books
<where>
<if test="filter('title') != null">
title ilike #{filter('title')}||'%'
</if>
</where>
order by ${orderCols}
</select>
I have an interface that extends CrudRepository and implements a method with #Query annotation with the attribute nativeQuery setted to true. This method returns a list of an entity.
Example:
public interface MessageTemplateRepository extends CrudRepository<MessageTemplate, Integer> {
#Query(nativeQuery = true, "select template.* from plan_granted_template granted join license license on granted.fk_plan = license.fk_plan join message_template template on granted.fk_message_template = template.id where license.fk_garage = ?2 and template.message_type = ?1")
public List<MessageTemplate> findGrantedTemplatesByMessageTypeAndGarage(MessageType messageType, Garage garage);
}
The Garage has one License
The License has one Plan and Garage
The Plan has many MessageTemplate
The class License has a ManyToOne relation with Plan and OneToOne relation with Garage
(table license - columns fk_plan and fk_garage)
The class Plan has a ManyToMany relation with MessageTemplate
(table plan_granted_template - columns fk_plan and fk_message_template)
The class MessageTemplate has the atribute messageType
(table message_template - column message_type)
This method should return all the MessageTemplate entity found on query but always returns an empty list.
Executing this query in mysql returns the correct result.
Also looking at the Hibernate logs, the query is executed with the correct parameters, but an empty list is returned anyway.
I think that Spring executes the query but it's fail to convert the resultSet into an MessageTemplate instance.
You needed to pass the parameters as String and Integer. The Hibernate logs this way on both cases: [BasicBinder:83]: binding parameter [1] as [INTEGER] - 1 [BasicBinder:83]: binding parameter [2] as [VARCHAR] - WELCOME.
Not sure if your parameters MessageTemplate messageTemplate plays well with template.message_type = ?1, have you tried to pass String messageTemplate, I mean conversion between the messageTemplate and String needed in the query, try to add DEBUG to see the parameters send to the database
UPDATE
And the end parameters in the method should be string and int, not the Classes.
public IQueryable<Story> FindAllStories(){
var stories = (from s in db.Stories
orderby s.DateEntered descending
select new Story
{
Title = s.Title,
UserName = s.aspnet_User.UserName
}
);
return stories;
}
I need to pass this as IQueryable so the pagination helper I found online can only pull the items I need. The problem is that at runtime when the helper tries to do source.Count(); the compiler isn't a happy camper because it's an 'explicit contruction of an entity type query'.
How would I alter this LINQ to SQL method so this does not happen?
Also, to help me grasp this, why does the previous code not work and this one does?
public IQueryable<Story> FindAllStories(){
var stories = (from s in db.Stories
orderby s.DateEntered descending
select s);
return stories;
}
Update
I'm beginning to think the way to accomplish this (verified it works) is to create a POCO called UserStory. The new class has 2 properties: one of type Story and the other string UserName. I can then return an IQueryable of UserStory without a problem.
That's great; however, I still don't get why that method would work and my other doesn't. The other is adding a property of string UserName to my partial class Story and passing that object between layers. What's the difference?
The following link is to a blog post tat describes an issue similar to yours. It seems like the solution was to return a type that inherets from Story instead of a Story type:
http://devlicio.us/blogs/derik_whittaker/archive/2008/04/25/linq2sql-explicit-construction-of-entity-exception.aspx
Hope this helps.