I am using spring NamedParameterJdbcTemplate with my sql.I am executing 2 queries through jbdc template it is taking 1.1 and 4 s respectively. but the same query if I am running in my sql it is taking 0.5 and 1 s respectively I don't understand what could be my bottleneck. my application and db resides on same server so there can be no network overhead, I have my connection pooled. I can say it is working because query with less amount of data taking 50 ms through application. please let me know what could be my bottleneck
below is my code of namedjdbcTemplate
MapSqlParameterSource parameters = new MapSqlParameterSource();
parameters.addValue("organizationIds", list);
parameters.addValue("fromDate", fromDate);
parameters.addValue("toDate", toDate);
long startTime = System.currentTimeMillis();
List<MappingUI> list =namedParameterJdbcTemplate.query(QueryList.CALCULATE_SCORE,parameters,new RowMapper<MappingUI>() {
#Override
public MappingUI mapRow(ResultSet rs, int rowNum) throws SQLException {
MappingUI mapping= new MappingUI();
mapping.setCompetitor(rs.getInt("ID"));
mapping.setTotalRepufactScore(rs.getFloat("AVG_SCORE"));
return mapping;
}
});
below is my query
SELECT AVG(SCORE.SCORE) AS AVG_SCORE,ANALYSIS.ID AS ID FROM SCORE SCORE,QC_ANALYSIS ANALYSIS WHERE SCORE.TAG_ID = ANALYSIS.ID AND ANALYSIS.ORGANIZATION_ID IN (:organizationIds) AND DATE(ANALYSIS.DATES) BETWEEN DATE(:fromDate) AND DATE(:toDate) GROUP BY ANALYSIS.ORGANIZATION_ID
Related
I have a server running Spring boot + JPA + Hibernate. I am using MySQL database (Using InnoDb engine by default).
The implementation draws inspiration from many articles I had search on Internet.
I have implemented REST API to facilitate building a website dynamically.
I wanted to log all the API requests into a log (audit log). So when the API is called,
I store the request method name and few parameters into auditlog table in MySql.
Just before I return from the API, I store the response as well by updating the same record.
I was reviewing the code logs of Hibernate when I make API requests using the web application client as well as Postman.
I noticed that for every API, it takes on an average 150ms - 200ms for inserts and updates.
This is proving to be costly for APIs which fetch very less information.
So I want to know how I can speed up the inserts so that my inserts/updates take less than 10 -20 ms.
My Auditlog entity is
#Entity
#Table(name="auditlog")
public class AuditLog{
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
#Column(nullable = false, updatable = false)
#Temporal(TemporalType.TIMESTAMP)
#CreatedDate
private Date created_at;
#Column(nullable = false)
#Temporal(TemporalType.TIMESTAMP)
#LastModifiedDate
private Date updated_at;
#NotBlank
private String methodName;
private String param1;
// Text field with private information like password masked
#Column(length = 65535, columnDefinition = "text")
private String request;
// Text field
#Column(length = 65535, columnDefinition = "text")
private String response;
private Integer result;
... // getters and setters
}
My AuditLogRepository is :
public interface AuditLogRepository extends JpaRepository<AuditLog, Long>{
}
In my REST API controller I am doing the following
...
AuditLog logEntry = new AuditLog();
// set all the values except generated ones like id, created_at and updated_at
logEntry.setMethodName(...);
logEntry.setParam1(...);
logEntry.setRequest(...);
// Save into the table using autowired repoitory
auditLogRepoitory.saveAndFlush(logEntry);
// ... do the operation of the API
// Update the logEntry
logEntry.setResult(...);
logEntry.setResponse(...);
auditLogRepoitory.saveAndFlush(logEntry);
...
Please help me in improving the insert and updates to the table.
Or please help in improving the code so that I can make APIs response faster.
Thanks,
Sri Prad
First tips
if you want to speed up insert/update don't user JpaRepository.save method (notice that saveAndFlush() internally calls save method).
Because JpaRepository.save internal select the entity in order to know if the entity is new or if it exists in database.
Here is the default implementation of jpaRepository.save :
#Transactional
public <S extends T> S save(S entity) {
Assert.notNull(entity, "Entity must not be null.");
if (this.entityInformation.isNew(entity)) {
this.em.persist(entity);
return entity;
} else {
return this.em.merge(entity);
}
}
I think using jdbcTemplate is the best option.
Second tips
when thinking about optimizing the inserts, it is probably useful to think about doing bulk inserts. According to mysql documentation website , The time required for inserting a row is determined by the following factors, where the numbers indicate approximate proportions:
Connecting: (3)
Sending query to server: (2)
Parsing query: (2)
Inserting row: (1 × size of row)
Inserting indexes: (1 × number of indexes)
Closing: (1)
So you can easily see how bulk insert can help you improve insert speed.
Third tips
You probably need to tune your mysql instance settings as explained in this stackeroverflow anwser
Others options
Make sur you have selected the right ID generation strategy as explained here https://dzone.com/articles/spring-boot-boost-jpa-bulk-insert-performance-by-100x
If your framework allows for it, do
START TRANSACTION
at the beginning of building the page and storing the auditing. And
COMMIT
at the end.
Using EF Core 2.2.6 and Pomelo.EntityFrameworkCore.MySql 2.2.6 (with MySqlConnector 0.59.2)). I have a model for UserData:
public class UserData
{
[DatabaseGenerated(DatabaseGeneratedOption.None)]
public ulong ID { get; private set; }
[Required]
public Dictionary<string, InventoryItem> Inventory { get; set; }
public UserData()
{
Data = new Dictionary<string, string>();
}
}
I have a REST method that can be called that will add items to the user inventory:
using (var transaction = context.Database.BeginTransaction())
{
UserData data = await context.UserData.FindAsync(userId);
// there is code here to detect duplicate entries/etc, but I've removed it for brevity
foreach (var item in items) data.Inventory.Add(item.ItemId, item);
context.UserData.Update(data);
await context.SaveChangesAsync();
transaction.Commit();
}
If two or more calls to this method are made with the same user id then I get concurrent accesses (despite the transaction). This causes the data to sometimes be incorrect. For example, if the inventory is empty and then two calls are made to add items simultaneously (item A and item B), sometimes the database will only contain either A or B, and not both. From logging it appears that it is possible for EF to read from the database while the other read/write is still occurring, causing the code to have the incorrect state of the inventory for when it tries to write back to the db. So I tried marking the isolation level as serializable.
using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))
Now I sometimes see an exception:
MySql.Data.MySqlClient.MySqlException (0x80004005): Deadlock found when trying to get lock; try restarting transaction
I don't understand how this code could deadlock... Anyways, I tried to proceed by wrapping this whole thing in a try/catch, and retry:
public static async Task<ResponseError> AddUserItem(Controller controller, MyContext context, ulong userId, List<InventoryItem> items, int retry = 5)
{
ResponseError result = null;
try
{
using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))
{
UserData data = await context.UserData.FindAsync(userId);
// there is code here to detect duplicate entries/etc, but I've removed it for brevity
foreach (var item in items) data.Inventory.Add(item.ItemId, item);
context.UserData.Update(data);
await context.SaveChangesAsync();
transaction.Commit();
}
}
catch (Exception e)
{
if (retry > 0)
{
await Task.Delay(SafeRandomGenerator(10, 500));
return await AddUserItem(controller, context, userId, items, retry--);
}
else
{
// store exception and return error
}
}
return result;
}
And now I am back to the data being sometimes correct, sometimes not. So I think the deadlock is another problem, but this is the only method accessing this data. So, I'm at a loss. Is there a simple way to read from the database (locking the row in the process) and then writing back (releasing the lock on write) using EF Core? I've looked at using concurrency tokens, but this seems overkill for what appears (on the surface to me) to be a trivial task.
I added logging for mysql connector as well as asp.net server and can see the following failure:
fail: Microsoft.EntityFrameworkCore.Database.Command[20102]
=> RequestId:0HLUD39EILP3R:00000001 RequestPath:/client/AddUserItem => Server.Controllers.ClientController.AddUserItem (ServerSoftware)
Failed executing DbCommand (78ms) [Parameters=[#p1='?' (DbType = UInt64), #p0='?' (Size = 4000)], CommandType='Text', CommandTimeout='30']
UPDATE `UserData` SET `Inventory` = #p0
WHERE `ID` = #p1;
SELECT ROW_COUNT();
A total hack is to just delay the arrival of the queries by a bit. This works because the client is most likely to generate these calls on load. Normally back-to-back calls aren't expected, so spreading them out in time by delaying on arrival works. However, I'd rather find a correct approach, since this just makes it less likely to be an issue:
ResponseError result = null;
await Task.Delay(SafeRandomGenerator(100, 500));
using (var transaction = context.Database.BeginTransaction(System.Data.IsolationLevel.Serializable))
// etc
This isn't a good answer, because it isn't what I wanted to do, but I'll post it here as it did solve my problem. My problem was that I was trying to read the database row, modify it in asp.net, and then write it back, all within a single transaction and while avoiding deadlocks. The backing field is JSON type, and MySQL provides some JSON functions to help modify that JSON directly in the database. This required me to write SQL statements directly instead of using EF, but it did work.
The first trick was to ensure I could create the row if it didn't exist, without requiring a transaction and lock.
INSERT INTO UserData VALUES ({0},'{{}}','{{}}') ON DUPLICATE KEY UPDATE ID = {0};
I used JSON_REMOVE to delete keys from the JSON field:
UPDATE UserData as S set S.Inventory = JSON_REMOVE(S.Inventory,{1}) WHERE S.ID = {0};
and JSON_SET to add/modify entries:
UPDATE UserData as S set S.Inventory = JSON_SET(S.Inventory,{1},CAST({2} as JSON)) WHERE S.ID = {0};
Note, if you're using EF Core and want to call this using FromSql then you need to return the entity as part of your SQL statement. So you'll need to add something like this to each SQL statement:
SELECT * from UserData where ID = {0} LIMIT 1;
Here is a full working example as an extension method:
public static async Task<UserData> FindOrCreateAsync(this IQueryable<UserData> table, ulong userId)
{
string sql = "INSERT INTO UserData VALUES ({0},'{{}}','{{}}') ON DUPLICATE KEY UPDATE ID = {0}; SELECT * FROM UserData WHERE ID={0} LIMIT 1;";
return await table.FromSql(sql, userId).SingleOrDefaultAsync();
}
public static async Task<UserData> JsonRemoveInventory(this DbSet<UserData> table, ulong userId, string key)
{
if (!key.StartsWith("$.")) key = $"$.\"{key}\"";
string sql = "UPDATE UserData as S set S.Inventory = JSON_REMOVE(S.Inventory,{1}) WHERE S.ID = {0}; SELECT * from UserData where ID = {0} LIMIT 1;";
return await table.AsNoTracking().FromSql(sql, userId, key).SingleOrDefaultAsync();
}
Usage:
var data = await context.UserData.FindOrCreateAsync(userId);
await context.UserData.JsonRemoveInventory(userId, itemId);
I have a game where I use (Spring, Hibernate and MySQL 5.7) in the back-end. At the end of each game, I execute a native query to bulk update balance of the winners. Most games the update executes successfully for all winners, but for a few games I face the problem of updating the balance for most of them except for one or two (random). The update doesn't throw an error and as said Most of the winners' balance is updated successfully. Here is the method in my DAO (I summed it up) with the native query:
#Override
public Integer addAmountToPlayerBalance(List<Long> playerIds, Double amount, Long gameId) {
try
{
StringBuilder queryNative = new StringBuilder();
queryNative.append("update game_player_user set balance = ifnull(balance,0) + :amount where id in (:playerIds)");
Session session = teleEM.unwrap(Session.class);
org.hibernate.Query query = session.createSQLQuery(queryNative.toString());
query.setParameter("amount", amount);
query.setParameterList("playerIds", playerIds);
int numOfUpdatedRecords = query.executeUpdate();
return numOfUpdatedRecords;
}
catch (NoResultException e)
{
return null;
}
}
Notes:
1) I added the code for returning the number of updated records a week ago but it hasn't occurred on production since then. but as I said, it happens occasionally.
2) The list of player Ids includes all of the winners even in the games where the issue appears.
3) The amount is double but rounded to 2 decimal places.
I am using Mahout to build an Item-based Cf recommendation engine.
I create an MahoutHelper class which has a constructor:
public MahoutHelper(String serverName, String user, String password,
String DatabaseName, String tableName) {
source = new MysqlConnectionPoolDataSource();
source.setServerName(serverName);
source.setUser(user);
source.setPassword(password);
source.setDatabaseName(DatabaseName);
source.setCachePreparedStatements(true);
source.setCachePrepStmts(true);
source.setCacheResultSetMetadata(true);
source.setAlwaysSendSetIsolation(true);
source.setElideSetAutoCommits(true);
DBmodel = new MySQLJDBCDataModel(source, tableName, "userId", "itemId",
"value", null);
similarity = new TanimotoCoefficientSimilarity(DBmodel);
}
and the recommend method is:
public List<RecommendedItem> recommendation() throws TasteException {
Recommender recommender = null;
recommender = new GenericItemBasedRecommender(DBmodel, similarity);
List<RecommendedItem> recommendations = null;
recommendations = recommender.recommend(userId, maxNum);
System.out.println("query completed");
return recommendations;
}
It's using datasource to build datamodel but the problem is that when mysql has only a few data (less than 100) the program works fine for me, while when the scale turns to be over 1,000,000, the program stacks at doing recommendation and never goes forward. I have no idea how it happens. By the way I used the same data to build a FileDataModel with a .dat file, and it takes only 2~3 second to complete analysis. I am confused.
Using the database directly will only work for tiny data sets, like maybe a hundred thousand data points. Beyond that the overhead of such data-intensive applications will never run quickly; a query takes thousands of SQL queries or more.
Instead you must load and re-load into memory. You can still pull from the database; look at ReloadFromJDBCDataModel as a wrapper.
I'm running this query in LINQ:
var unalloc = db.slot_sp_getUnallocatedJobs("Repair",
RadComboBox1.SelectedValue, 20);
It runs when I first open the page, but when I go back to it and try to run the same query with a different value, "Con", being passed through, the linq to sql designer.cs tells me that I've got a timeout error.
Any ideas?
Edit: This is what's in the designer:
[Function(Name="dbo.slot_sp_getUnallocatedJobs")]
Public ISingleResult<slot_sp_getUnallocatedJobsResult> slot_sp_getUnallocatedJobs([Parameter(Name="JobType", DbType="VarChar(20)")] string jobType, [Parameter(Name="Contract", DbType="VarChar(10)")] string contract, [Parameter(Name="Num", DbType="Int")] System.Nullable<int> num)
{
IExecuteResult result = this.ExecuteMethodCall(this, ((MethodInfo)(MethodInfo.GetCurrentMethod())), jobType, contract, num);
return ((ISingleResult<slot_sp_getUnallocatedJobsResult>)(result.ReturnValue));
}
}
This is the error:
SQLException was unhandled by user code
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.