I'm trying to make a query to retrieve some data which has been created between two dates (represented as Instant).
Here below an extract from the Entity I'm using:
#Entity
public class HistoricalData {
#Id
#GeneratedValue
private Long id;
#Column
private String name;
#CreationTimestamp
private Instant timestamp;
#Column
private Double price;
}
And the query I've written to retrieve the data between the two Instants;
#Query("select h from HistoricalData h where h.timestamp between :timestampStart and :timestampEnd and upper(name) = upper(:name)")
List<HistoricalData> findHistoricalDataBetween(#NonNull Instant timestampStart, #NonNull Instant timestampEnd, #NonNull String name);
Which produces this SQL query:
select historical0_.id as id1_5_, historical0_.price as price2_5_, historical0_.timestamp as timestam3_5_ from historical_data historical0_ where (historical0_.timestamp between ? and ?) and upper(historical0_.name)=upper(?)
Also I wrote the "hibernate JPA" query just to try but no success:
List<HistoricalData> findHistoricalDataByTimestampAfterAndTimestampBeforeAndName(#NonNull Instant timestampStart, #NonNull Instant timestampEnd, #NonNull String name);
Keep in mind that all the above queries compile correctly and do not throw any exception, they just retrieve nothing from the database
The database I'm using is a latest version of MariaDB and the connector version is the 2.7.2
Also the SpringBoot version I'm using is the 2.5.3
Here is DDL from the table definition (automatically generated from Hibernate):
create table historical_data
(
id bigint not null primary key,
price double null,
timestamp datetime not null,
name varchar not null
);
An this is how the timestamp looks like in the database:
Even though records between those two Instants are present in the database I'm still getting nothing as a result from the query.
Looks like the reason is a time zone.
MySQL driver uses incorrect time zone transformations, using a default local time zone in place of a connection time zone (or vice versa).
Just debug this query inside MySQL driver to have fun and figure out what happens.
You can add parameters to the database URL to see which actual values are passed for the prepare statement
jdbc:mysql://<DATABASE_URL>?logger=com.mysql.cj.log.Slf4JLogger&profileSQL=true
Related
I have a server running Spring boot + JPA + Hibernate. I am using MySQL database (Using InnoDb engine by default).
The implementation draws inspiration from many articles I had search on Internet.
I have implemented REST API to facilitate building a website dynamically.
I wanted to log all the API requests into a log (audit log). So when the API is called,
I store the request method name and few parameters into auditlog table in MySql.
Just before I return from the API, I store the response as well by updating the same record.
I was reviewing the code logs of Hibernate when I make API requests using the web application client as well as Postman.
I noticed that for every API, it takes on an average 150ms - 200ms for inserts and updates.
This is proving to be costly for APIs which fetch very less information.
So I want to know how I can speed up the inserts so that my inserts/updates take less than 10 -20 ms.
My Auditlog entity is
#Entity
#Table(name="auditlog")
public class AuditLog{
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
#Column(nullable = false, updatable = false)
#Temporal(TemporalType.TIMESTAMP)
#CreatedDate
private Date created_at;
#Column(nullable = false)
#Temporal(TemporalType.TIMESTAMP)
#LastModifiedDate
private Date updated_at;
#NotBlank
private String methodName;
private String param1;
// Text field with private information like password masked
#Column(length = 65535, columnDefinition = "text")
private String request;
// Text field
#Column(length = 65535, columnDefinition = "text")
private String response;
private Integer result;
... // getters and setters
}
My AuditLogRepository is :
public interface AuditLogRepository extends JpaRepository<AuditLog, Long>{
}
In my REST API controller I am doing the following
...
AuditLog logEntry = new AuditLog();
// set all the values except generated ones like id, created_at and updated_at
logEntry.setMethodName(...);
logEntry.setParam1(...);
logEntry.setRequest(...);
// Save into the table using autowired repoitory
auditLogRepoitory.saveAndFlush(logEntry);
// ... do the operation of the API
// Update the logEntry
logEntry.setResult(...);
logEntry.setResponse(...);
auditLogRepoitory.saveAndFlush(logEntry);
...
Please help me in improving the insert and updates to the table.
Or please help in improving the code so that I can make APIs response faster.
Thanks,
Sri Prad
First tips
if you want to speed up insert/update don't user JpaRepository.save method (notice that saveAndFlush() internally calls save method).
Because JpaRepository.save internal select the entity in order to know if the entity is new or if it exists in database.
Here is the default implementation of jpaRepository.save :
#Transactional
public <S extends T> S save(S entity) {
Assert.notNull(entity, "Entity must not be null.");
if (this.entityInformation.isNew(entity)) {
this.em.persist(entity);
return entity;
} else {
return this.em.merge(entity);
}
}
I think using jdbcTemplate is the best option.
Second tips
when thinking about optimizing the inserts, it is probably useful to think about doing bulk inserts. According to mysql documentation website , The time required for inserting a row is determined by the following factors, where the numbers indicate approximate proportions:
Connecting: (3)
Sending query to server: (2)
Parsing query: (2)
Inserting row: (1 × size of row)
Inserting indexes: (1 × number of indexes)
Closing: (1)
So you can easily see how bulk insert can help you improve insert speed.
Third tips
You probably need to tune your mysql instance settings as explained in this stackeroverflow anwser
Others options
Make sur you have selected the right ID generation strategy as explained here https://dzone.com/articles/spring-boot-boost-jpa-bulk-insert-performance-by-100x
If your framework allows for it, do
START TRANSACTION
at the beginning of building the page and storing the auditing. And
COMMIT
at the end.
I have a table called 'otp', I need to delete records that is older than a certain period, let say 5 minutes for now.
I use Spring Framework with hibernate with MySql. I tried all possible ways, writing a '#Query' in the DTO interface, trying the Spring Data query way, I even tried to do a 'Select' on the records. When running in the code it does not select any records nor does it delete any records older than the period asked for. When I copy the 'Query" statement into the MySQL workbench it do work, I only need to put the DB name in front of the table name to get it to work. Below is snippets of the code. I removed the "#Query" statement and kept the Spring Data query.
Thank you in advance for your help.
The Entity or model.
'
#Getter
#Setter
#NoArgsConstructor
#Entity
#Table(name = "otp")
public class Otp {
#Id
#Column(name="phoneNumber")
private String phoneNumber;
#Column(name="otp")
private String otp;
#Column(name="createdOn", nullable = false, updatable = false)
#CreationTimestamp
private Timestamp createdOn;
}
`
Code that must delete the rows.
'
#Repository
public interface OTPdto extends CrudRepository<Otp, String> {
#Modifying
#Transactional
void deleteByCreatedOnLessThan(Object now);
}
'
Code that build the date for the query.
'
Date now = Date.from(Instant.now());
Object param = new java.sql.Timestamp(now.getTime());
otPdto.deleteByCreatedOnLessThan(param);
'
Hard to tell by provided info, but maybe I can help.
First of all, I would make sure that CreatedOn column gets filled in with correct information. Also I would try something like deleteByCreatedOnBefore(java.sql.Timestamp time) instead of deleteByCreatedOnLessThan(Object now). java.sql.Timestamp is not raw timestamp value, so maybe comparasing "LessThan" doesn't work well with that.
I will go another route and just clear the otp field. Seems I need to keep the phone number in the Db. Will use the phone number in future to remove the record.
I receive a MySql database and one table inside it have a Date column in string format, now I need to build a .Net core server with Pomelo and EF Core and requirement is my server can query data from that table in a range of date, but because Date column of that table is in string format so I don't know how to query it, please help.
Thank you!
You are going to have to get that string into a date in order to query it.
I would probably add a new datetime column to the table and then create a simple console app that reads in each string date, try to parse this as a datetime and save it to the new datetime column.
Then you should see how many rows have valid datetimes and correct the others
Finally, you can then query using Entity Framework
how to convert a string to date in mysql?
As was told here
You can Query string to date
SELECT STR_TO_DATE(yourdatefield, '%m/%d/%Y')
FROM yourtable
With database schema change
If you can (i.e. are allowed) to change the schema of the table in question, then just add a new datetime or date column, copy the data over from the old column to the new one, and drop the column:
ALTER TABLE `YourTable` ADD COLUMN `NewDateColumn` date NOT NULL;
UPDATE `YourTable` SET `NewDateColumn` = STR_TO_DATE(`OldDateColumn`,'%Y-%m-%d');
ALTER TABLE `YourTable` DROP COLUMN `OldDateColumn`;
You can run these statements just using MySQLWorkbench or the commmand line tool. Of course you first test them with a local copy, to see that everything works fine.
With value converter
If you cannot change the schema of the table, then you can still query date ranges from the database, as long as the date strings in the database are in a string format, that sorts alphabetically (e.g. YYYY-MM-DD). In that case, you can just use a value converter in your actual app code and don't need to alter the database at all:
public class SomeModel
{
public int SomeModelId {get; set;}
public DateTime YourDateProperty {get; set;} // <-- the type you want to use in .NET
}
public class Context : DbContext
{
public virtual DbSet<SomeModel> SomeModels { get; set; }
// ...
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<SomeModel>(
entity =>
{
entity.Property(e => e.YourDateProperty)
.HasColumnType("varchar(255)") // <-- the type it has in the database table
.HasConversion(
v => v.ToString(#"yyyy\-MM\-dd"),
v => DateTime.Parse(v, CultureInfo.InvariantCulture));
});
}
}
// Here is how a sample query in your app would look like:
var query = context.SomeModels
.Where(m => m.YourDateProperty >= new DateTime(2020, 9, 1) &&
m.YourDateProperty < new DateTime(2020, 9, 10))
.ToList();
When a run a GET request i get an exception o.h.engine.jdbc.spi.SqlExceptionHelper : Unknown column 'disburseme0_.reason_type' in 'field list' in stack trace even though i have configured the field correctly in the entity class. I have a Spring Boot SOAP interface that is querying a MySql database view. I have assigned one of the unique keys from the parent tables as the view Id in JPA.
Part of my entity class has:
#Entity
#Table(name="disbursement_payload")
public class Disbursement {
#Id
#Column(name="ID")
private long disbursementId;
#Column(name="ReasonType")
private String reasonType;
public long getDisbursementId() {
return disbursementId;
}
public void setDisbursementId(long disbursementId) {
this.disbursementId = disbursementId;
public String getReasonType() {
return reasonType;
}
public void setReasonType(String reasonType) {
this.reasonType = reasonType;
}
I have the view as:
CREATE VIEW disbursement_payload AS (
SELECT
iso_number AS Currency,
trans_desc AS ReasonType,
account_number AS ReceiverParty,
amount AS Amount
FROM m_payment_detail, m_loan_transaction
WHERE m_payment_detail.`id`= m_loan_transaction.`payment_detail_id` AND
m_payment_detail.`payment_type_id`=2
);
Is there something im missing , in the entity or view definition? I have read one of the comments here could not extract ResultSet in hibernate that i might have to explicitly define the parent schemas. Any assistance, greatly appreciated.
do the mapping for db column and class var name based on camelCase conversion basded on underscore _ separated name
you could try using
CREATE VIEW disbursement_payload AS (
SELECT iso_number AS currency
, trans_desc AS reason_type
, account_number AS receiver_rarty
, amount AS amount
FROM m_payment_detail
INNER JOIN m_loan_transaction
ON m_payment_detail.`id`= m_loan_transaction.`payment_detail_id`
AND m_payment_detail.`payment_type_id`=2
);
the view code is SQL code and hibernate see a view as a table, so the conversion of column name is base on the same rules
and a suggestion you should not use (older) implicit join based on where condition you should use (more recent) explici join sintax ..
I am using JPA (Hibernate) to store an entity on a MySQL 5.0 server.
Here is the entity in simplified form:
#Entity
#Table(name = "messages")
public class Message implements Serializable
{
#Id
#GeneratedValue
#Column
private long id;
#Column
private String content;
#Column(insertable = false)
private Date read;
#Column(insertable = false)
private Date deleted;
}
The columns "read" and "deleted" in table "messages" are defined so that they can contain NULL values. When I first tried to persist one of these entities I got an exception. Apparently, what Hibernate was doing there was listing the "read" and "deleted" columns in the column list of the insert statement but not in the value list. I got around that problem with the "insertable = false" statement in the #Column annotations you see above.
Now, however, I have a bigger problem. I want to set the read or date fields to non-null values. When I do that, I get a similar exception "You have an error in your SQL syntax". What he is doing now is listing all fields in the where part of the update statement, including "read" and "deleted". And what he does is check like "...and read=NULL". Which, in MySQL, of course should be "...and read IS NULL".
Rummaging around, I already found the "updatable" parameter for the #Column annotation. But if I set that to false, both "read" and "deleted" are never updated at all, so that is not what I am looking for, either.
...Help?
read is a reserved word in mysql and it looks like the driver is not escaping the name with back quotes. I think the best solution is to rename the column.