Query from database with date range in .Net Core - mysql

I receive a MySql database and one table inside it have a Date column in string format, now I need to build a .Net core server with Pomelo and EF Core and requirement is my server can query data from that table in a range of date, but because Date column of that table is in string format so I don't know how to query it, please help.
Thank you!

You are going to have to get that string into a date in order to query it.
I would probably add a new datetime column to the table and then create a simple console app that reads in each string date, try to parse this as a datetime and save it to the new datetime column.
Then you should see how many rows have valid datetimes and correct the others
Finally, you can then query using Entity Framework

how to convert a string to date in mysql?
As was told here
You can Query string to date
SELECT STR_TO_DATE(yourdatefield, '%m/%d/%Y')
FROM yourtable

With database schema change
If you can (i.e. are allowed) to change the schema of the table in question, then just add a new datetime or date column, copy the data over from the old column to the new one, and drop the column:
ALTER TABLE `YourTable` ADD COLUMN `NewDateColumn` date NOT NULL;
UPDATE `YourTable` SET `NewDateColumn` = STR_TO_DATE(`OldDateColumn`,'%Y-%m-%d');
ALTER TABLE `YourTable` DROP COLUMN `OldDateColumn`;
You can run these statements just using MySQLWorkbench or the commmand line tool. Of course you first test them with a local copy, to see that everything works fine.
With value converter
If you cannot change the schema of the table, then you can still query date ranges from the database, as long as the date strings in the database are in a string format, that sorts alphabetically (e.g. YYYY-MM-DD). In that case, you can just use a value converter in your actual app code and don't need to alter the database at all:
public class SomeModel
{
public int SomeModelId {get; set;}
public DateTime YourDateProperty {get; set;} // <-- the type you want to use in .NET
}
public class Context : DbContext
{
public virtual DbSet<SomeModel> SomeModels { get; set; }
// ...
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.Entity<SomeModel>(
entity =>
{
entity.Property(e => e.YourDateProperty)
.HasColumnType("varchar(255)") // <-- the type it has in the database table
.HasConversion(
v => v.ToString(#"yyyy\-MM\-dd"),
v => DateTime.Parse(v, CultureInfo.InvariantCulture));
});
}
}
// Here is how a sample query in your app would look like:
var query = context.SomeModels
.Where(m => m.YourDateProperty >= new DateTime(2020, 9, 1) &&
m.YourDateProperty < new DateTime(2020, 9, 10))
.ToList();

Related

JPA Query using between and Instant not working

I'm trying to make a query to retrieve some data which has been created between two dates (represented as Instant).
Here below an extract from the Entity I'm using:
#Entity
public class HistoricalData {
#Id
#GeneratedValue
private Long id;
#Column
private String name;
#CreationTimestamp
private Instant timestamp;
#Column
private Double price;
}
And the query I've written to retrieve the data between the two Instants;
#Query("select h from HistoricalData h where h.timestamp between :timestampStart and :timestampEnd and upper(name) = upper(:name)")
List<HistoricalData> findHistoricalDataBetween(#NonNull Instant timestampStart, #NonNull Instant timestampEnd, #NonNull String name);
Which produces this SQL query:
select historical0_.id as id1_5_, historical0_.price as price2_5_, historical0_.timestamp as timestam3_5_ from historical_data historical0_ where (historical0_.timestamp between ? and ?) and upper(historical0_.name)=upper(?)
Also I wrote the "hibernate JPA" query just to try but no success:
List<HistoricalData> findHistoricalDataByTimestampAfterAndTimestampBeforeAndName(#NonNull Instant timestampStart, #NonNull Instant timestampEnd, #NonNull String name);
Keep in mind that all the above queries compile correctly and do not throw any exception, they just retrieve nothing from the database
The database I'm using is a latest version of MariaDB and the connector version is the 2.7.2
Also the SpringBoot version I'm using is the 2.5.3
Here is DDL from the table definition (automatically generated from Hibernate):
create table historical_data
(
id bigint not null primary key,
price double null,
timestamp datetime not null,
name varchar not null
);
An this is how the timestamp looks like in the database:
Even though records between those two Instants are present in the database I'm still getting nothing as a result from the query.
Looks like the reason is a time zone.
MySQL driver uses incorrect time zone transformations, using a default local time zone in place of a connection time zone (or vice versa).
Just debug this query inside MySQL driver to have fun and figure out what happens.
You can add parameters to the database URL to see which actual values are passed for the prepare statement
jdbc:mysql://<DATABASE_URL>?logger=com.mysql.cj.log.Slf4JLogger&profileSQL=true

SQLGrammar error when querying MySql view

When a run a GET request i get an exception o.h.engine.jdbc.spi.SqlExceptionHelper : Unknown column 'disburseme0_.reason_type' in 'field list' in stack trace even though i have configured the field correctly in the entity class. I have a Spring Boot SOAP interface that is querying a MySql database view. I have assigned one of the unique keys from the parent tables as the view Id in JPA.
Part of my entity class has:
#Entity
#Table(name="disbursement_payload")
public class Disbursement {
#Id
#Column(name="ID")
private long disbursementId;
#Column(name="ReasonType")
private String reasonType;
public long getDisbursementId() {
return disbursementId;
}
public void setDisbursementId(long disbursementId) {
this.disbursementId = disbursementId;
public String getReasonType() {
return reasonType;
}
public void setReasonType(String reasonType) {
this.reasonType = reasonType;
}
I have the view as:
CREATE VIEW disbursement_payload AS (
SELECT
iso_number AS Currency,
trans_desc AS ReasonType,
account_number AS ReceiverParty,
amount AS Amount
FROM m_payment_detail, m_loan_transaction
WHERE m_payment_detail.`id`= m_loan_transaction.`payment_detail_id` AND
m_payment_detail.`payment_type_id`=2
);
Is there something im missing , in the entity or view definition? I have read one of the comments here could not extract ResultSet in hibernate that i might have to explicitly define the parent schemas. Any assistance, greatly appreciated.
do the mapping for db column and class var name based on camelCase conversion basded on underscore _ separated name
you could try using
CREATE VIEW disbursement_payload AS (
SELECT iso_number AS currency
, trans_desc AS reason_type
, account_number AS receiver_rarty
, amount AS amount
FROM m_payment_detail
INNER JOIN m_loan_transaction
ON m_payment_detail.`id`= m_loan_transaction.`payment_detail_id`
AND m_payment_detail.`payment_type_id`=2
);
the view code is SQL code and hibernate see a view as a table, so the conversion of column name is base on the same rules
and a suggestion you should not use (older) implicit join based on where condition you should use (more recent) explici join sintax ..

DbUpdateConcurrencyException using Entity Framework 6 with MySql

I'm having trouble with concurrency checks using EF6 and MySQL.
The problem I'm having is that I get a concurrency exception thrown when I try to save data to the database. If you examine the sql that is output to the console it tries to query the concurrency field from the database using the old value in the where clause. Because this field has been updated by the database.
Environment:
Windows 7 64 bit
Visual Studio 2013
Nuget packages installed:
EF 6.0.1
MySql.ConnectorNET.Data 6.8.3.2
MySql.ConnectorNET.Entity 6.8.3.2
Demo Database SQL:
DROP DATABASE IF EXISTS `bugreport`;
CREATE DATABASE IF NOT EXISTS `bugreport`;
USE `bugreport`;
DROP TABLE IF EXISTS `test`;
CREATE TABLE IF NOT EXISTS `test` (
`TestId` int(10) NOT NULL AUTO_INCREMENT,
`AStringField` varchar(50) DEFAULT NULL,
`DateModified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`TestId`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;
INSERT INTO `test` (`TestId`, `AStringField`, `DateModified`) VALUES
(1, 'Initial Value', '2014-07-11 09:15:52');
Demo code:
using System;
using System.Data.Entity.Infrastructure;
using System.Linq;
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;
using System.Data.Entity;
namespace BugReport
{
class Program
{
static void Main(string[] args)
{
using (var context = new BugReportModel())
{
context.Database.Log = (s => Console.WriteLine(s));
var firstTest = context.tests.First();
firstTest.AStringField = "First Value";
// Exception is thrown when changes are saved.
context.SaveChanges();
Console.ReadLine();
}
}
}
public class BugReportModel : DbContext
{
public BugReportModel()
: base("name=Model1")
{
}
public virtual DbSet<test> tests { get; set; }
}
[Table("test")]
public class test
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int TestId { get; set; }
[StringLength(50)]
public string AStringField { get; set; }
[ConcurrencyCheck()]
[DatabaseGenerated(DatabaseGeneratedOption.Computed)]
[Column(TypeName = "timestamp")]
public System.DateTime DateModified { get; set; }
}
}
Update:
Filed bug with MySql.
You should be trying to use the DB Timestamp / Rowversion feature.
In EF you declare a ByteArray and nominate it as the Concurrency check field.
DB sets the value on creation. All subsequent updates can check the value hasnt changed
DB updates rowversion as appropriate. This approach works on SQL server.
It should behave the same way on MYSql.
public abstract class BaseObject {
[Key]
[Required]
public virtual int Id { set; get; }
[ConcurrencyCheck()]
public virtual byte[] RowVersion { get; set; }
}
or via fluent if you like
// Primary Key
this.HasKey(t => t.Id);
// Properties
//Id is an int allocated by DB , with string keys, no db generation now
this.Property(t => t.Id).HasDatabaseGeneratedOption(DatabaseGeneratedOption.Identity); // default to db generated
this.Property(t => t.RowVersion)
.IsRequired()
.IsFixedLength()
.HasMaxLength(8)
.IsRowVersion(); //<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Docu on the optimistic concurrency pattern
Workarround with Interceptor
I see that MySQL Connector bug pointed in question do not have a fix yet (since 2014) and I wrote that "solution" (I know this is ugly) until the they fix it.
I create an DBCommandInterceptor and override the ReaderExecuting to replace the equal operator (=) in last WHERE to a not equal operator (<>) because the pattern for the update is something like "UPDATE ...; SELECT ... WHERE (row_version_field = #parameter)"
In the code bellow replace row_version in regular expression with the name of your row version field.
public class ConcurrencyFixInterceptor : DbCommandInterceptor
{
private static Regex concurrencyPattern =
new Regex(#"^UPDATE[\S\s]+SELECT[\S\s]+\(.?row_version.?\s(=)\s#[\w\d]+\)$",
RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.Multiline);
public override void ReaderExecuting(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext)
{
Match macth = concurrencyPattern.Match(command.CommandText);
if (macth.Success)
{
command.CommandText =
command.CommandText.
Remove(macth.Groups[1].Index, 1).
Insert(macth.Groups[1].Index, "<>");
}
base.ReaderExecuting(command, interceptionContext);
}
}
I use a row version in MySQL with a TIMESTAMP(5) field type.
I have just submitted a PR to MySQL .NET Connector v6.9.10 that provides a workaround solution for this issue.
The workaround avoids use of TIMESTAMP or DATETIME values to perform optimistic locking using a safer BIGINT RowVersion value that is incremented via a BEFORE UPDATE trigger. This fix will now support optimistic locking with an external (non-EF) application. If I can fix a 2nd bug related to TIMESTAMP / DATETIME then ConcurrencyCheck should work with these types as well.
EF6:
public class MyTable
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.None)]
public virtual int Id { get; set; }
[Required, MaxLength(45)]
public virtual string Name { get; set; }
[ConcurrencyCheck, DatabaseGenerated(DatabaseGeneratedOption.Computed)]
[Column(TypeName = "bigint")]
public virtual long RowVersion { get; set; }
}
SQL:
CREATE TABLE IF NOT EXISTS `mytable` (
Id int(11) NOT NULL,
Name varchar(45) NOT NULL,
RowVersion bigint NOT NULL DEFAULT 0,
PRIMARY KEY (`Id`)
) ENGINE=InnoDB
CREATE TRIGGER `trg_mytable_before_update`
BEFORE UPDATE ON `mytable`
FOR EACH ROW SET NEW.RowVersion = OLD.RowVersion + 1;
TIMESTAMP Solution?
I'm also investigating how to performing optimistic locking with a TIMESTAMP field.
Firstly, you need to use a more fine grained timestamp value.
So for example if you use the following, your timestamp value will be truncated to the nearest second (not very safe for optimistic locking).
UpdatedAt TIMESTAMP NOT NULL DEFAULT CURRENT_TIME ON UPDATE CURRENT_TIME
Instead you should use following to record microsecond precision.
UpdatedAt TIMESTAMP(6) NOT NULL DEFAULT NOW(6) ON UPDATE NOW(6)
Secondly, I'm observing a bug that I'm reproducing within the environment of the MySQL .NET Connector unit test suite combined with the PR patch I've just submitted. EF6 now generates the correct optimistic locking SQL to perform an UPDATE followed by the SELECT (now fixed) that returns the updated TIMESTAMP field. However the MySQL connector returns a zero TIMESTAMP (0000-00-00 00:00:00.000000) even though executing the exact same UPDATE and SELECT in MySQL Workbench it returns a valid non-zero TIMESTAMP value. I've observed the packets read via the connection socket return the string '0000-00-00 00:00:00.000000' so its probably related to the MySQL session configuration in some way. Hints welcome! I'm currently testing this with MySQL v5.6.26 (Windows).
Multiple optimistic lock fields
In our case, we have a legacy MS-Access app that uses a TIMESTAMP in most tables to perform optimistic locking. This is a convenient solution for MS-Access as it detects the presence of any TIMESTAMP column and automatically applies optimistic locking to this column when it finds one.
Since we currently don't have optimistic locking working with EF6 for TIMESTAMP columns we've added a second optimistic lock column on each table we care about by creating a BIGINT RowVersion column as that is incremented via a BEFORE INSERT trigger. So now for each UPDATE both the existing TIMESTAMP column and the new RowVersion column are update so either can be used to detect a change. Not ideal but it works!

Using LongListSelector with a Deployed Database

I'm trying to create an app for Windows Phone 8 that displays data in a LongListSelector that's populated from a SQL CE database that's shipped with the app. I think I have the opening and reading from the database functions down, but I can't correctly use LINQ to SQL to group the data for the LLS.
I've got a database class with a table and corresponding columns. I'm using a helper class "KeyedList" to add a public name for the data from msdn sample code:
public class KeyedList<TKey, TItem> : List<TItem>
{
public TKey Key { protected set; get; }
public KeyedList(TKey key, IEnumerable<TItem> items)
: base(items)
{
Key = key;
}
public KeyedList(IGrouping<TKey, TItem> grouping)
: base(grouping)
{
Key = grouping.Key;
}
}
Then I've got my database context:
dB = new DataContext(DataContext.DBConnectionString);
Finally, here's the LINQ to SQL I'm trying to use:
var items =
from item in dB.TableName
orderby dB.ID
group item by dB.Generation into generation
select new <KeyedList<string,Item>(generation);
var allItems = new List<KeyedList<string, Item>>(items)
I've pretty much taken this code from the sample, but I can't get the grouping and ordering to work when creating allItems for binding to the LongListSelector. I keep getting invalid arguments error.
I'm very new at VB programming and appreciate all the help!
I found the issue. When creating the new Keyed list make sure to use the correct key type and item type. The key type will be the type of the data used by group by, and the item type is your DataContext. So in my case db.Generation is a string and the DataContext type is of type Item.

How to work with zero dates ("0000-00-00") in Hibernate?

I have MySql table that has a date field with zeroes ("0000-00-00") as its default value (field cannot be null, I can't change table structure). Hibernate doesn't like zero dates and throws exception during read or save.
I managed to make it read records by setting MySql connection setting "zeroDateTimeBehavior=convertToNull" that converts zero dates to nulls while retrieving records. It is all working fine until I try to save the record that has null date - it throws exception that date cannot be null.
So the question is - how to save record through Hibernate so date will appear as zeroes in a table?
Thanks.
I'd try to add an Hibernate Interceptor (API, Doc) and try to implement something in the onSave() method.
The following code may work:
static final Date ZERO_DATE = //0000-00-00
public boolean onSave(Object entity,
Serializable id,
Object[] state,
String[] propertyNames,
Type[] types)
throws CallbackException {
for(int i = 0; i< propertyNames.length; i++) {
if(propertyNames[i].equals("dateFieldName") && state[i]==null) {
state[i] = ZERO_DATE;
return; //or may continue, if there are several such fields.
}
}
}
Ready and working solution for DATE '0000-00-00' and TIME '00:00:00': How to map MySQL DATE '0000-00-00' & TIME '00:00:00' with Hibernate
Thanks Preston for the code and ChssPly76 for useful comments.