we have an existing .net application that makes heavy use of EntityFramework (6.4.4), as well as some usage of Dapper (2.0.35) both with MySql.Data provider (8.0.23).
we are in the process of exploring how to move to a global RDS Aurora database with write forwarding, and i have been working with a minimal simulation of our application.
so far in the secondary region, inserting data with EntityFramework always fails on SaveChanges; if connection pooling is off: DbUpdateConcurrencyException: Store update, insert, or delete statement affected an unexpected number of rows (0) (i assume the concurrency here is the write-forwarding, as i am the only one using this test database). if connection pooling is on: with a timeout (i think) trying to read the affected rows.
using dapper (as a comparison), running INSERT INTO...; SELECT last_insert_id(); or INSERT INTO...; DO SLEEP(2); SELECT last_insert_id(); in a single call always returns an id of 0, while running INSERT INTO ...; in one call and then SELECT last_insert_id(); in a second call produces the correct id.
changing the value of aurora_replica_read_consistency doesn't seem to make a difference.
(in the primary region, everything works as expected - no timeouts or errors and the last_insert_id is correct)
i've been trying to play with settings, but i haven't really made any progress
i have opened an aws support case as well, but i was wondering if anyone has examples of a .net application working with write forwarding for a global aurora mysql database? especially with entityframework?
or knows of any gotchas or tips on where to focus an investigation?
=====================================
table creation:
CREATE TABLE IF NOT EXISTS `data` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`Key` varchar(100) NOT NULL,
`Value` varchar(45) DEFAULT NULL,
`Date` datetime DEFAULT NULL,
PRIMARY KEY (`ID`),
UNIQUE KEY `key_UNIQUE` (`Key`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
database entity:
[Table("data")]
public class SavedData
{
[Key]
public int ID { get; set; }
public string Key { get; set; }
public string Value { get; set; }
public DateTime Date { get; set; }
}
context:
[DbConfigurationType(typeof(MySqlEFConfiguration))]
public class DatabaseContext : DbContext, IDbContext
{
public DatabaseContext()
{
Configuration.LazyLoadingEnabled = false;
Configuration.UseDatabaseNullSemantics = true;
Database.SetInitializer<DatabaseContext>(null);
}
public virtual DbSet<SavedData> SavedData { get; set; }
}
access:
using var dbContext = new DatabaseContext();
var dbEntity = new Entities.SavedData
{
Key = toSave.Key,
Value = toSave.Value,
Date = DateTime.UtcNow
};
dbContext.SavedData.Add(dbEntity);
dbContext.SaveChanges();
Console.WriteLine($"updated entity id {dbEntity.ID}");
return dbEntity.ID;
the connection string looks like (connection pooling off)
password=*****;User Id=failover_user;server=********;port=3306;database=failover_test;Pooling=false;
or (connection pooling on)
password=*****;User Id=failover_user;server=********;port=3306;database=failover_test;MinimumPoolSize=1;maximumpoolsize=5;
Related
I'm having an issue with inserting new rows into my MySQL database. I'm using Spring Boot with Spring Boot Data JPA.
Since MySQL doesn't support sequences, I decided to try and make my own sequence generator table. This is basically what I've done.
I created a sequences table that uses an auto increment field (used as my id's for my tables).
Created a function, sequences_nextvalue() which inserts into the sequences table and returns the new auto incremented id.
I then created triggers on each table that get triggered before insertion and replaces the id field with the result of calling sequences_nextvalue().
So this is working fine when inserting new rows. I'm getting unique ids across all tables. The issue I'm having is with my JPA entities.
#Entity
#Inheritance(strategy=InheritanceType.TABLE_PER_CLASS)
public abstract class AbstractBaseClass {
#Id
private Integer id = -1;
...
}
#Entity
public class ConcreteClass1 extends AbstractBaseClass {
...
}
#Entity
public class ConcreteClass2 extends AbstractBaseClass {
...
}
I want to be able to query from the abstract base class so I've placed my #Id column in that class and used #Entity with InheritanceType.TABLE_PER_CLASS. I've also initialized the id to -1 since an id is required to call save() from my spring crud repository.
After calling the save() function of my Spring data CrudRepository, the -1 for id properly gets replaced by the MySQL trigger but the resulting entity returned by save() doesn't return with the new id but instead retains the -1. After looking at the SQL logs, a select statement is not being called after insertion to get the new id but instead the original entity is being returned.
Is it possible to force Hibnerate to re-select the entity after insertion to get the new id when you're not using #GeneratedValue?
Any help is greatly appreciated.
Just wanted to provide an update on this question. Here is my solution.
Instead of creating MySQL TRIGGER's to replace the id on INSERT, I created a Hibernate IdentifierGenerator which executes a CallableStatement to get and return a new id.
My abstract base class now looks like this.
#Entity
#Inheritance(strategy=InheritanceType.TABLE_PER_CLASS)
public abstract class AbstractBaseClass {
#Id
#GenericGenerator(name="MyIdGenerator", strategy="com.sample.model.CustomIdGenerator" )
#GeneratedValue(generator="MyIdGenerator" )
private Integer id;
...
}
and my generator looks like this.
public class CustomIdGenerator implements IdentifierGenerator {
private Logger log = LoggerFactory.getLogger(CustomIdGenerator.class);
private static final String QUERY = "{? = call sequence_nextvalue()}";
#Override
public Serializable generate(SessionImplementor session, Object object) throws HibernateException {
Integer id = null;
try {
Connection connection = session.connection();
CallableStatement statement = connection.prepareCall(QUERY);
statement.registerOutParameter(1, java.sql.Types.INTEGER);
statement.execute();
id = statement.getInt(1);
} catch(SQLException e) {
log.error("Error getting id", e);
throw new HibernateException(e);
}
return id;
}
}
And just for reference
The sequences table.
CREATE TABLE sequences (
id INT AUTO_INCREMENT PRIMARY KEY,
thread_id INT NOT NULL,
created DATETIME DEFAULT CURRENT_TIMESTAMP
) ^;
The sequence_nextvalue function
CREATE FUNCTION sequence_nextvalue()
RETURNS INTEGER
NOT DETERMINISTIC
MODIFIES SQL DATA
BEGIN
DECLARE nextvalue INTEGER;
INSERT INTO sequences (thread_id) VALUE (CONNECTION_ID());
SELECT id FROM sequence_values ORDER BY created DESC LIMIT 1 INTO nextvalue;
RETURN nextvalue;
END ^;
I'm attempting to configure Fluent NHibernate with Automapping for MySQL and I've run into a problem. I've googled quite a bit and tried many different things, but haven't found a solution yet.
Anyway, here's my NHibernate configuration
return
Fluently.Configure()
.Database(
MySQLConfiguration.Standard.ConnectionString(cs => cs.FromConnectionStringWithKey("Db"))
.Dialect<NHibernate.Dialect.MySQL5Dialect>()
.ShowSql()
)
.Mappings(
x =>
x.AutoMappings
.Add(
FluentNHibernate.Automapping.AutoMap.AssemblyOf<CoreRegistry>()
.Where (t => t.IsDefined(typeof(AutoMap), false))
.Conventions.Setup (c => {
c.Add<NullConvention>();
c.Add<PrimaryKeyConvention> ();
c.Add<TableNameConvention> ();
})
)
).BuildSessionFactory();
Here's my model:
[AutoMap]
public class MyNewTable
{
public virtual int Id {get; set;}
public virtual string Column1 {get; set;}
[Null]
public virtual string Column2 {get; set;}
}
Here's my code attempting to insert the record:
using (var session = sf.OpenSession())
using (var tx = session.BeginTransaction()) {
var myReallyTable = new MyNewTable
{
Column2 = null,
Column1 = "ghrtehrthrtete"
};
session.Save (myReallyTable);
tx.Commit ();
}
Lastly, here is the sql that it is actually generating:
INSERT INTO myproject.Core.MyNewTable, myproject.Core, Version=1.0.4911.33346, Culture=neutral, PublicKeyToken=null (Column1, Column2) VALUES (?, ?)
For some reason it is injecting my project's assembly name before my table name and also appending the assembly version and stuff after it. I've tried setting the default schema to my database name. I tried switching the dialect to MySQLDialect, but that didn't work either.
If anyone's got any light to shine on this I'd be very grateful.
Thanks
Well I get to come back and show what a jackass I am.
I had a table name convention that was using instance.Name instead of instance.EntityType.Name. Oh well, at least this post will be here if anyone wants to know how to AutoMap with MySQL. :D
I'm using Spring 3.2, Hibernate 4 and MySQL. I have a self referencing class called Lecturers which has annotations implementing a parent/child one to many relationship. I have a problem with implementing a controller and form for saving a parent and child from the same table. It's a self-referencing class.
My DB:
CREATE TABLE `lecturers` (
`lecturer_id` BIGINT(10) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(255) NULL DEFAULT NULL,
`email` VARCHAR(255) NULL DEFAULT NULL,
`checker_id` BIGINT(20) NULL DEFAULT NULL,
PRIMARY KEY (`lecturer_id`),
FOREIGN KEY (`checker_id`) REFERENCES `lecturers` (`lecturer_id`)
The Java class
#ManyToOne
#JoinColumn(name="checker_id")
private Lecturer checker;
#OneToMany(cascade = CascadeType.ALL, mappedBy="checker", orphanRemoval=true)
private List<Lecturer> lecturers = new ArrayList<Lecturer>();
And the class also has this method
#Transient
public void addLecturer(Lecturer lecturer) {
if(lecturers == null) {
lecturers = new ArrayList<Lecturer>();
//lecturers = new HashSet<Lecturer>();
}
lecturer.setChecker(this);
lecturer.setLecturers(lecturers);
//lecturer.setLecturers(lecturers);
lecturers.add(lecturer);
// TODO Auto-generated method stub
}
I then set up a DAO and Service layer for implementing a CRUD operations. The create method is this:
Session session = sessionFactory.getCurrentSession();
transaction = session.beginTransaction();
// Create new lecturers
Lecturer lecturer1 = new Lecturer();
lecturer1.setName(name);
lecturer1.setEmail(email);
Lecturer lecturer2 = new Lecturer();
lecturer2.setName(name);
lecturer2.setEmail(email);
// Create new checker
Lecturer checker = new Lecturer();
checker.setName(name);
checker.setEmail(email);
checker.setChecker(checker);
List<Lecturer> lecturers = new ArrayList<Lecturer>();
lecturers.add(lecturer1);
lecturers.add(lecturer2);
lecturer1.setChecker(checker);
lecturer2.setChecker(checker);
checker.addLecturer(lecturer1);
checker.addLecturer(lecturer2);
checker.setLecturers(lecturers);
session.save(checker);
session.save(lecturer1);
session.save(lecturer2);
My requirement is now to provide a form that will be used to match a parent (Checker) to one or more children (Lecturers) and save the match to the database. I'm asking how I should go about saving the relationship. Should I create the parent and children separately, then match a parent using the id to a children selected from say a drop down list? I'm not sure how to make sure the relationship between a checker and its respective lecturers is saved.
I then created a main class for testing the relationship and to see if it works. Inserting data into the db works but when I want to list it I get this:
Name: Mark
Email: ma#msn.com
Checker: com.professional.project.domain.Lecturer#439942
ID: 22
I should get the name of the checker back which I already added but it's not coming back.
I would appreciate some help on how to proceed.
First of all, your addLecturer() method has a bug. It shouldn't set the lecturers list of the child to the current lecturer's list:
public void addLecturer(Lecturer lecturer) {
if (lecturers == null) {
lecturers = new ArrayList<Lecturer>(); // OK : lazy initialization
}
lecturer.setChecker(this); // OK : set the parent of the child to this
lecturer.setLecturers(lecturers); // this line should be removed : the child's children shouldn't be the same as this lecturer's children
lecturers.add(lecturer); // OK : ad the child to the list of children
}
When you get a lecturer, you obtain the following as the checker :
Checker: com.professional.project.domain.Lecturer#439942
The above is just the result of the call to the default toString() method on the checker. To get its name, call getName() on the checker. If you want the toString() method to return the name, then implement it that way:
#Override
public String toString() {
return name;
}
I have a hierarchical relationship defined for one of my tables where the relationship is stored in a separate join table. The join table also contains information about the type of relationship. The (simplified) schema looks like:
Bills
ID int IDENTITY(1,1) NOT NULL (PK)
Code varchar(5) NOT NULL
Number varchar(5) NOT NULL
...
BillRelations
ID int IDENTITY(1,1) NOT NULL (PK)
BillID int NOT NULL
RelatedBillID int NOT NULL
Relationship int NOT NULL
...
I have FK relationships defined on BillID and RelatedBillID to ID in the Bills table.
I'm trying to map this in Entity Framework Code First with little success. My classes look like the following. Ignore the RelationshipWrapper stuff, it's a wrapper class around an enum named Relationship that corresponds to the value in the Relationship column.
public class Bill
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int ID { get; set; }
[Required]
[StringLength(5)]
public string Code { get; set; }
[Required]
[StringLength(5)]
public string Number { get; set; }
public virtual ICollection<BillRelation> RelatedBills { get; set; }
...
}
public class BillRelation
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int ID { get; set; }
public long BillID { get; set; }
[ForeignKey("BillID")]
public virtual Bill Bill { get; set; }
public long RelatedBillID { get; set; }
[ForeignKey("RelatedBillID")]
public virtual Bill RelatedBill { get; set; }
public RelationshipWrapper Relationship { get; set; }
...
}
I've tried this in various incarnations, both using explicitly defined relationships via the ModelBuilder on the DbContext and data annotations only, but the above defines what I'm looking for. I'm using collections because the the foreign key properties aren't primary keys.
Using this set up I get an error: Bill_ID column is not defined (or something similar).
If I got the ModelBuilder route using the following, I get an error that "The table BillRelations does not exist in the database."
modelBuilder.Entity<Bill>().HasMany( b => b.RelatedBills )
.WithRequired( r => r.Bill )
.Map( m => m.MapKey( "BillID" ).ToTable( "BillRelations" ) );
modelBuilder.Entity<BillRelation>().HasRequired( r => r.RelatedBill )
.WithRequiredDependent()
.Map( m => m.MapKey( "RemoteBillID" ).ToTable( "BillRelations" ) );
I have been able to make it work by defining only one half of the relationship, Bills -> BillRelations, then using a Join in my repository to fill in an [NotMapped] RelatedBill property on the BillRelations class for each of the related bills in the Bill's RelatedBills collection. I'd rather not do this if I can help it.
The only other solution I've thought of is to model each relationship in a separate table (there are 4 types) and use a standard Bill<->Bill mapping through the join table for each of the 4 relationship types -- again I'd rather not do this if I can avoid it.
Can anyone see what I'm doing wrong or tell me if what I want to do is even possible in EF Code First 4.1?
Just a few ideas:
Your mapping with data annotations doesn't work because EF Code First conventions don't recognize which navigation properties belong together. Obviously you want to associate Bill.RelatedBills with BillRelation.Bill. But because there is a second navigation property BillRelation.RelatedBill refering to the Bill entity as well the AssociationInverseDiscoveryConvention can't be applied to recognize the correct relation. This convention only works if you have exactly one pair of navigation properties on the entities. As a consequence, EF assumes actually three relationships, each with only one exposed end in the model. The relationship where Bill.RelatedBills belongs to assumes a not exposed foreign key on the other side according to EF default naming conventions - which is Bill_ID with underscore. It doesn't exist in the database, hence the exception.
In your Fluent API mapping I would just try to remove ...ToTable(...) altogether. I believe that it is not necessary as the mapping knows anyway which table the foreign key belongs to. This will possibly fix the second error ("Table ... does not exist....")
Your second mapping - the one-to-one relationship - does possibly not work as expected because one-to-one relationships are "usually" mapped with a shared primary key association between the tables in the database. (I am not sure though if shared primary keys are really required by EF.) Because your BillId column seems to be a foreign key which is not a primary key at the same time I would try to map the relationship as one-to-many. Moreover because your foreign key columns are exposed as properties in the model you should use HasForeignKey instead of MapKey:
modelBuilder.Entity<Bill>()
.HasMany( b => b.RelatedBills )
.WithRequired( r => r.Bill )
.HasForeignKey ( r => r.BillID );
// turn off/on cascade delete by chaining
// .WillCascadeOnDelete(true/false)
// here at the end
modelBuilder.Entity<BillRelation>()
.HasRequired( r => r.RelatedBill )
.WithMany()
.HasForeignKey ( r => r.RelatedBillID );
// turn off/on cascade delete by chaining
// .WillCascadeOnDelete(true/false)
// here at the end
Edit
It is possible that the whole Fluent mapping is not necessary if you put the [InverseProperty] attribute on one of the navigation properties - for example in your Bill class:
[InverseProperty("Bill")]
public virtual ICollection<BillRelation> RelatedBills { get; set; }
In this attribute you specify the name of the associated navigation property on the related entity. This binds Bill.RelatedBills and BillRelation.Bill together to a pair of navigation properties being the ends of the same association. I hope that EF will do the correct thing with the remaining navigation property BillRelation.RelatedBill, i.e. create a one-to-many relationship - I hope... If the default cascading delete won't work for you, you are forced to use Fluent API though since there is no data annotation attribute to configure cascading delete.
I have an JPA entity like this:
#Entity
#Table(name = "category")
public class Category implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Basic(optional = false)
#Column(name = "id")
private Integer id;
#Basic(optional = false)
#Column(name = "name")
private String name;
#OneToMany(cascade = CascadeType.ALL, mappedBy = "category")
private Collection<ItemCategory> itemCategoryCollection;
//...
}
Use Mysql as the underlying database. "name" is designed as a unique key. Use Hibernate as JPA provider.
The problem with using merge method is that because pk is generated by db, so if the record already exist (the name is already there) then Hibernate will trying inserting it to db and I will get an unique key constrain violation exception and not doing the update . Does any one have a good practice to handle that? Thank you!
P.S: my workaround is like this:
public void save(Category entity) {
Category existingEntity = this.find(entity.getName());
if (existingEntity == null) {
em.persist(entity);
//code to commit ...
} else {
entity.setId(existingEntity.getId());
em.merge(entity);
//code to commit ...
}
}
public Category find(String categoryName) {
try {
return (Category) getEm().createNamedQuery("Category.findByName").
setParameter("name", categoryName).getSingleResult();
} catch (NoResultException e) {
return null;
}
}
How to use em.merge() to insert OR update for jpa entities if primary key is generated by database?
Whether you're using generated identifiers or not is IMO irrelevant. The problem here is that you want to implement an "upsert" on some unique key other than the PK and JPA doesn't really provide support for that (merge relies on database identity).
So you have AFAIK 2 options.
Either perform an INSERT first and implement some retry mechanism in case of failure because of a unique constraint violation and then find and update the existing record (using a new entity manager).
Or, perform a SELECT first and then insert or update depending on the outcome of the SELECT (this is what you did). This works but is not 100% guaranteed as you can have a race condition between two concurrent threads (they might not find a record for a given categoryName and try to insert in parallel; the slowest thread will fail). If this is unlikely, it might be an acceptable solution.
Update: There might be a 3rd bonus option if you don't mind using a MySQL proprietary feature, see 12.2.5.3. INSERT ... ON DUPLICATE KEY UPDATE Syntax. Never tested with JPA though.
I haven't seen this mentioned before so I just would like to add a possible solution that avoids making multiple queries. Versioning.
Normally used as a simple way to check whether a record being updated has gone stale in optimistic locking scenario's, columns annotated with #Version can also be used to check whether a record is persistent (present in the db) or not.
This all may sound complicated, but it really isn't. What it boils down to is an extra column on the record whose value changes on every update. We define an extra column version in our database like this:
CREATE TABLE example
(
id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
version INT, -- <== It really is that simple!
value VARCHAR(255)
);
And mark the corresponding field in our Java class with #Version like this:
#Entity
public class Example {
#Id
#GeneratedValue
private Integer id;
#Version // <-- that's the trick!
private Integer version;
#Column(length=255)
private String value;
}
The #Version annotation will make JPA use this column with optimistic locking by including it as a condition in any update statements, like this:
UPDATE example
SET value = 'Hello, World!'
WHERE id = 23
AND version = 2 -- <-- if version has changed, update won't happen
(JPA does this automatically, no need to write it yourself)
Then afterwards it checks whether one record was updated (as expected) or not (in which case the object was stale).
We must make sure nobody can set the version field or it would mess up optimistic locking, but we can make a getter on version if we want. We can also use the version field in a method isPersistent that will check whether the record is in the DB already or not without ever making a query:
#Entity
public class Example {
// ...
/** Indicates whether this entity is present in the database. */
public boolean isPersistent() {
return version != null;
}
}
Finally, we can use this method in our insertOrUpdate method:
public insertOrUpdate(Example example) {
if (example.isPersistent()) {
// record is already present in the db
// update it here
}
else {
// record is not present in the db
// insert it here
}
}