What is the proper way to ensure EntityManager connections are closed? - mysql

There are 19 methods in our DAO layer, each is some variation of this:
public TicketProp saveTicketProp(TicketProp prop) {
EntityManager em = this.emf.createEntityManager();
try {
em.getTransaction().begin();
prop = (TicketProp) em.merge(prop);
em.getTransaction().commit();
return prop;
} finally {
em.close();
}
}
Meaning: In each method we handle our own transaction and close it in a finally block. We're testing a Jersey app, so our JUnit tests extend JerseyTest. Each test method instantiates a Grizzly container, runs the test, then shuts down the container. EntityManagerFactory is injected by spring. We're using JPA over Hibernate.
I'm monitoring the connections to our MySQL test DB and they're always high. One test alone runs the MySQL "Max_used_connections" variable to 38. For fun, I went and commented out all the em.close() calls, and the test still uses 38 connections.
I'm using Hibernate's built-in connection pooling (not for prod use, I know). I still expected some sort of intelligent pooling.
Am I handling the EntityManager wrong? How else can I close connections?

You should close the EntityManagerFactory at the end of your test. From the javadoc of EntityManagerFactory#close():
void javax.persistence.EntityManagerFactory.close()
Close the factory, releasing any resources that it holds. After a factory instance has been closed, all methods invoked on it will throw the IllegalStateException, except for isOpen, which will return false. Once an EntityManagerFactory has been closed, all its entity managers are considered to be in the closed state.
As a side note, you should actually rollback the transaction before closing the EM in the finally clause:
public TicketProp saveTicketProp(TicketProp prop) {
EntityManager em = this.emf.createEntityManager();
try {
em.getTransaction().begin();
prop = (TicketProp) em.merge(prop);
em.getTransaction().commit();
return prop;
} finally {
if (em.getTransaction().isActive()) {
em.getTransaction().rollback();
}
if (em.isOpen()) {
em.close();
}
}
}

Why do you think that EntityManager.close() always physically closes underlying connection? It's up to connection pool (you probably need to configure it and set the maximum number of simultaneously open connections).

Related

Spring Boot+ MySQL: LAZY Loading issue - No operations allowed after statement closed

I implement a Dashboard functionality that checks every time at program start a list of Requirement-Objects for a bunch of different characteristics like progress, missing data and alike and sets for each characteristic a dedicated beacon on the UI.
protected void initializePerformanceIndicator() {
try {
updateA();
updateB();
...
updateF();
updateG();
} catch (Exception e) {
ErrorHandler.showError("Cannot show KPI Performance", e);
}
}
The checks have different compute demands some are faster some slower, therefore each of this checks runs in a dedicated Task to provide some feedback to the user. The skeleton of such a Task is always the same
protected void updateA() throws Exception {
Task<Void> task = new Task<Void>() {
#Override
protected Void call() throws Exception {
embeddedBudgetKPIController.setHyperlink("Budget", null);
embeddedBudgetKPIController.setToolTip("...");
ObservableList<UserRequirement> issues = FXCollections.observableArrayList();
List<UserRequirement> requirements = reqService.getAllUserRequirements(false); // all requirements of the selected product
for(UserRequirement req: requirements) {
if(*some criteria*) {
issues.add(req);
}
}
if(issues.isEmpty()) {
embeddedBudgetKPIController.setBeaconColor(Color.GREEN);
} else {
embeddedBudgetKPIController.setBeaconColor(Color.RED);
}
return null;
};
};
task.setOnSucceeded(e -> {
// Nothing to do
});
Thread tt = new Thread(task);
tt.start();
}
Before initializePerformanceIndicator is called, I retrieved already elsewhere the data from the database querying a number Spring Repositories:
protected final ObservableList<UserRequirement> allUserRequirements = FXCollections.observableArrayList();
public synchronized ObservableList<UserRequirement> getAllUserRequirements(boolean forceUpdate) throws Exception {
logger.debug(""); // show that this method is called
Product selectedProduct = SelectedScope.getSelectedProduct();
if(selectedProduct == null) {
throw new Exception("No selProduct selected");
}
if(forceUpdate || allUserRequirements.isEmpty()) {
allUserRequirements.clear();
allUserRequirements.addAll(epicRepository.findByProductAndRevisionSuccessorIsNull(selectedProduct));
allUserRequirements.addAll(themeRepository.findByProductAndRevisionSuccessorIsNull(selectedProduct));
allUserRequirements.addAll(userStoryRepository.findByProductAndRevisionSuccessorIsNull(selectedProduct));
allUserRequirements.addAll(tangibleRepository.findByProductAndRevisionSuccessorIsNull(selectedProduct));
}
return allUserRequirements;
}
and as you see updateBudgetKPIController calls getallUserRequirements with the parameter false. Therefore it returns the buffered result set and is not re-fetching data from database. So far everything is fine.
I can run each of these Tasks individually without problem. I tried a number combinations with 2 Tasks. Works fine, but the program will never show more than three or four beacons. Which ones are shown differs as well - what is expected as a consequence of the different Tasks. If I exceed three or four Tasks I often get no error at all, but the UI is just not showing more than three to four beacons.
Sometimes I do get an error message, which is
WARN 08:14 o.h.e.j.s.SqlExceptionHelper.logExceptions:137: SQL Error: 0, SQLState: S1009
ERROR 08:14 o.h.e.j.s.SqlExceptionHelper.logExceptions:142: No operations allowed after statement closed.
I debugged it, and realized that I was generating way too many select statements. The UserRequirement entity has almost a dozen OneToMany relations, some where defined with FetchType.LAZY, so I thought it would be better anyway to configure all these relations as
#OneToMany(fetch = FetchType.LAZY, mappedBy="parent", cascade = CascadeType.ALL)
Because of the LAZY loading, every Task tries to load additional data in the if(*some criteria*) part.
The problem did not disappear but I get more information, as the error is now
WARN 11:02 o.h.c.i.AbstractPersistentCollection.withTemporarySessionIfNeeded:278: Unable to close temporary session used to load lazy collection associated to no session
WARN 11:02 o.h.e.j.s.SqlExceptionHelper.logExceptions:137: SQL Error: 0, SQLState: S1009
ERROR 11:02 o.h.e.j.s.SqlExceptionHelper.logExceptions:142: No operations allowed after statement closed.
So I do have a LAZY loading issue.
I am using Spring Boot 2.1.6, MySQL 8.0.15 Community Server, Hibernate Core {5.3.10.Final}, Java 1.8.0_211 and the com.mysql.cj.jdbc.Driver
From a former issue, I have in my properties file the following configuration
# Prevent LazyInitializationException
spring.jpa.properties.hibernate.enable_lazy_load_no_trans=true
Don't know whether this has a side effect?!
Probably changing the LAZY loading to EAGER will fix it - haven't tried yet - but it would delay program start significantly. Therefore I would prefer a solution with LAZY loading.
Any ideas? I also appreciate any ideas regarding how to further isolate the root cause as the error message is not really explicit and I can't see which part of my code triggers it. Plus when I debug it, the behavior changes as I compute all Tasks sequentially rather then in parallel. Thank you in advance.
The issue was caused by different Tasks accessing the same getter of some of the entities. If the first getter call opened a connection, the second call got on it, and then the first call closed the ResultSet, the second call one was in trouble. Synchronizing the getter method solved the problem.

Spring retry connection until datasource is available

I have a docker-compose setup to start my SpringBoot application and a MySQL database. If the database starts first, then my application can connect successfully. But if my application starts first, no database exists yet, so the application throws the following exception and exits:
app_1 | 2018-05-27 14:15:03.415 INFO 1 --- [ main]
com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
app_1 | 2018-05-27 14:15:06.770 ERROR 1 --- [ main]
com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization
app_1 | com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
I could edit my docker-compose file to make sure the database is always up before the application starts up, but I want the application to be able to handle this case on its own, and not immediately exit when it cannot reach the database address.
There are ways to configure the datasource in the application.properties file to make the application reconnect to the database, as answered here and here. But that doesn't work for a startup connection to the datasource.
How can I make my SpringBoot application retry the connection at startup to the database at a given interval until it successfully connects to the database?
Set HikariCP's initializationFailTimeout property to 0 (zero), or a negative number. As documented here:
⌚initializationFailTimeout
This property controls whether the pool will "fail fast" if the pool cannot be seeded with an initial connection successfully. Any positive number is taken to be the number of milliseconds to attempt to acquire an initial connection; the application thread will be blocked during this period. If a connection cannot be acquired before this timeout occurs, an exception will be thrown. This timeout is applied after the connectionTimeout period. If the value is zero (0), HikariCP will attempt to obtain and validate a connection. If a connection is obtained, but fails validation, an exception will be thrown and the pool not started. However, if a connection cannot be obtained, the pool will start, but later efforts to obtain a connection may fail. A value less than zero will bypass any initial connection attempt, and the pool will start immediately while trying to obtain connections in the background. Consequently, later efforts to obtain a connection may fail. Default: 1
There is an alternative way to do this, which doesn't rely on a specific Connection Pool library or a specific database. Note that you will need to use spring-retry to achieve the desired behaviour with this approach
First you need to add spring-retry to your dependencies :
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
<version>${spring-retry.version}</version>
</dependency>
Then you can create a decorator over DataSource that will extends AbstractDataSource like bellow :
#Slf4j
#RequiredArgsConstructor
public class RetryableDataSource extends AbstractDataSource {
private final DataSource dataSource;
#Override
#Retryable(maxAttempts = 5, backoff = #Backoff(multiplier = 1.3, maxDelay = 10000))
public Connection getConnection() throws SQLException {
log.info("getting connection ...");
return dataSource.getConnection();
}
#Override
#Retryable(maxAttempts = 5, backoff = #Backoff(multiplier = 2.3, maxDelay = 10000))
public Connection getConnection(String username, String password) throws SQLException {
log.info("getting connection by username and password ...");
return dataSource.getConnection(username, password);
}
}
Then you will need to inject this custom DataSource decorator into Spring context by creating a custom BeanPostProcessor :
#Slf4j
#Order(value = Ordered.HIGHEST_PRECEDENCE)
#Component
public class RetryableDatabasePostProcessor implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
if(bean instanceof DataSource) {
log.info("-----> configuring a retryable datasource for beanName = {}", beanName);
return new RetryableDataSource((DataSource) bean);
}
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
return bean;
}
}
Last but not least you will need to enable Spring retry by adding #EnableRetry annotation to spring main class, example :
#EnableRetry
#SpringBootApplication
public class RetryableDbConnectionApplication {
public static void main(String[] args) {
SpringApplication.run(RetryableDbConnectionApplication.class, args);
}
}

Exception not cuaght with Entity Manager

I have an Entity Manager in my EJB
#PersistenceContext(unitName = "cnsbEntities")
private EntityManager em;
I populate an object and then I commit it in my DB, but if I have an exception, for duplicate ID, I can't catch it and I don't know why.
try{
em.merge(boelLog);
} catch (Exception e){
System.out.println("Generic Exception");
}
JPA uses transactions to send entity modifications to the database. You can specify those transactions manually through Bean Managed Transactions (BMT) or let the application server do it for you (Container Managed Transactions; the default).
So, you need to catch the exception at the end of the transaction, and not after calling merge() or persist() methods of EntityManager class. In your case, probably the transaction will end when you return from the last EJB object.
Example for Container Managed Transactions (the default):
#Stateless
public class OneEjbClass {
#Inject
private MyPersistenceEJB persistenceEJB;
public void someMethod() {
try {
persistenceEJB.persistAnEntity();
} catch(PersistenceException e) {
// here you can catch persistence exceptions!
}
}
}
...
#Stateless
public class MyPersistenceEJB {
// this annotation forces application server to create a new
// transaction when calling this method, and to commit all
// modifications at the end of it!
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public void persistAnEntity() {
// merge stuff with EntityManager
}
}
It's possible to specify when a method call (or any method call of an object of an EJB) must, can or must not create a new transaction. This is done through the #TransactionAttribute annotation. By default, every method of an EJB is configured as REQUIRED (same as specifying #TransactionAttribute(TransactionAttributeType.REQUIRED)), that tells the application to reuse (continue) the transaction that is active when that method was called, and create a new transaction if needed.
More about transactions here: http://docs.oracle.com/javaee/7/tutorial/doc/transactions.htm#BNCIH
More about JPA and JTA here: http://en.wikibooks.org/wiki/Java_Persistence/Transactions

Lock statement vs AutoResetEvent for LocalDb Thread Synchronization

I am writing an application that utilizes Windows Phone's LocalDB feature. I realized that I need to ensure that only one thread is performing operations on a given database, so I have created an AutoResetEvent object to coordinate the various threads vying for access to the database. My code goes pretty much like this:
class SomeClass
{
AutoResetEvent DatabaseLock = new AutoResetEvent(true);
public async void AddData(Person person)
{
await Task.Run(() =>
{
MyDataContext db = null;
try
{
this.DatabaseLock.WaitOne();
db = MyDataContext.GetInstance();
db.People.InsertOnSubmit(person);
db.SubmitChanges();
}
finally
{
if (db == null)
db.Dispose();
this.DatabaseLock.Set();
}
}
}
}
Obviously that's not the real code, but it's the same general pattern. Anyway, I decided to use the AutoResetEvent object here, as I have seen suggested online in multiple locations. However, I would be inclined to use a lock {...} statement instead.
Is there any reason to use AutoResetEvent? I feel like it's slow compared to locking an object.
You should use an AutoResetEvent when you need to signal to another thread.
In this case, you're just locking a resource, so the lock statement would be a better choice.

Handling database connection exceptions with Linq to SQL and Rx

I am trying learn how to best use the Reactive Extensions library and have set up simple test WPF application to view a logging database table. In a ViewModel class I am populating an ObservableCollection with the first 100 log entries from a Linq to Sql DataContext and I'm trying to use Rx to keep the UI responsive.
The following snippet works unless the database is unavailable at which point the app throws an exception and crashes. Where would be the best place to handle database connection exceptions and why are they not handled by the OnError method of the Observer?
ObservableCollection<LogEntry> _logEntries = new ObservableCollection<LogEntry>();
DataContext dataContext = new DataContext( "connection string" );
(from e in dataContext.LogEntries
select e).Take( 100 ).ToObservable()
.SubscribeOn( Scheduler.ThreadPool )
.ObserveOnDispatcher()
.Subscribe( _logEntries.Add, ex => System.Diagnostics.Debug.WriteLine( ex.ToString() ) );
Try this instead of ToObservable:
public static IObservable<T> SafeToObservable(this IEnumerable<T> This)
{
return Observable.Create(subj => {
try {
foreach(var v in This) {
subj.OnNext(v);
}
subj.OnCompleted();
} catch (Exception ex) {
subj.OnError(ex);
}
return Disposable.Empty;
});
}
In general though, this isn't a great use of Rx since the data source isn't very easy to Rx'ify - in fact, the code will execute most of the work on the UI thread, send it out to random worker threads, then send it back (i.e. completely wasted work). Task + Dispatcher.BeginInvoke might suit you better here.