Spring Data JPA Too many connections - mysql

me and my collegues are working on a SpringBoot project, we work simultaneously and we all connect to the same mysql database.
Trouble is that after a while some of us will no longer be able to connect, error is Too many connections, now I've spoken to the db administrator and he raised the number of max connections to 500 (it was something like 150 before this), but we still get the error, how can I fix this? the only configuration properties we use are these:
spring.datasource.url= ...
spring.datasource.username= ...
spring.datasource.password= ...
spring.datasource.hikari.maximum-pool-size=25
Maybe jpa opens a new connection every time he does a query but doesn't close it? I don't know I'm clueless here
EDIT:
I've been asked to show some code regarding the interactions with the database so here it is:
#Autowired
private EmployeeDAO employeeDAO;
#Autowired
private LogDAO logDAO;
#Autowired
private ContrattoLavoroDAO contrattoLavoroDAO;
#Override
public void deleteEmployeeById(Long idEmployee, String username) {
contrattoLavoroDAO.deleteContrattoByEmpId(idEmployee);
employeeDAO.deleteById(idEmployee);
LogEntity log = new LogEntity();
LocalDateTime date = LocalDateTime.now();
log.setData(date);
log.setUser(username);
log.setCrud("delete");
log.setTabella("Employees");
log.setDescrizione("l'utente " + username + " ha rimosso il dipendente con matricola " + idEmployee);
logDAO.save(log);
}
and here's the model for a DAO:
public interface ContrattoLavoroDAO extends JpaRepository<ContrattoLavoroEntity, Long> {
#Modifying
#Query(value = "DELETE contratto_lavoro, employee_contratto FROM contratto_lavoro" + " INNER JOIN"
+ " employee_contratto ON employee_contratto.id_contratto = contratto_lavoro.id_contratto" + " WHERE"
+ " contratto_lavoro.id_contratto = ?1", nativeQuery = true)
public void deleteContrattoByEmpId(Long empId);
}

You set the Hikari maximum pool size to 25.
Which is pretty high for a development environment.
But this shouldn't be a problem, because it's only the maximum, right?
Well Hikaris documentation says:
🔢minimumIdle
This property controls the minimum number of idle connections that HikariCP tries to maintain in the pool. If the idle connections dip below this value and total connections in the pool are less than maximumPoolSize, HikariCP will make a best effort to add additional connections quickly and efficiently. However, for maximum performance and responsiveness to spike demands, we recommend not setting this value and instead allowing HikariCP to act as a fixed size connection pool. Default: same as maximumPoolSize
🔢maximumPoolSize
This property controls the maximum size that the pool is allowed to reach, including both idle and in-use connections. Basically this value will determine the maximum number of actual connections to the database backend. A reasonable value for this is best determined by your execution environment. When the pool reaches this size, and no idle connections are available, calls to getConnection() will block for up to connectionTimeout milliseconds before timing out. Please read about pool sizing. Default: 10
If I'm reading this correct it means each developer on your team opens 25 connections when they start the application. Another 25 when they start an integration test which starts a new application context. If the tests of the test suite have different configuration each set of configurations will have their own set of 25 connections in use.
The quick solution is to reduce the maximumPoolSize significantly for your development environment. I'd recommend 2. This is enough to allow for one normal transaction and one background process.
It will throw exceptions if the application requires more connections, which is probably a good thing since in most cases it shouldn't.
Above that you might want to set the minimumIdle to 1 or 0, so the application doesn't consume shared resources if an application is just running because no one shut it down yet.
Mid term you probably want to get rid of having a central database for development.
With the availability of TestContainers there really isn't a reason anymore to not have a local database for each developer.
As a nice side effect it will ensure that all the schema update scripts work properly.

Related

Managing DBCP connections with 45 second idle drop

We have a Java web application that utilizes several different connections to a variety of database servers. Somewhat recently, a couple databases servers (managed by a different department) were reconfigured to sit behind a firewall which cuts any 45 second idle connection. I was told that there should be a keepAlive property for our connections that we can set. However, that doesn't seem to be the case after some research. Instead I have to come to the conclusion that I must use properties like:
validationQuery
testOnBorrow
timeBetweenEvictionRunsMillis
etc. (Source: Commons DBCP)
The short of it is I have to ensure that the application retains its connection to these servers regardless of the 45 second timeout on the other side. I've run through a few configurations with little success (Example: When the connection drops, the connection is re-established sooner, but still drops rather quickly).
Here is the original code for the BasicDataSource object:
private BasicDataSource createDefaultDataSource() {
BasicDataSource ds = new BasicDataSource();
ds.setInitialSize(3);
ds.setMinIdle(1);
ds.setMaxIdle(5);
ds.setMaxActive(100);
ds.setMaxWait(3 * 1000);
ds.setTimeBetweenEvictionRunsMillis(30 * 1000);
ds.setMinEvictableIdleTimeMillis(60 * 1000);
ds.setLogAbandoned(true);
ds.setRemoveAbandoned(true);
ds.setRemoveAbandonedTimeout(60);
return ds;
}
And here is what I assumed would be the solution, yet doesn't seem to work how I expect:
private BasicDataSource createDefaultDataSource() {
BasicDataSource ds = new BasicDataSource();
ds.setInitialSize(3);
ds.setMinIdle(1);
ds.setMaxIdle(5);
ds.setMaxActive(100);
ds.setMaxWait(3 * 1000);
ds.setTimeBetweenEvictionRunsMillis(15 * 1000); // was 30 seconds
ds.setMinEvictableIdleTimeMillis(30 * 1000); // was 60 seconds
ds.setLogAbandoned(true);
ds.setRemoveAbandoned(true);
ds.setRemoveAbandonedTimeout(60);
ds.setValidationQuery("select 1");
ds.setTestOnBorrow(true);
ds.setTestOnReturn(true);
ds.setTestWhileIdle(true);
return ds;
}
Maybe my timing is off, perhaps there are other properties I have to set in conjunction to these, or maybe I should be going about this a different way.
What properties should be set/changed in order to make up for a 45 second idle drop policy?

How to implement using MYSQL jdbc in an Akka Actor

Hey I read this jdbc docs
https://www.playframework.com/documentation/2.1.0/ScalaDatabase
and this question
Is it good to put jdbc operations in actors?
Now I have an ActorClass for my mysql transaction, and this actor instantiated several times, whenever request comes. So each request would instantiate new actor. Is it safe for connection pool?
Can I use
val connection = DB.getConnection()
is connection object could handle async transaction?
So I could just a singleton to handle mysql connection and used it in all instantiated actors. Also if I want to use anorm, how do I make an implicit connection object?
Thanks
Your DB.getConnection() should be a promise[Connection] or a future[Connection] if you don't want to block the actor. (caveats at the end of the answer)
If your DB.getConnection() is synchronous (returning only connection without wrapping type) your actor will hang until it actually gets a connection from the pool while processing the actual message. It doesn't matter your DB being singleton or not, in the end it will hit the connection pool.
That being said, you can create actors to handle the messaging and other actors to handle persistence in the database, put them in different thread dispatchers giving more thread to database intensive. This is suggested also in the PlayFramework.
Caveats:
If you run futures inside the actor you are not ensure of the thread/timing it will run, I'm assuming you did something in the line of these (read the comments)
def receive = {
case aMessage =>
val aFuture = future(db.getConnection)
aFuture.map { theConn => //from previous line when you acquire the conn and when you execute the next line
//it could pass a long time they run in different threads/time
//that's why you should better create an actor that handles this sync and let
//akka do the async part
theConn.prepareStatemnt(someSQL)
//omitted code...
}
}
so my suggestion would be
//actor A receives,
//actor B proccess db (and have multiple instances of this one due to slowness from db)
class ActorA(routerOfB : ActorRef) extends Actor {
def recieve = {
case aMessage =>
routerOfB ! aMessage
}
}
class ActorB(db : DB) extends Actor {
def receive = {
case receive = {
val conn = db.getConnection //this blocks but we have multiple instances
//and enforces to run in same thread
val ps = conn.prepareStatement(someSQL)
}
}
}
You will need routing: http://doc.akka.io/docs/akka/2.4.1/scala/routing.html
As I know you couldn't run multiple concurrent query on a single connection for RDBMS ( I've not seen any resource/reference for async/non-blocking call for mysql even in C-API; ). To run your queries concurrently, you most have multiple connection instances.
DB.getConnection isn't expensive while you have multiple instances of connection. The most expensive area for working with DB is running sql query and waiting for its response.
To being async with your DB-calls, you should run them in other threads (not in the main thread-pool of Akka or Play); Slick does it for you. It manages a thread-pool and run your DB-calls on them, then your main threads will be free for processing income requests. Then you dont need to wrap your DB-calls in actors to being async.
I think you should take a connection from pool and return when it is done. If we go by single connection per actor what if that connection gets disconnect, you may need to reinitialize it.
For transactions you may want to try
DB.withTransaction { conn => // do whatever you need with the connection}
For more functional way of database access I would recommend to look into slick which has a nice API that can be integrated with plain actors and going further with streams.

Scala / Slick, "Timeout after 20000ms of waiting for a connection" error

The block of code below has been throwing an error.
Timeout after 20000ms of waiting for a connection.","stackTrace":[{"file":"BaseHikariPool.java","line":228,"className":"com.zaxxer.hikari.pool.BaseHikariPool","method":"getConnection"
Also, my database accesses seem too slow, with each element of xs.map() taking about 1 second. Below, getFutureItem() calls db.run().
xs.map{ x =>
val item: Future[List[Sometype], List(Tables.myRow)] = getFutureItem(x)
Await.valueAfter(item, 100.seconds) match {
case Some(i) => i
case None => println("Timeout getting items after 100 seconds")
}
}
Slick logs this with each iteration of an "x" value:
[akka.actor.default-dispatcher-3] [akka://user/IO-HTTP/listener-0/24] Connection was PeerClosed, awaiting TcpConnection termination...
[akka.actor.default-dispatcher-3] [akka://user/IO-HTTP/listener-0/24] TcpConnection terminated, stopping
[akka.actor.default-dispatcher-3] [akka://system/IO-TCP/selectors/$a/0] New connection accepted
[akka.actor.default-dispatcher-7] [akka://user/IO-HTTP/listener-0/25] Dispatching POST request to http://localhost:8080/progress to handler Actor[akka://system/IO-TCP/selectors/$a/26#-934408297]
My configuration:
"com.zaxxer" % "HikariCP" % "2.3.2"
default_db {
url = ...
user = ...
password = ...
queueSize = -1
numThreads = 16
connectionPool = HikariCP
connectionTimeout = 20000
maxConnections = 40
}
Is there anything obvious that I'm doing wrong that is causing these database accesses to be so slow and throw this error? I can provide more information if needed.
EDIT: I have received one recommendation that the issue could be a classloader error, and that I could resolve it by deploying the project as a single .jar, rather than running it with sbt.
EDIT2: After further inspection, it appears that many connections were being left open, which eventually led to no connections being available. This can likely be resolved by calling db.close() to close the connection at the appropriate time.
EDIT3: Solved. The connections made by slick exceeded the max connections allowed by my mysql config.
OP wrote:
EDIT2: After further inspection, it appears that many connections were being left open, which eventually led to no connections being available. This can likely be resolved by calling db.close() to close the connection at the appropriate time.
EDIT3: Solved. The connections made by slick exceeded the max connections allowed by my mysql config.

all pooled connections were in use and max pool size was reached

I am writing a .NET 4.0 console app that
Opens up a connection Uses a Data Reader to cursor through a list of keys
For each key read, calls a web service
Stores the result of a web service in the database
I then spawn multiple threads of this process in order to improve the maximum number of records that I can process per second.
When I up the process beyond about 30 or so threads, I get the following error:
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Is there an Server or client side option to tweak to allow me to obtain more connections fromn the connection pool?
I am calling a sql 2008 r2 DATABASE.
tHx
This sounds like a design issue. What's your total record count from the database? Iterating through the reader will be really fast. Even if you have hundreds of thousands of rows, going through that reader will be quick. Here's a different approach you could take:
Iterate through the reader and store the data in a list of objects. Then iterate through your list of objects at a number of your choice (e.g. two at a time, three at a time, etc) and spawn that number of threads to make calls to your web service in parallel.
This way you won't be opening multiple connections to the database, and you're dealing with what is likely the true bottleneck (the HTTP call to the web service) in parallel.
Here's an example:
List<SomeObject> yourObjects = new List<SomeObject>();
if (yourReader.HasRows) {
while (yourReader.Read()) {
SomeObject foo = new SomeObject();
foo.SomeProperty = myReader.GetInt32(0);
yourObjects.Add(foo);
}
}
for (int i = 0; i < yourObjects.Count; i = i + 2) {
//Kick off your web service calls in parallel. You will likely want to do something with the result.
Task[] tasks = new Task[2] {
Task.Factory.StartNew(() => yourService.MethodName(yourObjects[i].SomeProperty)),
Task.Factory.StartNew(() => yourService.MethodName(yourObjects[i+1].SomeProperty)),
};
Task.WaitAll(tasks);
}
//Now do your database INSERT.
Opening up a new connection for all your requests is incredibly inefficient. If you simply want to use the same connection to keep requesting things, that is more than possible. You can open a connection, and then run as many SqlCommand commands through that one connection. Simply keep the ONE connection around, and dispose of it after all your threading is done.
Please restart the IIS you will be able to connect

DataContext connection closed or transaction completed unexpectedly while submitting changes within a TransactionScope transaction?

Code
double timeout_in_hours = 6.0;
MyDataContext db = new MyDataContext();
using (TransactionScope tran = new TransactionScope( TransactionScopeOption.Required, new TransactionOptions(){ IsolationLevel= System.Transactions.IsolationLevel.ReadCommitted, Timeout=TimeSpan.FromHours( timeout_in_hours )}, EnterpriseServicesInteropOption.Automatic ))
{
int total_records_processed = 0;
foreach (DataRow datarow in data.Rows)
{
//Code runs some commands on the DataContext (db),
//possibly reading/writing records and calling db.SubmitChanges
total_records_processed++;
try
{
db.SubmitChanges();
}
catch (Exception err)
{
MessageBox.Show( err.Message );
}
}
tran.Complete();
return total_records_processed;
}
While the above code is running, it successfully completes 6 or 7 hundred loop iterations. However, after 10 to 20 minutes, the catch block above catches the following error:
{"The transaction associated with the current connection has completed but has not been disposed. The transaction must be disposed before the connection can be used to execute SQL statements."}
The tran.Complete call is never made, so why is it saying the transaction associated with the connection is completed?
Why, after successfully submitting hundreds of changes, does the connection associated with the DataContext suddenly enter a closed state? (That's the other error I sometimes get here).
When profiling SQL Server, there are just a lot of consecutive selects and inserts with really nothing else while its running. The very last thing the profiler catches is a sudden "Audit Logout", which I'm not sure if that's the cause of the problem or a side-effect of it.
Wow, the max timeout is limited by machine.config: http://forums.asp.net/t/1587009.aspx/1
"OK, we resolved this issue. apparently the .net 4.0 framework doesn't
allow you to set your transactionscope timeouts in the code as we have
done in the past. we had to make the machine.config changes by adding
< system.transactions> < machineSettings maxTimeout="02:00:00"/>
< defaultSettings timeout="02:00:00"/> < /system.transactions>
to the machine.config file. using the 2.0 framework we did not have
to make these entries as our code was overriding teh default value to
begin with."
It seems that the timeout you set in TransactionScope's constructor is ignored or defeated by a maximum timeout setting in the machine.config file. There is no mention of this in the documentation for the TransactionScope's constructor that accepts a time out parameter: http://msdn.microsoft.com/en-us/library/9wykw3s2.aspx
This makes me wonder, what if this was a shared hosting environment I was dealing with, where I could not access the machine.config file? There's really no way to break up the transaction, since it involves creating data in multiple tables with relationships and identity columns whose values are auto-incremented. What a poor design decision. If this was meant to protect servers with shared hosting, it's pointless, because such a long-running transaction would be isolated to my own database only. Also, if a program specifies a longer timeout, then it obviously expects a transaction to take a longer amount of time, so it should be allowed. This limitation is just a pointless handicap IMO that's going to cause problems. See also: TransactionScope maximumTimeout