How do I debug HikariCP losing connections? - mysql

Question

I use HikariCP with Play 2.6.10. The application will run fine for days, and then all of our connections will leak. I have leakDetectionThreshold turned on, so we get stack traces for leaked connections like:
2018-01-24 06:29:00,857 - [WARN] - from com.zaxxer.hikari.pool.ProxyLeakTask in
HikariPool-2 housekeeper
Connection leak detection triggered for com.mysql.jdbc.JDBC4Connection#65cd084 on thread pool-1-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.jav
a:85)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
at play.api.db.DefaultDatabase.withConnection(Databases.scala:152)
at play.api.db.DefaultDatabase.withConnection(Databases.scala:148)
at models.summaries.ActionSummary$.listByStripe(ActionSummary.scala:137)
I only use the connections through Play's withConnection, so they should be returned to the pool automatically. A thread dump, when the application is in a broken state, shows that all threads inside of a withConnection block are stuck on...
"application-akka.mysql-context-122" #32142 prio=5 os_prio=0 tid=0x00007fca7812a
000 nid=0x28e6 waiting on condition [0x00007fca7541c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for (a java.util.concurrent.Sync
hronousQueue$TransferQueue)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215
)
at java.util.concurrent.SynchronousQueue$TransferQueue.awaitFulfill(Sync
hronousQueue.java:764)
at java.util.concurrent.SynchronousQueue$TransferQueue.transfer(Synchron
ousQueue.java:695)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at com.zaxxer.hikari.util.ConcurrentBag.borrow(ConcurrentBag.java:157)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:165)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85)
at play.api.db.DefaultDatabase.getConnection(Databases.scala:142)
at play.api.db.DefaultDatabase.withConnection(Databases.scala:152)
at play.api.db.DefaultDatabase.withConnection(Databases.scala:148)
...waiting for a connection to be available, which should mean that none are currently holding a connection. I have no idea how any connection could possible be leaked, but apparently all of them have. We see logging lines like:
2018-01-24 06:19:21,297 - [DEBUG] - from com.zaxxer.hikari.pool.HikariPool in application-akka.mysql-context-129
HikariPool-2 - Timeout failure stats (total=10, active=10, idle=0, waiting=15)
The only unusual thing we are doing is calling setNetworkTimeout on each connection we obtain, sometimes with a timeout as low as 10 seconds. This is done to ensure that queries fail fast if we lose connection to the DB.
I'm not sure what to do next debugging this. It looks like maybe a potential issue between Hikari and Play, or something broken with MySQL and setNetworkTimeout.