HikariCP connection pools hangs in spring-batch 4.0 - mysql

Question

We have updated our Spring Batch based application to the newest version of Spring Batch 4.x and Boot 2.0.x and having some problems with the HikariCP MySQL connection pool.
If I use the default maxPoolSize of 10 connections and I'm using 5 threads which each starts a simple dummy job this works ok. (Using SimpleJobLauncher). All Jobs are started immediately and after that the HikariCP pool has all used connections returned.
The problem starts when I do the same but then with 11 threads. Then I see that the application hangs until a connection timeout, and then the jobs are started. I expected that the first 10 jobs will start and after the first one finished the 11th job...
Before the jobs start I see this:
HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
Now I start 11 jobs and the application will hang for 30 seconds...
2019-01-21 11:25:01.256 INFO 10194 --- [ver (CliServer)] c.n.r.i.batch.cli.CliClientHandler : Start 11 jobs
2019-01-21 11:25:30.928 DEBUG 10194 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=10, active=10, idle=0, waiting=2)
2019-01-21 11:25:35.990 DEBUG 10194 --- [ Thread-10] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Timeout failure stats (total=10, active=10, idle=0, waiting=1)
2019-01-21 11:25:35.992 DEBUG 10194 --- [ Thread-6] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Timeout failure stats (total=10, active=10, idle=0, waiting=1)
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:305)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:378)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:474)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:289)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:181)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy74.createJobExecution(Unknown Source)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:131)
at com.xx.yy.zz.batch.cli.commands.DummyCliCommand$1.run(DummyCliCommand.java:54)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:697)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:196)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:161)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:262)
... 12 more
And after this timeout the first jobs are started...
2019-01-21 11:25:36.061 INFO 10194 --- [ Thread-11] c.n.r.i.b.u.c.BlockingThreadPoolExecutor : [sftp-executor] 3 running jobs, 0 queued jobs, 2000 jobs allowed
2019-01-21 11:25:36.062 INFO 10194 --- [ Thread-5] c.n.r.i.b.u.c.BlockingThreadPoolExecutor : [sftp-executor] 3 running jobs, 0 queued jobs, 2000 jobs allowed
2019-01-21 11:25:36.061 INFO 10194 --- [ Thread-3] c.n.r.i.b.u.c.BlockingThreadPoolExecutor : [sftp-executor] 3 running jobs, 0 queued jobs, 2000 jobs allowed
2019-01-21 11:25:36.063 INFO 10194 --- [pool-1-thread-2] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=dummyJob]] launched with the following parameters: [{dummy=f5c864d9-8a74-4d51-9a53-e3fc5c0b731d}]
Version:
- HikariCP 2.7.9
- spring-batch-core 4.0.1
- Hibernate Core 5.2.17.Final
- spring-boot 2.0.5
I don't understand why the pool hangs? (The Dummy Jobs do only sleep a second)
HikariCP config
allowPoolSuspension.............false
autoCommit......................true
catalog.........................none
connectionInitSql...............none
connectionTestQuery.............none
connectionTimeout...............30000
dataSource......................none
dataSourceClassName.............none
dataSourceJNDI..................none
dataSourceProperties............{usePipelineAuth=false, password=<masked>, prepStmtCacheSqlLimit=2048, rewriteBatchedStatements=false, useSSL=false, cachePrepStmts=true, useServerPrepStmts=true, prepStmtCacheSize=250}
driverClassName................."com.mysql.jdbc.Driver"
healthCheckProperties...........{}
healthCheckRegistry.............none
idleTimeout.....................600000
initializationFailTimeout.......1
isolateInternalQueries..........false
jdbcUrl.........................jdbc:mysql://localhost:3306/development
leakDetectionThreshold..........60000
maxLifetime.....................1800000
maximumPoolSize.................10
metricRegistry..................none
metricsTrackerFactory...........none
minimumIdle.....................10
password........................<masked>
poolName........................"HikariPool-1"
readOnly........................false
registerMbeans..................true
scheduledExecutor...............none
schema..........................none
threadFactory...................internal
transactionIsolation............default
username........................"user"
validationTimeout...............5000

Answer 1

The thread limit needs to be less than the maximum pool size of the datasource that spring batch is using with its jobrepository. So for your case, since your datasource pool size is 10, you should use no more than 9 threads.
In the case of an asynchronous taskExecutor(TaskExecutor) the number of concurrent tasklet executions can be throttled (beyond any throttling provided by a thread pool). The throttle limit should be less than the data source pool size used in the job repository for this step.
org.springframework.batch.core.step.builder.AbstractTaskletStepBuilder#throttleLimit