Spring Batch - Partitioning Timeout - mysql

Question

I have to migrate around millions of blob records from multiple mysql databases to a physical location as files over WAN network.
I chose to use Spring Batch and has already made it work. However, I am struggling with a timeout error happen with random partitioned steps.
Here is some context,
There are multiple MySql database store >10m records in 20 years.
The source tables indexed two composite keys in varchar datatype (there is no ID key) so I have to use an UN-indexed column in date-time format to partitioning the records by year and week to keep the number of records per partition reasonably at average 200 records. If there is any better advice, it would be welcome!
My issue: When the records per partition is high enough, the stepExecutors will randomly failed due to time out
Could not open JDBC Con nection for transaction; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms
I have done some tweaks with the DataSource properties and Transaction properties but no luck. Can I get some advice please! Thanks
Terminal log:
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Con
nection for transaction; nested exception is
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:309)
~[spring-jdbc-5.3.16.jar:5.3.16]
...
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
2022-03-05 10:05:43.146 ERROR 15624 --- [main] o.s.batch.core.step.AbstractStep : Encountered an error executing step managerStep in job mainJob
org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step at ...
The job is marked as [FAILED] or [UNKNOWN] sometimes, and not restartable.
org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:112) ~[spring-batch-core-4.3.5.jar:4.3.5]
2022-03-05 10:05:43.213 INFO 15624 --- [main] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=mainJob]] completed with the following parameters: [{run.id=20}] and the following status: [FAILED] in 3m13s783ms
2022-03-05 10:05:43.590 INFO 15624 --- [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Shutdown initiated...
2022-03-05 10:05:43.624 INFO 15624 --- [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Shutdown completed.
2022-03-05 10:05:43.626 INFO 15624 --- [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2022-03-05 10:05:43.637 INFO 15624 --- [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
Datasource builder: I have tried to increase the connection timeout and pool size, but it seems not be applied.
#Bean(name = "srcDataSource")
// #ConfigurationProperties(prefix = "spring.datasource.hikari")
public HikariDataSource dataSource() {
HikariDataSource hikariDS = new HikariDataSource();
hikariDS.setDriverClassName("com.mysql.jdbc.Driver");
hikariDS.setJdbcUrl("jdbc:mysql://dburl");
hikariDS.setUsername("dbuser");
hikariDS.setPassword("dbpwd");
// properties below does not solve the problem
hikariDS.setMaximumPoolSize(16);
hikariDS.setConnectionTimeout(30000);
// hikariDS.addDataSourceProperty("serverName",
// getConfig().getString("mysql.host"));
// hikariDS.addDataSourceProperty("port", getConfig().getString("mysql.port"));
// hikariDS.addDataSourceProperty("databaseName",
// getConfig().getString("mysql.database"));
// hikariDS.addDataSourceProperty("user", getConfig().getString("mysql.user"));
// hikariDS.addDataSourceProperty("password",
// getConfig().getString("mysql.password"));
// hikariDS.addDataSourceProperty("autoReconnect", true);
// hikariDS.addDataSourceProperty("cachePrepStmts", true);
// hikariDS.addDataSourceProperty("prepStmtCacheSize", 250);
// hikariDS.addDataSourceProperty("prepStmtCacheSqlLimit", 2048);
// hikariDS.addDataSourceProperty("useServerPrepStmts", true);
// hikariDS.addDataSourceProperty("cacheResultSetMetadata", true);
return hikariDS;
}
ManagerStep:
#Bean
public Step managerStep() {
return stepBuilderFactory.get("managerStep")
.partitioner(workerStep().getName(), dateRangePartitioner())
.step(workerStep())
// .gridSize(52) // number of worker, which is not necessary with datepartition
.taskExecutor(new SimpleAsyncTaskExecutor())
.build();
}
WorkerStep: I also tried to increase the transaction properties timeout, but not luck
#Bean
public Step workerStep() {
DefaultTransactionAttribute attribute = new DefaultTransactionAttribute();
attribute.setPropagationBehavior(Propagation.REQUIRED.value());
attribute.setIsolationLevel(Isolation.DEFAULT.value());
// attribute.setTimeout(30);
attribute.setTimeout(1000000);
return stepBuilderFactory.get("workerStep")
.<Image, Image>chunk(10)
.reader(jdbcPagingReader(null))
.processor(new ImageItemProcessor())
.writer(imageConverter())
// .listener(wrkrStepExecutionListener)
.transactionAttribute(attribute)
.build();
}
Job builder:
#Bean
public Job mainJob() {
return jobBuilderFactory.get("mainJob")
// .incrementer(new RunIdIncrementer())
.start(managerStep())
// .listener()
.build();
}