Spring Boot SQL JPA not using correct replica - mysql

Question

I have an application where I am trying to distribute reads & writes between two replicas. For some reason JPA is only using my read-replica, not the write replica. The write replica is the primary replica. The result is that when I use JPA to try and write data I get and 'UPDATE command denied' error because it is using the read only datasource. I have tried doing my own annotation and using the #Transactional annotation. Both annotations are called via AOP with the correct datasource but JPA will not use it.
FYI Spring JDBC works correctly via the custom annotation. This is strictly a JPA issue. Below is some code:
My AOP class:
#Aspect
#Order(20)
#Component
public class RouteDataSourceInterceptor {
#Around("#annotation(com.kenect.db.common.annotations.UseDataSource) && execution(* *(..))")
public Object proceed(ProceedingJoinPoint pjp) throws Throwable {
try {
MethodSignature signature = (MethodSignature) pjp.getSignature();
Method method = signature.getMethod();
UseDataSource annotation = method.getAnnotation(UseDataSource.class);
RoutingDataSource.setDataSourceName(annotation.value());
return pjp.proceed();
} finally {
RoutingDataSource.resetDataSource();
}
}
#Around("#annotation(transactional)")
public Object proceed(ProceedingJoinPoint proceedingJoinPoint, Transactional transactional) throws Throwable {
try {
if (transactional.readOnly()) {
RoutingDataSource.setDataSourceName(SQL_READ_REPLICA);
Klogger.info("Routing database call to the read replica");
} else {
RoutingDataSource.setDataSourceName(SQL_MASTER_REPLICA);
Klogger.info("Routing database call to the primary replica");
}
return proceedingJoinPoint.proceed();
} finally {
RoutingDataSource.resetDataSource();
}
}
}
My RoutingDataSource class:
public class RoutingDataSource extends AbstractRoutingDataSource {
private static final ThreadLocal<String> currentDataSourceName = new ThreadLocal<>();
public static synchronized void setDataSourceName(String name) {
currentDataSourceName.set(name);
}
public static synchronized void resetDataSource() {
currentDataSourceName.remove();
}
#Override
protected Object determineCurrentLookupKey() {
return currentDataSourceName.get();
}
}
AbstractDynamicDataSourceConfig
public abstract class AbstractDynamicDataSourceConfig {
private final ConfigurableEnvironment environment;
public AbstractDynamicDataSourceConfig(ConfigurableEnvironment environment) {
this.environment = environment;
}
protected DataSource getRoutingDataSource() {
Map<String, String> props = DBConfigurationUtils.getAllPropertiesStartingWith("spring.datasource", environment);
List<String> dataSourceNames = DBConfigurationUtils.getDataSourceNames(props.keySet());
RoutingDataSource routingDataSource = new RoutingDataSource();
Map<Object, Object> dataSources = new HashMap<>();
DataSource masterDataSource = null;
for (String name : dataSourceNames) {
DataSource dataSource = getDataSource("spring.datasource." + name);
dataSources.put(name, dataSource);
if (masterDataSource == null && name.toLowerCase().contains("master")) {
masterDataSource = dataSource;
}
}
if (dataSources.isEmpty()) {
throw new KenectInvalidParameterException("No datasources found.");
}
routingDataSource.setTargetDataSources(dataSources);
if (masterDataSource == null) {
masterDataSource = (DataSource) dataSources.get(dataSourceNames.get(0));
}
routingDataSource.setDefaultTargetDataSource(masterDataSource);
return routingDataSource;
}
protected DataSource getDataSource(String prefix) {
HikariConfig hikariConfig = new HikariConfig();
hikariConfig.setJdbcUrl(environment.getProperty(prefix + ".jdbcUrl"));
hikariConfig.setUsername(environment.getProperty(prefix + ".username"));
hikariConfig.setPassword(environment.getProperty(prefix + ".password"));
return new HikariDataSource(hikariConfig);
}
}
application.yaml
spring:
datasource:
master:
jdbcUrl: jdbc:mysql://my-main-replica
username: some-user
password: some-password
read-replica:
jdbcUrl: jdbc:mysql://my-read-replica
username: another-user
password: another-password
If I use the annotation on with JDBC template then it works as expected:
THIS WORKS:
// Uses main replica as it is not specified
public Message insertMessage(Message message) {
String sql = "INSERT INTO message(" +
" `conversationId`," +
" `body`)" +
" VALUE (" +
" :conversationId," +
" :body" +
")";
MapSqlParameterSource parameters = new MapSqlParameterSource();
parameters.addValue("conversationId", message.getConversationId());
parameters.addValue("body", message.getBody());
namedJdbcTemplate.update(sql, parameters);
}
// Uses read replica
#UseDataSource(SQL_READ_REPLICA)
public List<Message> getMessage(long id) {
MapSqlParameterSource parameters = new MapSqlParameterSource();
parameters.addValue("id", id);
String sql = "SELECT " +
" conversationId," +
" body" +
" FROM message"
" WHERE id = :id";
return namedJdbcTemplate.query(sql, parameters, new BeanPropertyRowMapper<>(Message.class));
}
If I use a JPA interface it always uses the read replica:
THIS FAILS:
#Repository
public interface MessageJpaRepository extends JpaRepository<MessageEntity, Long> {
// Should use the main-replica but always uses the read-replica
#Modifying
#Query(value =
"UPDATE clarioMessage SET" +
" body = :body" +
" WHERE id = :id" +
" AND organizationId = :organizationId",
nativeQuery = true)
#Transactional
int updateMessageBodyByIdAndOrganizationId(#Param("body") String body, #Param("id")long id, #Param("organizationId")long organizationId);
}
So I am just getting the error below when I try to use the main-replica. I have tried using the #UseDataSource annotation and AOP does actually intercept it. But, it still uses the read-replica.
java.sql.SQLSyntaxErrorException: UPDATE command denied to user 'read-replica-user'#'read replica IP' for table 'message'
What am I missing?

Answer 1

When you use #UseDataSource, it is working so it seems rules out any issues with implementation of aspect.
And When you #Transactional, it uses the secondary replica, regardless of your your AOP being invoked. My suspicion is by the TransactionInterceptor created by spring is invoked before your RouteDataSourceInterceptor. You can try the following:
Put a breakpoint in your aop method as well as a break point in org.springframework.transaction.interceptor.TransactionInterceptor.invoke method to see which one invokes first. You want your interceptor invoked first
If your interceptor is not invoked first, I would modify your interceptor to have high order as follows.
#Aspect
#Order(Ordered.HIGHEST_PRECEDENCE)
#Component
public class RouteDataSourceInterceptor {
I still don't understand how you are telling TransactionInterceptor to choose the DataSource you set in RouteDataSourceInterceptor. I have not used multi tenant setup but recently I came across a question which I helped to solve and I can see it is implementing AbstractDataSourceBasedMultiTenantConnectionProviderImpl. So I hope you have something similar. Not able to switch database after defining Spring AOP