Recreate replica when master is recreated - mysql

Question

I have a QA setup with a master and a replica. Both are AWS RDS MySQL. It's provisioned with Terraform and the gist is like this
data "aws_db_snapshot" "latest_prod_snapshot" {
db_instance_identifier = var.production_instance_id
most_recent = true
}
resource "aws_db_instance" "qa_master" {
apply_immediately = true
snapshot_identifier = data.aws_db_snapshot.latest_prod_snapshot.id
availability_zone = var.qa_master_zone
instance_class = var.master_instance_class
identifier = var.master_name
parameter_group_name = var.parameter_group_name
auto_minor_version_upgrade = false
multi_az = false
performance_insights_enabled = true
performance_insights_retention_period = 7
vpc_security_group_ids = [var.security_group_id]
option_group_name = var.option_group_name
backup_retention_period = 5
skip_final_snapshot = true
enabled_cloudwatch_logs_exports = ["audit", "error", "slowquery"]
}
resource "aws_db_instance" "qa_replica" {
apply_immediately = true
replicate_source_db = aws_db_instance.qa_master.id
availability_zone = var.qa_replica_zone
instance_class = var.replica_instance_class
identifier = var.replica_name
parameter_group_name = var.parameter_group_name
auto_minor_version_upgrade = false
multi_az = false
performance_insights_enabled = true
performance_insights_retention_period = 7
vpc_security_group_ids = [var.security_group_id]
skip_final_snapshot = true
enabled_cloudwatch_logs_exports = ["audit", "error", "slowquery"]
}
When I want to update it from a new snapshot, the master is always marked for recreation and replica for "change in place". But replication stops working after the update. Is there a workaround for that? Am I doing something weird here? Can I somehow force replica to recreate too?
So far I have been doing terraform destroy before doing the terraform apply.

Answer 1

To trigger the replica being recreated you will need to have a ForceNew parameter on the replica resource change when the non replica changes.
With RDS there is a resource_id attribute that isn't normally surfaced in places people care about (you don't specify it, you don't use it to connect to it and it's not shown in the RDS console) but is different for each database instance (compared to the normal identifier which is specified).
This then brings us to the other part which is that that attribute needs to be in a ForceNew parameter on the replica resource. The obvious choices here are either the identifier or identifier_prefix parameters. The biggest impact here is that these are also used to identify the database instance from others but also used as part of the DNS address to connect to the database in the form of ${identifier}.${random_hash_of_account_and_region}.${region}.rds.amazonaws.com. So if you're needing to connect to the instance then you'll either need to have the client discover the replica address as it will now contain the randomly generated resource_id of the non replica instance or you'll need to have Terraform create a DNS record that either CNAMEs or aliases the RDS address.
So in your case you might want something like this:
data "aws_db_snapshot" "latest_prod_snapshot" {
db_instance_identifier = var.production_instance_id
most_recent = true
}
resource "aws_db_instance" "qa_master" {
apply_immediately = true
snapshot_identifier = data.aws_db_snapshot.latest_prod_snapshot.id
availability_zone = var.qa_master_zone
instance_class = var.master_instance_class
identifier = var.master_name
parameter_group_name = var.parameter_group_name
auto_minor_version_upgrade = false
multi_az = false
performance_insights_enabled = true
performance_insights_retention_period = 7
vpc_security_group_ids = [var.security_group_id]
option_group_name = var.option_group_name
backup_retention_period = 5
skip_final_snapshot = true
enabled_cloudwatch_logs_exports = ["audit", "error", "slowquery"]
}
resource "aws_db_instance" "qa_replica" {
apply_immediately = true
replicate_source_db = aws_db_instance.qa_master.id
availability_zone = var.qa_replica_zone
instance_class = var.replica_instance_class
identifier = "${var.replica_name}-${aws_db_instance.qa_master.resource_id}"
parameter_group_name = var.parameter_group_name
auto_minor_version_upgrade = false
multi_az = false
performance_insights_enabled = true
performance_insights_retention_period = 7
vpc_security_group_ids = [var.security_group_id]
skip_final_snapshot = true
enabled_cloudwatch_logs_exports = ["audit", "error", "slowquery"]
}
resource "aws_route53_zone" "example" {
name = "example.com"
}
resource "aws_route53_record" "replica_instance" {
zone_id = data.aws_route53_zone.example.zone_id
name = "qa-replica-database.${data.aws_route53_zone.example.name}"
type = "CNAME"
ttl = 60
records = [aws_db_instance.replica_qa.address]
}
Now, if the production snapshot changes then the qa_master database instance resource will be recreated which will lead to the qa_replica database instance resource also being recreated and then the Route53 record for the qa_replica instance will be updated with the new address, allowing you to always connect to the replica at qa-replica-database.example.com.