Error creating AWS MySQL RDS instance with Terraform - mysql

When creating an Amazon AWS RDS MySQL 5.7 db instance using Terraform "terraform-aws-modules/rds/aws" module, I started getting a strange error after > 1 hour. In other contexts in the past the same script worked (even more involved versions creating cross-region read replica in 2 other regions (3 in total).
When I tried to deploy to a different VPC recently I started getting an error after spending ~1 hour on the db options group resource (so not even reaching db deploy).
The error message is:
aws_db_option_group.this: Error creating DB Option Group: InternalFailure: An internal error has occurred. Please try your query again at a later time.
status code: 500 root.rds-virginia.db.db_option_group: eval: *terraform.EvalSequence
How to resolve or work around this?

Creating a dummy db option group (even though we don't need it for this use case) seems to work around this issue:
resource "aws_db_option_group" "some-option-group" {
name = "dummy-mysql-option-group"
option_group_description = "Dummy Mysql option group"
engine_name = "mysql"
major_engine_version = "5.7"
}
Terraform db option group docs: https://www.terraform.io/docs/providers/aws/r/db_option_group.html

Related

MySQL 'max_connections_per_hour' error AWS RDS

I'm getting the following error for a DB I have.
“1226 (42000): User ‘?????????’ has exceeded the ‘max_connections_per_hour’ resource (current value: 85)”
I did create a simple script that opens and closes the database connection to try and troubleshoot, which I was able to replicate.
What can I do to increase max connections per hour?
The DB is managed through AWS RDS. Most of the solutions that I read about include upgrading the instance.
85 connections in an hour doesn't seem like much so there has to be another solution.

All Metabase operations returning "Field 'id' doesn't have a default value"

We're moving our Metabase v0.24.1 deployment from AWS RDS MySQL 5.7 to AWS Aurora MySQL Serverless (5.6 compatible). When we launch Metabase every query the the application tries to run returns will some form of this error:
Field 'id' doesn't have a default value
Full Error:
03-17 16:07:57 [1mERROR metabase.middleware[0m :: [31mPOST /api/database 500 (68 ms) (0 DB calls)[0m
{:message "Field 'id' doesn't have a default value",
:stacktrace
["api.database$fn__29894$fn__29897.invoke(database.clj:233)"
"api.common.internal$do_with_caught_api_exceptions.invokeStatic(internal.clj:229)"
"api.common.internal$do_with_caught_api_exceptions.invoke(internal.clj:224)"
"api.database$fn__29894.invokeStatic(database.clj:215)"
"api.database$fn__29894.invoke(database.clj:215)"
"middleware$enforce_authentication$fn__38784.invoke(middleware.clj:120)"
"api.routes$fn__38908.invokeStatic(routes.clj:58)"
"api.routes$fn__38908.invoke(routes.clj:58)"
"routes$fn__39540$fn__39541.doInvoke(routes.clj:64)"
"routes$fn__39540.invokeStatic(routes.clj:60)"
"routes$fn__39540.invoke(routes.clj:60)"
"middleware$log_api_call$fn__38883$fn__38885.invoke(middleware.clj:329)"
"middleware$log_api_call$fn__38883.invoke(middleware.clj:328)"
"middleware$add_security_headers$fn__38833.invoke(middleware.clj:243)"
"middleware$bind_current_user$fn__38788.invoke(middleware.clj:140)"
"middleware$maybe_set_site_url$fn__38837.invoke(middleware.clj:266)"],
:sql-exception-chain ["SQLException:" "Message: Field 'id' doesn't have a default value" "SQLState: HY000" "Error Code: 1364"]}
Attempts to add a new database also return the same error through the Metabase UI.
We have validated that the RDS db parameters are identical between the database we are migrating to and from excluding the engine default configurations that may change between Aurora Serverless and MySQL 5.7 defaults.
Another notable change is that we're moving from normal ECS tasks running on EC2 instances that we own over to Fargate tasks, but the task definitions are identical between the two.
Edit: I've found another configuration inconsistency between Aurora Serverless and MySQL 5.7 in the sql_mode parameter. In 5.7, it is set to NO_ENGINE_SUBSTITUTION by default and in serverless it is set to 0.
After updating sql_mode on my serverless deployment to NO_ENGINE_SUBSTITUTION I've found that the actual configuration remains inconsistent in the database itself. SELECT ##sql_mode; returns NO_ENGINE_SUBSTITUTION on the 5.7 deployment and it returns STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION on the serverless deployment even though the parameter group is not set to this value.
Edit 2: I failed to mention the method that we are using to transfer the data to the new database and that is AWS DMS version 3.3.1. It appears that DMS does not replicate the auto_increment table attribute which needs to be present on the id columns in the Metabase database. This could likely be the root issue.
The root cause of this issue was that the id columns in the Metabase DB did not have the auto_increment attribute and were being treated as if they did not have values even though they did.
This was because, AWS DMS, which we used to transfer the data, does not replicate all details of the schema:
https://dba.stackexchange.com/questions/138946/aws-dms-creates-tables-with-no-auto-increment
The fix to this issue was to perform a schema migration first using a standard mysqldump --no-data and then we populated the data with DMS.

AWSCLI: Can't specify db parameter group when creating mysql read replica

Using awscli, I'm trying to create a cross-region read replica, in us-west-1, of a mysql RDS in us-east-1. The db must have the lower_case_table_names parameter set to 1(default is 0). I have created a custom db parameter group with this setting. When I call "aws rds create-db-instance-read-replica" and specify my custom parameter group with "--db-parameter-group-name", the command fails with the following error:
An error occurred (InvalidParameterCombination) when calling the CreateDBInstanceReadReplica operation: A parameter group can't be specified during Read Replica creation for the following DB engine: mysql
AWS documentation makes no mention of this limitation(that I can find). Obviously, in this case, changing the parameter group after the replicant is created is not an option. Has anyone else encountered this, and is there a work-around?
Edit: Wound up just letting the replica come up with default parameters. Even though that caused the replication to fail and left status at "error", once the replica was available I switched it to my custom parameter group. Then I rebooted it, and it came right up and replicated without issue. May not work in every case, but seems to have worked in mine.

AWS RDS Read Replica on a different VPC

I am trying to create an rds mysql read replica in a different vpc in the same region. This doesn't seem to work. I am getting the below error.
I am able to create a cross region read replica, here obviously the vpcs are different. But it works there and not within the same region.
Any idea why this could be the case?
The DB instance and EC2 security group are in different VPCs. The DB instance is in vpc-b40d62d3 and the EC2 security group is in vpc-3f6cc45b (Service: AmazonRDS; Status Code: 400; Error Code: InvalidParameterCombination; Request ID: 56d7eb7c-8cd7-490a-b979-ef678f4f6ed7)
This was asked earlier and the answer was not supported and use a custom solution. Asking again as that was a 3 year old post and cloud moves fast. :)
Cheers.
Tried this a week ago in RDS console but didn't work. I got the same error. But It is supported. At least in aws-cli version 1.16.
aws rds create-db-instance-read-replica \
--db-instance-identifier [yourmaindb] \
--source-db-instance-identifier [arn resource url of the source] \
--db-subnet-group-name [subnet in a different VPC] \
--vpc-security-group-ids [security group in a different VPC]

aws DMS replicate-changes-only error

I have prod aws Aurora DB and I want
to replicate changes to test mysql DB (schema is same - Aurora is based on mysql)
I am using aws DMS for this.
When performing full replication for certain tables the replication works fine,
When I want to perform replicate-changes-only, the replication fails.
I've set binlog_checksum=NONE and binlog_format=ROW in the parameter group.
The error I am receiving while running is:
Last Error The task stopped abnormally Stop Reason RECOVERABLE_ERROR Error Level RECOVERABLE
Last Error Task 'task-id' was suspended due to 6 successive unexpected failures Stop Reason FATAL_ERROR Error Level FATAL
Loading a snapshot to the test db isn't an option.
I just want to replicate the changes between specific tables.
Thanks in advance.
I am having the same error, it was always stopping 10min after starting. Adding more verbose logs didn't show more information but by changing the task configuration, especially the parameter MaxFullLoadSubTasks.
By default the value is "MaxFullLoadSubTasks": 8,, I changed it to "MaxFullLoadSubTasks": 1,.
It is slower but it's working for now. You may be able to increase it a bit to be quicker without having the same error.
You can modify the task configuration by first copying the task json settings you will find under DMS > TASK > overview, then changing the value and saving it to a file and then:
aws dms modify-replication-task --replication-task-arn <TASK_ARN_ID> --replication-task-settings file:///path/to/your/task_config.json