I have 2x RDS single instances (MySQL) in the same region.
The goal is to copy (A) RDS to (B) RDS on nightly basis.
Is there any way to have this configuration auto sync between 2 RDS instances?
If your goal is to copy the contents of one Amazon RDS database such that another RDS database contains identical content, then it is easiest to use a snapshot:
Create an Amazon RDS Snapshot of the source database
Launch a new Amazon RDS database instance from the Snapshot
Delete the previous Amazon RDS database 'B' database, since it is now out-of-date
The new database launched from the Snapshot will be identical to the source database (except for the instance DNS name).
This operation can be done via AWS CLI commands.
I would suggest looking into DMS ongoing replication as it seems like a potential solution for your use case https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html#CHAP_BestPractices.OnGoingReplication
There are three approaches here.
One is that if you want to "refresh" the database schema on nightly basis, you can use logical backup by using mysqldump/mydumper to export/import the schema and put the script in the scheduler (eg. cron).
The second one you can use is snapshot backup that is provided by AWS, but you need to destroy the previous RDS instances, otherwise you will have many restored RDS from daily snapshot. Ideally, you can create some script that call AWS CLI to remove the previous RDS and restore the snapshot from daily backup. Put the script in the scheduler as well as first option.
A third option is that you can create synchronization using Amazon DMS (Data Migration Service) to migrate the data in a real-time fashion.
Related
From my understanding, Aws RDS facilitate backup for the mysql database, but it is not cheap.
While using docker image for mysql may save us more in terms of cost? Because we only need to download the docker image for dockerhub and directly use it for free(e.g. create an instance and run the container).
Is there another reason of using RDS other than facilitating backup for the database?
I list several features of RDS which may warrant using it over self-managed MySQL docker container on an EC2 insistence or ECS:
RDS is managed service, so all OS updates, MySQL patches are managed by AWS and you don't have to worry about them.
RDS supports storage auto-scaling - you can start with small db, and RDS will extend storage automatically as needed.
Point-in-time recovery allowing you to "rewind" your recent db changes.
Read replicas - you can create up to 5 read replicas of your database to off-load read intensive applications from your primary db instance.
Cross-region read replica - you can have your replica in different region which is good for disaster recovery (entire AWS region goes down)
Automated and manual backups, including backups to a different region.
IAM authentication to your db instead of regular username/password.
Multi-AZ - RDS can keep a stand-by replica of your primary database instance in different availability zone, for quick recovery if it fails.
CloudWatch integrated db metrics and logs.
RDS event notifications allow you for straight-forward development of automations e.g. invoke lambda automatically for every backup, or if something fails.
Easier integration with other services, e.g. use of RDS Proxy in Lambda functions.
All these and other features of RDS make it much more expensive then hosting a self-managed MySQL docker container. But if MySQL in docker container meets all your requirements, then there is no need to use RDS. You can always start with the docker, and if your data and requirements grow, you can migrate to RDS.
I have a production DB that is on RDS Aurora MySQL. I would like to create a "staging" version of it, so I need a complete duplicate/clone of the production version.
Most importantly I need the staging version to have write access to the new instance.
Is this possible?
Review Cloning Databases in an Aurora DB Cluster in the RDS User Guide.
Clones are not the same thing as replicas. A replica, in Aurora, has read-only access to the same data store allowing you to spread out your read workload across multiple instances... but a clone is a readable/writable moment-in-time fork of your original database. Any changes after the clone is created don't change the data on the original database instances (or on any other clones, and up to 15 independent clones are currently supported).
You can also create a new Aurora cluster from a snapshot of your production database, but a clone is probably the preferred solution for two reasons: it's faster to create a clone... but perhaps more importantly, clones use copy-on-write, so until you change the data on either the clone or the master it was cloned from, they share common storage space in the Aurora Cluster Volume that stores the data -- so you're only paying once for storage of the data that never gets changed. How this works is explained, with diagrams, in the RDS User Guide at the link, above.
You can take Backup ( Database snapshot ) on prod and restore the backup into new RDS Aurora servers ( during RDS Aurora instance creation) . It is simple GUI interface in AWS. You can change your permission after database has been restored into stage.
I need to create an RDS MySQL instance in us-east-1 and would like to replicate this data in another region (say eu-west-1). I know about read-replicas, and I will be using them also, but I wanted to have a back-up non-read-replica in another region.
Has anyone done this?
Is there code out there that does this replication?
I highly recommend switching from RDS MySQL to RDS Aurora, which is MySQL compatible. You won't have to change anything in your code, the database will still appear to be MySQL to any apps that connect to it. Among the many advantages Aurora has over MySQL is that it supports cross-region replication.
There are several ways to do this. I will cover two easy methods. RDS features are based upon engine (MySQL, Aurora, PostreSQL, etc.). Review as required.
1) Create a read-replica of your existing RDS instance in another region. Then promote that read-replica to master. Now you have a separate instance running in another region. You can then stop this instance to minimize billing (for up to seven days).
2) Create an RDS snapshot. Copy that snapshot to another region. When needed launch a new RDS instance from the snapshot.
We have an AWS Aurora database sitting on an instance that holds all of our production data. I want to be able to perform analytics on that data without doing it in our production environment, so I want to copy the production data on a daily basis to another AWS Aurora database on a completely different instance. Within that "analytics" database, I'll build out all the needed views and stored procedures to aggregate whatever transformed data I need to store.
At first I thought of creating an Aurora replica, but of course that's read-only. I need to find a way to do this outside of the production environment and I feel it's an easy enough task to do, but I just can't find out how to do it. Maybe I haven't been able to ask the write questions, so I came here. How can I achieve this?
This is simple AWS replication.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Replication.CrossRegion.html
Also if you prefer to use mysql or any other RDBMS use
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html
It is similar to master slave replication with little difference in sharded data mainted in Aurora.
Replication is the correct (subjective, of course) solution, but you can't use a managed Aurora replica, which is to say you can't use an Aurora replica in the cluster.
That does not, however, mean you can't create your own asynchronous Aurora replica... which would be a second Aurora cluster, an independent master that is writable, but that uses the replication stream (the binary logs, also called "binlogs,") from the master cluster to keep its data in sync.
The one caveat: you must be extremely cautious not to write to any of the tables on the asynchronous cluster that are being replicated from the production master. Do that, of course, and replication breaks. The master cluster will be completely unaffected, but the replica cluster will stop replicating once inconsistent data is detected. But you can create additonal tables, views, and stored programs without issue.
Within an Aurora cluster, there is no need for replication in the traditional sense -- the replicas use the same backing store as the master (the "cluster volume.") Here, we're just replicating from cluster to cluster, identical to the way two ordinary MySQL servers would replicate (in one direction, only, of course).
The setup is essentially identical to the setup for replicating in and out of Aurora, to or from MySQL. Since this solution uses MySQL native replication, the steps are the same.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Overview.Replication.MySQLReplication.html
How can I copy the content of a specific table (or the table as is) from my local database to a database instance stored on a cloud, lets say Amazon RDS?
note: it has to be done ones every hour.
EDIT:
Other I/O operations on the local database should no be suspended (e.g. no READ LOCKS).
You can set your local database server to be a master to the Amazon RDS instance which means the Amazon RDS instance becomes a slave in this setup. This is possible to do as mentioned in the AWS documentation here http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Importing.External.Repl.html
You can also configure the slave to update only a specific table in the database after a specified interval of time.