How can I copy the content of a specific table (or the table as is) from my local database to a database instance stored on a cloud, lets say Amazon RDS?
note: it has to be done ones every hour.
EDIT:
Other I/O operations on the local database should no be suspended (e.g. no READ LOCKS).
You can set your local database server to be a master to the Amazon RDS instance which means the Amazon RDS instance becomes a slave in this setup. This is possible to do as mentioned in the AWS documentation here http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Importing.External.Repl.html
You can also configure the slave to update only a specific table in the database after a specified interval of time.
Related
I have 2x RDS single instances (MySQL) in the same region.
The goal is to copy (A) RDS to (B) RDS on nightly basis.
Is there any way to have this configuration auto sync between 2 RDS instances?
If your goal is to copy the contents of one Amazon RDS database such that another RDS database contains identical content, then it is easiest to use a snapshot:
Create an Amazon RDS Snapshot of the source database
Launch a new Amazon RDS database instance from the Snapshot
Delete the previous Amazon RDS database 'B' database, since it is now out-of-date
The new database launched from the Snapshot will be identical to the source database (except for the instance DNS name).
This operation can be done via AWS CLI commands.
I would suggest looking into DMS ongoing replication as it seems like a potential solution for your use case https://docs.aws.amazon.com/dms/latest/userguide/CHAP_BestPractices.html#CHAP_BestPractices.OnGoingReplication
There are three approaches here.
One is that if you want to "refresh" the database schema on nightly basis, you can use logical backup by using mysqldump/mydumper to export/import the schema and put the script in the scheduler (eg. cron).
The second one you can use is snapshot backup that is provided by AWS, but you need to destroy the previous RDS instances, otherwise you will have many restored RDS from daily snapshot. Ideally, you can create some script that call AWS CLI to remove the previous RDS and restore the snapshot from daily backup. Put the script in the scheduler as well as first option.
A third option is that you can create synchronization using Amazon DMS (Data Migration Service) to migrate the data in a real-time fashion.
From my understanding, Aws RDS facilitate backup for the mysql database, but it is not cheap.
While using docker image for mysql may save us more in terms of cost? Because we only need to download the docker image for dockerhub and directly use it for free(e.g. create an instance and run the container).
Is there another reason of using RDS other than facilitating backup for the database?
I list several features of RDS which may warrant using it over self-managed MySQL docker container on an EC2 insistence or ECS:
RDS is managed service, so all OS updates, MySQL patches are managed by AWS and you don't have to worry about them.
RDS supports storage auto-scaling - you can start with small db, and RDS will extend storage automatically as needed.
Point-in-time recovery allowing you to "rewind" your recent db changes.
Read replicas - you can create up to 5 read replicas of your database to off-load read intensive applications from your primary db instance.
Cross-region read replica - you can have your replica in different region which is good for disaster recovery (entire AWS region goes down)
Automated and manual backups, including backups to a different region.
IAM authentication to your db instead of regular username/password.
Multi-AZ - RDS can keep a stand-by replica of your primary database instance in different availability zone, for quick recovery if it fails.
CloudWatch integrated db metrics and logs.
RDS event notifications allow you for straight-forward development of automations e.g. invoke lambda automatically for every backup, or if something fails.
Easier integration with other services, e.g. use of RDS Proxy in Lambda functions.
All these and other features of RDS make it much more expensive then hosting a self-managed MySQL docker container. But if MySQL in docker container meets all your requirements, then there is no need to use RDS. You can always start with the docker, and if your data and requirements grow, you can migrate to RDS.
I have a RDS Mysql database instance on AWS with 1000 tables (lets call it root instance).
I need to create another instance of this database with only the rows that match some foreign key id. This new instance must be in mirror with the root instance so I can query the new values as soon as they get inserted. Question: Is there any way to achieve this with AWS tools? Or do I need to code id?
As far as I know, I can create instances in a cluster to be mirrored with the root instance, but these instances are a full copy and I need only some rows.
Neither AWS nor MySQL provide a solution for what you describe.
You would have to develop your own solution. For example a CDC (change data capture) client (Debezium is a popular open-source CDC implementation) to parse the binary logs of your RDS instance, filter for the rows you want, and insert them to the other instance.
This isn't possible with RDS.
You can "fake it" by converting the tables you don't want replicated to Engine=Blackhole, however you have to edit your parameter-group and set "read-only" to 0, instead of the default "{TrueIfReplica}".
and to handle your case you can create a view that pulls only these records
Alternately, you would need to run your own slave server on EC2 with the RDS server as the master (this is possible if you're running MySQL 5.6 on RDS, but not 5.5 or below), however it's extremely complicated to set up.
I have 2 MySQL databases running on a server called X and Y, which both have identical content. A series of updates run throughout the day, which changes the content of X. At the end of the day, a process runs that compares the content of X with the content of Y (for various tables) in order to discover new rows, updated row data etc. Once the updates have been processed, mysqldump is used to dump X and then Y is overwritten with the dump. Both X and Y are now the same again, and the whole process repeats.
I'm investigating migration of these databases to Amazon RDS. What's the most efficient way to accomplish the process outlined above?
I understand that I can take a snapshot of a DB and restore it, but I think this is at the instance level only? That would mean I have to run 2 instances, which seems unnecessary. I don't have a problem running both databases on the same instance (I don't want to pay for more than one instance unnecessarily).
Do I just do what I'm doing now i.e. mysqldump X and restore it to Y, or is there some other method/shortcut that RDS provides?
Since the title is concerned AWS instance migration the best way is with my case (can be vary to others case)
Goto -> https://console.aws.amazon.com/rds
Select your DB Instance
Actions -> Take Snapshot
Goto -> https://console.aws.amazon.com/rds
Snapshots from left pane
select your snapshot just created
Action -> Restore Snapshot
After above steps you will be redirected to RDS instance creation page fill out required fields as per requirements and you are done with migration :D
Consider migrating to RDS Aurora for MySQL.
It supports native copy-on-write clones of the entire database (meaning server instance, not schema) without the need to make an actual "copy."
Copy-on-write means the "original" server and the "clone" share the same physical disk (called an Aurora Cluster Volume, which is replicates itself twice across 3 availability zones, using a 4/6 quorum), with both servers sharing the same disk blocks until one of them makes a change... which is when the copy action actually occurs ("on write"). So, you only use as much storage as is required to store your original working data set plus changes that occurred after cloning.
No server is the master in such a setup -- they all operate independently after cloning. I suspect that I'm not doing this innovation justice with my description -- it involves quite a bit of dark magic. See the write-up (with illustrations of copy-on-write):
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Managing.Clone.html
Aurora is compatible with MySQL 5.6. To be more precise, Aurora is MySQL 5.6, with MyISAM removed and InnoDB heavily rewritten to optimize performance and work with the replicated Aurora Cluster Volume storage technology.
A bit late in the day but I have just managed to do this by (1) creating a database back up to S3 and then (2) restoring the backup from S3, i.e.
a. Create database back up in S3
EXEC msdb.dbo.rds_backup_database #source_db_name = '<database-name-goes-here>'
,#s3_arn_to_backup_to = 'arn:aws:s3:::<bucket-name-goes-here>/<backup-filename-goes-here>.bak'
,#overwrite_S3_backup_file = 1;
b. Wait for the task to complete. You can execute the following SQL to check this
exec msdb.dbo.rds_task_status #db_name='<database-name-goes-here>';
c. When the lifecycle is "SUCCCESS" you can then restore from the S3 bucket using the following command
exec msdb.dbo.rds_restore_database #restore_db_name='<new-database-name-goes-here>'
,#s3_arn_to_restore_from='arn:aws:s3:::<bucket-name-goes-here>/<backup-filename-goes-here>.bak';
d. Again you can monitor the status of the restore with the following SQL command
exec msdb.dbo.rds_task_status #db_name='<database-name-goes-here>';
You could setup AWS MySQL RDS instance as a slave of an external master.
After loading a full dump to RDS, Call the stored procedure mysql.rds_set_external_master like this:
mysql> call mysql.rds_set_external_master ('10.10.3.2', 3306, 'replica', 'password', 'mysql-bin-changelog.122', 108433, 0);
Then start the replication by doing:
mysql> call mysql.rds_start_replication;
Once you have data in sync you can promote RDS to master by doing:
mysql> call mysql.rds_stop_replication;
mysql> call mysql.rds_reset_external_master;
By doing this either using your external X or Y servers, the AWS RDS behaves like a replica, the one you could use as your future master if required.
AWS now allows you to replicate data from an RDS instance to an external MySQL database.
However, according to the docs:
Replication to an instance of MySQL running external to Amazon RDS is only supported during the time it takes to export a database from a MySQL DB instance. The replication should be terminated when the data has been exported and applications can start accessing the external instance.
Is there a reason for this? Can I choose to ignore this if I want the replication to be persistent and permanent? Or does AWS enforce this somehow? If so, are there any work-arounds?
It doesn't look like Amazon explicitly states why they don't support ongoing replication other than the statement you quoted. In my experience, if AWS doesn't explicitly document a reason for why they do something then you're not likely to find out unless they decide to document it at a later time.
My guess would be that it has to do with the dynamic nature of Amazon instances and how they operate within RDS. RDS instances can have their IP address change suddenly without warning. We've encountered that on more than one occasion with the RDS instances that we run. According to the RDS Best Practices guide :
If your client application is caching the DNS data of your DB instances, set a TTL of less than 30 seconds. Because the underlying IP address of a DB instance can change after a failover, caching the DNS data for an extended time can lead to connection failures if your application tries to connect to an IP address that no longer is in service.
Given that RDS instances can and do change their IP address from time to time my guess is that they simply want to avoid the possibility of having to support people who set up external replication only to have it suddenly break if/when an RDS instance gets assigned a new IP address. Unless you set the replication user and any firewalls protecting your external mysql server to be pretty wide open then replication could suddenly stop if the RDS master reboots for any reason (maintenance, hardware failure, etc). From a security point of view, opening up your replication user and firewall port like that are not a good idea.