I've AWS MySQL RDS instance located at Singapore, and I'm planning to migrate it to AWS Aurora (Mumbai). This process roughly takes about 30-40 minutes. I don't want to take any downtime while migration process. My App server created on node.js and also hosted on EC2 at Singapore. How Do I do HA (High Availability) of my database? What should I need to do? Should I make any changes in my server logic on node.js? or make any replication between 2 databases (MySql & Aurora)? or I need to create load balancer to achieve his?
In sort While migration process, I don't want any downtime and my database serves as usual to my users/client.
I have done both migrations Singapore to Mumbai and MySQL to Aurora. I will recommend to take one step at a time.. Looks like your website is important as you are so particular about zero downtime.. So, take measured risks.
We never know what fails during these big engineering release. Worst part is that MySQL to Aurora is a "No coming back" release as there is no migration from "Aurora to MySQL".
I will recommend that for Aurora to Mysql go through serious testing; because I did faced few issues like opt file missing, tmp database not supported in Aurora.
For Singapore to Mumbai; make sure your application uses an internal dns name like abc.internal which is CNAME to actual endpoint. Best way with minimal downtime is to create replica in Mumbai --> make master as readOnly --> promote replica to master --> now change abc.internal to new endpoint.
Aurora to Mysql there will be downtime depending on your db size. Please remember that some downtime is ok; but you cannot afford to loose any data.
Related
I have aws rds aurora database for the production environment. And I have to build a database for the development environment.
I tried it by using aws database migration service(DMS), snapshot, mysqldump.
First, DMS didn't support migrating Auto_Increment column and Indexes. But I need them.
Second, Snapshot overwrite production database's user data(MySQL user data - using when connecting to MySQL) to the development database. And I want to maintain them differently.
Last, mysqldump is so slow and I concerned about mysqldump may cause down performance to production database.
So, I'm looking for the other way.
The below things are what I want:
Every information(w/o MySQL user data) such as Auto_Increment, Indexes are must be migrated.
Development environment database is must be sync to production database(reset and re-migrate) every day automatically.
Migration process as faster and lower downing performance as possible is the best.
Does anyone know how to build it?
Using an AWS DMS task, you can specify what schema to migrate and the type of migration.
The page should look similar to the following
Amazon RDS automatically creates a primary
DB Instance and synchronously replicates the data to a standby instance in a different
Availability Zone (AZ). Each AZ runs on its own physically distinct, independent
infrastructure, and is engineered to be highly reliable.
Yes you can. Migrations are among the most time-consuming tasks handled by the database
administrators (DBAs). Although the task becomes easier with the advent of
managed migration services such as the AWS Database Migration Service (AWS
DMS), many large-scale database migrations still require a custom approach
due to performance, manageability, and compatibility requirements.
Extra
*Amazon RDS provides high availability and failover support for DB instances using Multi-AZ- deployments.
*ElastiCache improves the performance of your database through caching query results
What better way to do a quick migration from RDS MySQL 5.7 from São Paulo/Brazil to RDS Aurora in Northern Virginia, from a large database (probably more than 25GB of dump).
But I can not leave the database stopped for more than 3 hours (
or probably less), because this database is production of a company.
Thank you very much in advance.
In the region of São Paulo no have MySQL Aurora (One of the reasons to do the migration, in addition to the costs being twice as much compared to Northern Virginia)
RDS may be publicly accessible only during migration if necessary.
I will not be able to use Multi-AZ. Would it be feasible to use "AWS Database Migration Service"?
I will also have to migrate the instances EC2 and S3 linked to this database, mainly EC2 to avoid latency problems.
After the migration will be stopped all services in the region of São Paulo.
The main reasons as I said before is the reduction of costs in the long and short term (will be considered the use of reserved instances) and also performance, and instances EC2 to avoid problems of latency and instability.
You are making a mistake trying to move the database and change the engine from MySQL to Aurora at the same time.
Migrate the MySQL 5.7 system now, and convert to Aurora later. You do not need to ask for trouble, and doing both at the same time is exactly that.
It is not possible to "quickly" migrate a primary database over distance, but it is possible to make the amount of setup time irrelevant, and activation time near zero.
Instead of trying to do a copy, create an RDS cross-region replica of your data, and at the last moment, promote that replica to master.
Creating a Read Replica in a Different AWS Region
With Amazon RDS, you can create a MariaDB, MySQL, or PostgreSQL Read Replica in a different AWS Region than the source DB instance. You create a Read Replica to do the following:
Improve your disaster recovery capabilities.
Scale read operations into an AWS Region closer to your users.
Make it easier to migrate from a data center in one AWS Region to a data center in another AWS Region.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.XRgn
It doesn't matter how long it takes for RDS to copy the data and set up the replica, because as soon as it is copied, it starts replicating everything that changed on the master server since the process began.
Once you have verified that everything is correct and consistent, then you promote a replica. It is permanently and irrevocably detached from its original upstream instance, and becomes writable. This is the last thing you do and after the application starts writing to this new database, your original system in São Paulo is obsolete because changes to it will no longer replicate to the new system -- they're permanently isolated.
This arrangement does not require you to establish any networking or make the databases publicly accessible.
And, you can create and destroy multiple replicas to test this process, without disturbing production.
I have an application database running with MySQL engine on Amazon RDS. For better availability of our data for users in all parts of the world I'm looking for the best solution.
In the previous version of the application, we mirrored our database in US and Singapore, so that users got a better performance in terms of speed and on our side, we had backup if any disaster occurred.
Now as we moved to Amazon, will having Multi A-Z Deployment serve us in the same way? I mean replicates the database in all regions but will RDS still work in a single region only?
I have done some studies but still not sure so please ask me any further questions if I'm being puzzling.
Thank you.
I think you need both the Multi-AZ and the Read Replica features of AWS RDS.
Multi-AZ just creates a non-accessible secondary DB in another availability zone and in case the primary fails, AWS would switch over to the secondary DB. So you have failover.
In the case you want to increase the performance, and your application can work in read-only mode in Singapore (for example), the Read Replica would be perfect. If writes are also required, you would need to route them to the primary read-write database.
AWS supports a combination of the two approaches.
RDS MySQL Multi-AZ deployments currently works only inside an Amazon EC2 region. RDS Read replica's also need to be present inside the same Amazon EC2 region. Inter region replication is the most requested feature in AWS RDS to replicate data to another RDS in Alternative Amazon EC2 region. It is in their road map currently.
I've been playing with AWS EC2 and really like it. There is one drawback though, the instance could disappear due to hardware failure or whatever reason. This happened to me in my first week of operation. I was wondering whether there are good solutions to backup a MySQL database so that I don't lose my customer credentials?
You can transfer mysql database directly from EC2 machine to S3bucket but you will consume more cost for bandwidth and storage. You go for a third party application (which is safe) to backup your mysql or any plugins. Because they compress your data & encrypt and then save in S3 storage. Also, you can enable snap shot and take snap shots for volumes (hard drives)
I suggest you to use 'StoreGrid' backup software to backup your mysql database in EC2 machine. check this following link to know more about Online Backup Service on Amazon EC2/S3 http://storegrid.vembu.com/online-backup/amazon-ec2-s3-cloud-online-backup.php
Check this following link to configure MySQL database BACKUP http://storegrid.vembu.com/online-backup/mysql-backup.php?ct=1
Note: You have mentioned Hardware failure occurs often ! --- you can backup entire hard drives too using the above software.
I hope, now your MySQL data base is backed up from EC2 instance and stored in S3 storage safely.
Cheers !
Amazon now offers Relational Database Storage, that is, pre-configured EC2 instances, without any OS access to host MySQL (or Oracle, or T-SQL for real) for you, but aim to solve much of the availability, reliability and durability issues one faces when trying to host transactional data store yourself on a bare EC2 instance.
http://aws.amazon.com/rds/
"automated backups, DB snapshots, automatic host replacement, and Multi-AZ deployments"
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I realize a couple of basic differences between the two, i.e.
EC2 is going to be cheaper
RDS I wouldn't have to do maintenance
Other than those two, are there any advantages to running my database from RDS as opposed to a separate EC2 server acting as a MySQL server. Assuming similar instance sizes, are both going to run into the same limitations in terms of being able to handle a load?
To give you a little bit more info about my use, I've got a database, nothing too big or anything (biggest table 1 million rows), just high SELECT volume.
This is a simple question with a very complicated answer!
In short: EC2 will provide maximum performance if you go with a RAID0 EBS. Doing RAID0 EBS requires a pretty significant amount of maintenance overhead, for example:
http://alestic.com/2009/06/ec2-ebs-raid
http://alestic.com/2009/09/ec2-consistent-snapshot
EC2 without RAID0 EBS will provide crappy I/O performance, thus it's not even really an option.
RDS will provide very good (though not maximum) performance out of the box. The management console is fantastic and it's easy to upgrade instances. High availability and read only slaves are a click away. It's REALLY awesome.
Short answer: Go with RDS. Still on the fence? Go with RDS!!! if you enjoy headaches and tuning every last little bit for maximum performance, then you can consider EC2 + EBS RAID 0. Vanilla EC2 is a terrible option for MySQL hosting.
In this post there is an excellent benchmark between:
Running MySql on a Small EC2 + EBS
Running MySql on a Small EC2 + EBS + adjusted MySql parameters
A Small RDS
The benchmark is very good since it is not focused only in ideal conditions (only one thread) but also in more realistic scenarios, with 50 threads hitting the database.
RDS is not really a high availability system. Read the fine print in the RDS faq. During a failover event it can take up to 3 minutes to failover. Additional amazon will decide it needs to "upgrade" your rds instance and do a failover at that point which will take your database down for "up to 3 minutes" (our experience is that it can take a longer than that).
RDS high availability is very different than master - master or master - slave replication and is much slower. They don't use mysql replication but uses some kind of ebs replication. So in a failover situation it will mount the ebs on the backup machine, start mysql, wait for mysql to do failure recover (hopefully nothing got corrupted too bad), then do a dns switch.
I hope this helps you with you evaluation.
We chose to use EC2 MySQL instances because we have a high read volume and need master-slave replication. Of course, you can spin up multiple RDS instances and setup MySQL replication between them yourself, but we use Scalr.net, which manages that for you using EC2 instances.
Basically, we just tell Scalr how many MySQL instances we want at it keeps them up, automates the setup of replication, handles automatic failover of slave promotion to master if the master gets terminated etc. It does both SQL dump backups and EBS volume snapshots of the master. So, when it needs to create a new slave, it automatically temporarily mounts an EBS volume of the last master snapshot to initialize the slave DB, then starts replication from the appropriate point. All point and click :)
(and no, I don't work for Scalr or anything. Scalr is available as Open Source if you don't want to use their service)
Regarding the maintenance window question. If you use Multi-AZ then RDS will create a standby replica in another availability zone so that there's no down time for maintenance and you protect yourself against a zone failure.
That's what I'm planning to do in the next week or so. Of course it's going to cost you more but I haven't worked that bit out yet.
MySQL on EC2 vs RDS MySQL
Advantages of MySQL on EC2
Amazon EC2 Inter Region Replication
Copy Snapshots across Amazon EC2 regions
RAID 0 with EBS Striping in MySQL EC2
More than 3TB of Disk space ( You will not need this for your size) can be attached on MySQL on EC2.
Disadvantages of MySQL on EC2
Configuration, Monitoring and Maintenance compared to RDS
Point in time backups available in RDS
IOPS lesser than RDS MySQL ( even after RAID 0) currently, 10800 with 6 disks for MySQL on EC2 whereas 12500 IOPS 16KB on RDS MySQL
I have been trying out RDS for a few months and here are some issues I have:
Using SQL profiler is tricky. Since you cannot connect profiler directly to the server, you have to run some stored procedures to create a log file that you can analyze. While they offer some suggestions about how that is done, it is far from user friendly. I would only recommend that you have a certified SQL professional do this kind of work.
while Amazon backs up your instance, you cannot restore an individual database. I have a web app with several separate customer-specific databases and my solution was to launch an EC2 instance with SQL running on it to attach to the production RDB database and import the data and then back it up on the EC2 instance. The other solution was to use a 3rd party tool that creates a massive SQL script (on the app server) that will recreate the schema and populate the data back to a restore point.
I had the same question this weekend. There is a 4 hour downtime window per week for RDS where they do maintenance. RDS seemed more expensive if you can get away with a micro instance of EC2. (This is true of test instances which has minimum traffic) I also wasn't able to change the timezone of the RDS instance because I dont have permission.
I am now actually looking at http://xeround.com/ which is mysql on EC2 by another company. They do not use InnoDB, instead they have their own engine called IDG. I am just starting to investigate that but they are in BETA and will give 500MB of space.