I'm working on a project which for a long time will still be running alongside its predecessor. I need to import part of the old DB every 30 minutes.
In a testing environment, I used this solution:
run mysqldump from the source host (siteground.com) in a cronjob. Authorize the Siteground IP in the destination host.
In production I'm using a serverless AWS RDS instance, which cannot have public access.
What can I do?
Is there a way to pull the database tables in bulk (Siteground -> AWS) running the command from the AWS side?
Related
Is there any best practice to keep the staging database up to date with the production database?
For example,
every day at midnight, the production database overwrites the staging database.
If your goal is to make the Staging database an exact copy of the Production database, then you could:
Take a Snapshot of the Production database
Delete the Staging database
Restore a new Staging database from the Snapshot of the Production database
These steps can be automated via a script that calls the AWS Command-Line Interface (CLI). For example, it could use:
aws rds create-db-snapshot
aws rds delete-db-instance
aws rds restore-db-instance-from-db-snapshot
You can achieve the task as said by #John, but there are a few points which we can not know, such as
the status of the clone
notification once complete
Below official Blog will help with everything we need to know during every clone.
Blog: Orchestrating database refreshes for Amazon RDS and Amazon Aurora | AWS Database Blog
CloudFormation Git Repo: aws-samples
/
database-refresh-orchestrator-for-amazon-rds-and-amazon-aurora
[Optional] Migrate from RDS to Aurora
Migrating an RDS for MySQL snapshot to Aurora - Amazon Aurora
Is it possible to export an AWS Aurora Serverless MySQL database with mysqldump without using an EC2 instance?
At this time you cannot gain public access to Aurora, or use a site to site VPN connection to connect to it.
Whichever resource you use will need to reside within the VPC of the Aurora Serverless cluster. Possible solutions to bypass the requirement for EC2 would be either running a Fargate container to run your command or looking at Lambda to execute the command.
I finally decided to emulate mysqldump using Python and the AWS Data API.
To emulate mysqldump, I followed:
How can I dump a MySQL database without using mysqldump in Python
I would like to sync the local MySQL database to Amazon RDS MySQL database. I found a solution for EC2 to RDS but not for Local Database to RDS.
I built a database including 12 tables which all I want to get backup them to cloud periodically or automatically.
I do not want to run EC2 server since I need only MySQL database to get backup on cloud.
I need a solution like Microsoft Database Sync Agent. Whenever changes detected in Local Database, it should be synced to the cloud database. How can I make this happen?
You could use the AWS Database Migration Service:
AWS Database Migration Service (AWS DMS) is a cloud service that makes it easy to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the AWS Cloud, between on-premises instances (through an AWS Cloud setup), or between combinations of cloud and on-premises setups.
With AWS DMS, you can perform one-time migrations, and you can replicate ongoing changes to keep sources and targets in sync.
You can achieve this by following below steps.
Make a replica of local server to RDS.
Enable Query logging in local Database
Create a cron job which will process logging queries and it will execute queries on RDS instance in same order.
To generate a replica to RDS you can follow below steps.
You can't replicate your local database to RDS directly. Your need to dump you data and then after you can import it on RDS.
Instead of generating a dump file you can directly import data into RDS using below command.
mysqldump db_name | mysql -h 'other_hostname' db_name
You can find our more about this over here.
https://dev.mysql.com/doc/refman/5.7/en/copying-databases.html
Also first import tables & it's data then after import your triggers, routines & events. If you import together then there is a chances to get conflict and you job will be terminated.
Goal
I am trying to use MongoDB's BI Connector for Tableau, aka mongosqld. I have version 2.10, so here are the docs.
My long-term goal is to host mongosqld as a service on an AWS EC2 instance, and host MongoDB on AWS DocumentDB.
Background
A successful set of baby steps was:
Host MongoDB in a Docker container on my local machine via mongo image
Manually run mongosqld on my local machine, without a schema
Connect to it via mysql from my local machine
This works fine, I could see all of the databases via show databases;
My next set of steps was:
Host MongoDB in AWS DocumentDB
Host mongosqld on my EC2 instance at address 0.0.0.0:3307, without a schema
Enable TCP comms on port 3307 and 27017
Connect to it via mysql from my local machine
When I use mysql shell's show databases; command, I cannot see my databases, only information_schema and mysql.
Question
Given all of this information, does anyone here know what might have gone wrong? I am currently at a loss for what to try next.
I have a ruby on rails run on amazon ec2, database is mySQL.
Now I want to use Amazon RDS to backup this database in case the ec2 fails.
I read through the Amazon RDS user guide,it tells how to create a DB instance.
My question is:
1.what is the relation between created DB instance and my ec2 database?
2.When the DB instance "connect" to my ec2, what will happen?
The data used to send to ec2 mySQL server will be send to DB instance?
Is the database totally shifted to DB instance?So before I connect I should creare a same database in DB instance and tell ec2 send data to RDS ever since
3.If not,how the DB instance know the ec2 is down and takeover the data?
EDIT:
(Unsure)Is it that DB instance is just a place to put database.In order to use it,I need to set up database in DB instance,then connect to it by modifying database.yml file(this file tell rails server where the stored data goes).
How Is it possible to tell when the local database stops then at that time switch to DB instance?
There is absolutely no relationship between your EC2 MySQL DB and the RDS DB Instance. So what you are asking for is just not possible.
A better approach I would say is to have only RDS (No need to have MySQL running locally on EC2 instance). RDS takes snapshots regularly. Also, RDS is a managed service, so most of the MySQL administration tasks are handled by AWS and you don't have to worry. And you can rely on RDS snapshots from Backup perspective.