I have been working on a project in Mysql Master slave replication. I want to setup a master slave replication between AWS and GCP where AWS has the AWS RDS as the master and the slave or replica is in the GCP side. But I want to create this replica on GCP side without publicly exposing the master instance on AWS. That means this should happen in a private network.
I have found solutions where we can create proxy for the master instance and then create replica on the GCP side using the Cloud SQL migration services. But this is not what what I want. I don't want to assign a proxy to the master instance.
The replica creation process should be within a private network.
What should I do next? Help.
Also, please do let me know if the question is still unclear.
Create a Transit Gateway between AWS VPC and GCP private network.
https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html
If private network on the master (AWS) is a must, then this won't be possible. The documentation about using Cloud SQL as External replica is clear on the requirements for the source:
Ensure that the source database server meets these configuration requirements:
An externally accessible IPv4 address and TCP port.
Related
I see a lot of articles online where EC2 is involved, but since my backend is essentially serverless I have not found much information how to access my RDS once it is turned private. Can anyone point me in the right direction?
Current state:
Public MySQL RDS
RDS is accessed by a MySQL client on my local machine (MySQL Workbench) and AWS Lambda functions via my web application (both connecting via SSL)
Future state:
Private MySQL RDS
Private RDS would continue to be accessed by only my local machine and only the noted AWS Lambda functions via my web application (I assume continuing to use SSL?)
In your scenario your Lambda functions will need to be configured to run in the VPC if they are not already. That is the only change required for the Lambda functions.
However, When you switch the RDS instance to private, that means it only accepts connections from within the VPC. So you can't make connections directly from your local computer to the database anymore. You have to go through some sort of "bridge" to get your local computer into the AWS VPC network.
In this scenario people either use an EC2 instance as a bastion host, or they create a VPN connection from their local computer into the AWS VPC. AWS Client VPN is a managed service you could used for this.
You'll need to evaluate the Client VPN pricing, but I think you may find that a single t4g.nano EC2 bastion host is probably cheaper, and you can also stop the instance when you don't need it to really cut down the cost.
You can use VPC also with Lambda. Lambda and RDS can be in the same VPC, or in separate VPC’s peered together. Aws documentation for this scenario can be found here: https://aws.amazon.com/premiumsupport/knowledge-center/connect-lambda-to-an-rds-instance/
You can use Basti, which is a CLI tool that manages the EC2 bastion instance for you. The tool keeps the instance stopped when it's not used to minimize the solution cost, performs software updates to maintain the bastion instance secure, and provides a convenient CLI that can be used locally and in CI/CD pipelines.
I have an application running on EC2 instances that store data in an RDS instance. All of these instances are in an AWS VPC with security groups configured to allow them to connect to each other.
For reporting purposes, I would like to connect to the RDS instance from my laptop (e.g. using SQLAlchemy) to run simple queries. Every time I try to connect using the connection string that the EC2 apps use, the connection times out.
For Google, one can use the Cloud SQL proxy for this, but I can't find an analogous product for AWS. Instead, it seems like what I am supposed to do is attach an internet gateway to the VPC and configure the security groups to allow connection from my machine. However, the documents are unclear on how to do this other than allowing all inbound connections or allowing a static IP. Unfortunately my laptop doesn't have a static IP, and I'm uncomfortable allowing all inbound connections as it seems insecure and an invitation to attacks. I also have not been able to find a way to configure a security group to allow connections based on IAM credentials for example using the AWS CLI. Since I will be routinely generating reports, a solution that involves updating a security group (i.e. allowing my current IP) every time I want to connect seems suboptimal.
I have tried following the following documents, but so far have had no success in finding a solution that does not allow all connections:
Allow users to connect to RDS using IAM*
Connecting to RDS instance from command line
Connecting to RDS on VPC from internet
*My RDS instance configuration does not allow me to enable IAM authentication, I'm not sure why
IAM Database Authentication is not supported for the configuration in the DB Instance db.
Modify your Db Instance to another instance class and try again.
(Service: AmazonRDS; Status Code: 400; Error Code: InvalidParameterCombination;
Request ID: a6194fb8-2ab9-4a6a-a2be-63835e6e0184)
Is there something I'm not understanding or overlooking? Is allowing connections from all IPs not a big deal since the DB instance is still secured by DB user credentials?
Select this connection as per screenshot. Then fill up all details use your nat instance .pem file to connect. Its like you are connecting to VPC through Nat gateway or Internet gateway bypassing this.
Another option to install VPN on VPC and connect.
I would like to create a kubernetes cluster to deploy mysql databases, like a mysql farm. These databases should be accessible from internet.
All databases on the same node will have the port 3306 listening, the kube-proxy or the DNS addon could redirect each request to an specific container?
I would like to create url's like myDB1.example.com:3306, myDB2.example.com:3306 that goes to an specific container.
I'm deploying this environment in AWS.
It's possible to create this cluster?
Yes. Starting point would be a (customized) MySQL Docker image with EBS backed volumes and you'd be using it in an Replication Controller to handle failover. On top of that you would have a Service that provides a stable and routable interface to the outside world. Optionally, put an AWS Elastic Load Balancer in front of it.
We have a local mysql server(not on public domain) and want that server replicated to a google cloud sql instance that we have. My question is that:
1. is this possible?
2. our local server is accessible on the local network only. no public IP, etc. Although the server has internet. Im not sure how replication works, is this a one way traffic from master to slave?
if this is the case, then this might be possible even if the local mysql server is not accessible on the public domain since it only need to connect to the slave up on the cloud.
hope this makes sense
MySQL replication requires a connection from the slave to the master. The easiest way to make this possible is to set up a DMZ or port forwarding so your SQL instance is accessible over the internet.
Google Cloud SQL does not support replication from off-site to Cloud SQL, but you can do it with MySQL on GCE.
If you do use MySQL on GCE you have the added option of using Cloud VPN to allow a connection from your GCE instance to your local MySQL instance without requiring public exposure of either.
I have a MySQL database on Amazon RDS. When I created this database I unselected the "public access" option which can not be changed after creating the database. This means that my database instance can only be accessed from inside the VPC.
So now I would like to access the database in my local computer with setting the MySQL host with a EC2 VPS I have inside the network with access to the database. I want this server to act as my MySQL server so I can access it locally.
I just had to do this same thing. The process is to set up an SSH tunnel through the EC2 instance to the database. I wrote a post about the whole process that should be helpful
There's a couple of options -
Take a snapshot of the database and spin up a new copy that does allow public access. You can then use the security groups to only allow access from your ip - that way you have the benefit of the non public access security plus the ease of access from your machine.
If you don't want to do that, as datasage mentions your other option is to use an ssh tunnel - this will mean creating an ec2 instance in the same vpc that CAN access the rds, then using putty or your favourite ssh client to tunnel traffic through the 'bastion' ec2 instance to your database. This has the added layer of security but it's also more work to manage, depending on your familiarity with ssh. Not to mention the added cost of the ec2 instance.