Currently we have Redis master and Redis slave containers. MySQL master and MySQL slave containers. Both replicating.
How would we handle a failure on one of the master containers? Should I be using something like Nginx as a forward proxy to detect connection failures?
Already we do this on our API servers and Web servers.
For the replication of MySQL I suggest configuring MySQL in a master <-> master approach and setup an HAProxy load balancer over them, as eugeneware does in https://github.com/eugeneware/docker-mysql-replication. It is very easy to set up using an HAProxy Docker container.
For Redis it definitely looks like you need Sentinel: http://redis.io/topics/sentinel. In https://hub.docker.com/r/joshula/redis-sentinel/ you can find a docker image for Sentinel.
I don't think using a proxy like Nginx is an appropriate solution for both problems.
Related
As I found, the best way to have zero down time even when one datacenter is down, is using kubernetes between at least two servers from two datacenters.
So because I wanted to use servers in Iran. I've heard low performance about infrastructure.
The question is that if I want to have master-master replication for mysql, in one server failure, how can I sync repaired server in kubernetes clustring?
K8s is the platform, it doesn't change how MySQL HA works. Example, if you have dedicated servers for MySQL, these servers become "pods" in K8s. What you need to do at MySQL level when any of the server is gone for whatever reason; is the same as what you need to do when you run it as a pod. In fact, K8s help you by automatically start a new pod. Where in former case, you will need to provision a new physical server - the time required is obvious here. You will normally run script to re-establish the HA, the same apply to K8s where you can run the recovery script as the init container before the actual MySQL server container is started.
I currently have a working MySQL master and slave with asynchronous replication set up on a Kubernetes cluster.
I'm trying to plan for a contingency if the master goes down, the slave will pick up the slack and what not. I figured the first step would be to review the Dockerfiles and scripts that define the ENTRYPOINT within the Dockerfiles.
When I kubectl get svc, this is the information I get about the services.
This tells me on the cluster, the master has an IP 10.0.156.209.
Now when I flicked through the helper script docker-entrypoint.sh for the MySQL slave Docker image, I noticed this line which helps set up the master-slave scenario (I trimmed the line just to highlight this)
and when I boot into the slave pod on Kubernetes, the environment variable $MYSQL_MASTER_SERVICE_HOST is set to 10.0.156.209
My question: How did the MySQL slave pod know to use the master's cluster IP as the value for $MYSQL_MASTER_SERVICE_HOST? Is this a Kubernetes thing or a SQL thing?
Thank you :)
These environment variables are created automatically by Kubernetes for each Service. See: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#environment-variables
When a Pod is run on a Node, the kubelet adds a set of environment variables for each active Service.
On another note, mysql on Kubernetes is a complex topic. You probably don't want to roll your own solution. Have you looked at Vitess?
I need some advice on setting up HAProxy for MySQL master-slave replication cluster: I want all connections i.e. all reads and writes to go to the master and only if the Master fails should HAProxy forward the connections to the slave.
The HAProxy documentation isn't very clear on which balancing algorithm to use for this scenario. The way I figure it, using static-rr with very high weights for the master and the lowest weights for the slave is the closest I can get to configuring the scenario described above. But there is still a small chance that the slave gets the reads/writes even though the master is still up and healthy?
Which is the correct way to setup HAProxy for this strict master/slave setup? Is HAProxy the right tool to use or is there a better way to achieve this?
Use MaxScale from MariaDB. It is a real DB router. Easy to use and easy to configure. Is direct for MySQL/MariaDB or Galera Cluster.
If you have more question we can chat
There's a similar question on stackoverflow with a description of how to use a custom health check to force HAProxy to send all traffic to the master, falling back to the read-only slaves if it fails.
I am struggling to setup replica with Amazon RDS. I want to make RDS as my master but for slave I want to setup my server outside of RDS. For now I am setting my slave at EC2. I have checked AWS doc for the information but could not find the exact answer to my question.
However I did found a blog post which have POC for doing it http://www.ruempler.eu/2013/07/07/replicating-aws-rds-mysql-databases-to-external-slaves/
But the problem that I facing is I am getting error
ERROR: Got error reading packet from server: Slave can not handle replication events with the checksum that master is configured to log
I tried to set binlog_checksum = none at RDS but RDS is not letting me change its config file.
Please suggest me folks!!
You can use AWS DMS (Data migration service) for this purpose. It will support replication between any 2 MySQL DBs.
Please refer our blog:
https://medium.com/tensult/cross-account-and-cross-region-rds-mysql-db-replication-part-1-55d307c7ae65
The question is quite old, but I bumped into it while searching for similar functionality. Amazon supports both replication from an external master and replicating to an external slave these days:
Replication from external master:
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Importing.External.Repl.html
Replication to external slave:
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MySQL.Procedural.Exporting.NonRDSRepl.html
Hope this helps future searchers.
It does not appear to be possible as of now (December, 2017).
Looking at #walter-heck's second link above, it is actually (another) explanation of "... replication between an external master instance and a DB instance on Amazon RDS".
It does not appear you can setup an external replica which connects to a RDS running as master.
AWS does offer this service, of course, however each replica must be within RDS.
With all the service offerings AWS has, one would think this would be possible. However, it may not work well with the AWS business model, because you don't pay based on external replicas - you pay based on RDS size.
I'm trying to understand how to architect an Amazon Web Services application.
I have an instance running off of EBS. As far as I understand, I need to mount the EBS drive so that I can store my MySQL database on it.
When I later want to scale up, how do I do so? I understand that I can add more server instances, but how will they be accessing the database? Since from what I understand, the EBS volume can only be attached to one server instance.
I can't speak to this particular setup as I do not have experience using EBS with a MySQL instance but how this type of scaling is typically accomplished is by dedicating a particular instance as the master database server. Any time you spin up additional web servers those are still using the master DB IP to connect. At the time in which your database is the bottleneck you then spin up a slave DB instance on one of the boxes (or its own dedicated box). You can then configure replication in either a master to slave direction or a circular replication so that you can write to the slave instance as well.
If you choose the classic master to slave replication then you will have to make sure your writes are only performed on the master DB instance.
You can setup something like Zeus or any other connection load balancer so that you only ever have to connect to a single Database IP which will then round-robin route your read connections to your pool of servers. Otherwise you'd have to manage the connections yourself which is definitely not trivial. Good luck.
Growing Amazon EBS Volume sizes
You can give a try to MySQL clustering on your EBS backed instances. I have similar query, with more requirements, posted here.
EBS Volumes capacity can be scaled up using Snapshot->launch new volume technique, alternatively storage capacity can be scaled out using EBS Striping (RAID 0).
In AWS you cannot mount same EBS Volume to 2 EC2 instances simultaneously, so when you are scaling your application you need to scale out / up your MySQL DB either thru Replication or clustering. AWS RDS is a very good option for MySQL , if your application is read intensive then you can scale out using RDS Read replica's as well. If you need write scaling then functional partition or MySQL Shards can be explored.
AWS has an entire product dedicated to this: RDS.
In all but the rarest and most specialized of circumstances you're going to be better off using RDS than trying to create and tune your own EBS/EC2/MySQL infrastructure.
RDS also directly answers your question - they directly enable the creation of readonly databases to use as query slaves. RDS also performs backups, upgrades, and all sorts of fail-over infrastructure for you.
With EBS there's no way to attach a disk to multiple EC2 instances, so you're not going to be able to build out a failure cluster using that approach. Instead you're going to need replication or backup tools of some type.