As I found, the best way to have zero down time even when one datacenter is down, is using kubernetes between at least two servers from two datacenters.
So because I wanted to use servers in Iran. I've heard low performance about infrastructure.
The question is that if I want to have master-master replication for mysql, in one server failure, how can I sync repaired server in kubernetes clustring?
K8s is the platform, it doesn't change how MySQL HA works. Example, if you have dedicated servers for MySQL, these servers become "pods" in K8s. What you need to do at MySQL level when any of the server is gone for whatever reason; is the same as what you need to do when you run it as a pod. In fact, K8s help you by automatically start a new pod. Where in former case, you will need to provision a new physical server - the time required is obvious here. You will normally run script to re-establish the HA, the same apply to K8s where you can run the recovery script as the init container before the actual MySQL server container is started.
Related
I am trying to deploy two MySQL pods with the same PVC, but I am getting CrashLoopBackoff state when I create the second pod with the error in logs: "innoDB check that you do not already have another mysqld process using the same innodb log files". How to resolve this error?
There are different options to solve high availability. If you are running kubernetes with an infrastructure that can provision the volume to different nodes (f.e. in the cloud) and your pod/node crashes, kubernetes will restart the database on a different node with the same volume. Aside from a short downtime you will have the database back up running in a relatively short time.
The volume will be mounted to a single running mysql pod to prevent data corruption from concurrent access. (This is what mysql notices in your scenario as well, since it is no designed for shared storage as HA solution)
If you need more you can use the built in replication of mysql to create a mysql 'cluster' which can be used even if one node/pod should fail. Each instance of the mysql cluster will have an individual volume in that case. Look at the kubernetes stateful set example for this scenario: https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/
I'm kind of lost there, I want to setup a common MASTER/SLAVE replication on a MariaDB database. I choose MASTER/SLAVE over MASTER/MASTER to avoid complexifying things. The SLAVE will be used only if the MASTER server is down.
I've setup MariaDB 10.0.x, but when I start reading on how to achieve this replication, they introduce Galera, which, if I understand correctly, replaces MariaDB.
What do you use to tell the SLAVE server to take the relay if the MASTER server is down ? Is it handled automatically via the Galera Cluster ?
If possible, I don't want my application to be aware of the slave server : I just want to configure it with the IP of MariaDB MASTER, and if it can't be reached, to use the SLAVE instead. (But I do not want to specify this fallback in the application level)
Thanks
What you are looking for can be achieved. I just completed a setup of MariaDB 10 using asynchronous replication (not Galera). To ensure maximum uptime I setup master / master replication and used mysql-mmm to monitor the setup. This tool will manage a virtual IP and point it at one of the two masters for writing purposes. This ensures consistent writes against a single master as to avoid corruption of the data. If one master fails the virtual IP will be mapped to the other master. This provides the high availability aspect. The instructions stated below were very clear and easy to follow.
http://mysql-mmm.org/mmm2:guide
Good luck!
I launched an Amazon EC2 with Amazon Linux and Amazon-EBS as root volume. I also started tomcat7 and mysql 5.5 on this EBS volume.
Later I decided to change from Amazon Linux to Ubuntu. To do that I need to launch another Amazon EC2 instance with a new EBS root volume. Now I want to copy tomcat7 and mysql from older EBS volume to new one. I have tables and data in mysql which I don't want to loose and an application running on tomcat. How to go about it?
A couple of thoughts and suggestions.
First, if you are going to be having any kind of significant load on your database, running it on EBS-backed volume is probably not a great idea as EBS-backed storage is incredibly slow relative to the machine's local/ephemeral storage (/mnt). Now obviously you don't want DB data on ephemeral storage, so there is really nothing you can do about it if you want to run MySQL on EC2. So my suggestion would be to utilize an RDS instance for your DB if your infrastructure requirements allow for it.
Second, if this is a production application, you are undoubtedly going to have some down time as you make this transition. The question is whether you need to absolutely minimize the amount of downtime. If so, then you need to have an idea as to the size of your database. Is it going to take a long time to dump/load? If not, you could probably just get your new instance up and running, and tested on an older copy of your database and then just dump and load the current database at the time of cutover.
If it is a large database then perhaps you can turn on MySQL binary logging. Then make a dump of the database at a known binary log position. Then install this dump on your new instance. Then when ready to cutover, you can replay the binary logs on the new instance to bring it current. Similarly, you could just set up the DB on the new instance as a replica until the cutover, at which point you make it the master.
You may even consider just using rsync to sync the physical database files if you don't want to mess with binary logging, though this can be a problematic approach if you are not that familiar with dealing with the actual physical database files.
As far as your application goes, that should be much simpler to migrate assuming it is just a collection of files. I would not copy the Tomcat7 installation itself, but rather just install Tomcat on Ubuntu and then adjust the configuration to match current.
As far as the cutover itself goes, this should be pretty straightforward and would vary in approach depending on whether you are using an elastic IP for your server or whether it is behind a load balancer,
We have a Rails 3.1.3 app, connecting to MySQL via the mysql2 gem. Standard config. We also have a handful of Resque workers performing background jobs. The DB hostname we point to (in database.yml) is actually a Virtual IP (VIP) that points to either node1 or node2.
Behind the scenes, the two MySQL servers (nodes) are setup in a High Availability configuration. The data folders are replicated via DRBD, with mysqld only running on the "active" node. When the cluster detects that node1 is not available, it starts mysqld on node2 and points the VIP to it.
If you want more details on the specific setup, it's very similar to this MySQL HA cookbook.
Here's the issue: When a failover happens, it takes approx 30-60 seconds to complete, during which there is no MySQL server available. Any Resque jobs that are currently running fail badly.
Here's the question(s): How can we tell ActiveRecord to reconnect after a delay? Maybe attempt several reconnects with a backoff timer? Or is there a better way of dealing with this?
Your HA setup is going to cause you infinite amounts of pain in the future. Use database-layer replication instead of block-device-layer replication; MySQL Proxy was designed to do this.
I'm trying to understand how to architect an Amazon Web Services application.
I have an instance running off of EBS. As far as I understand, I need to mount the EBS drive so that I can store my MySQL database on it.
When I later want to scale up, how do I do so? I understand that I can add more server instances, but how will they be accessing the database? Since from what I understand, the EBS volume can only be attached to one server instance.
I can't speak to this particular setup as I do not have experience using EBS with a MySQL instance but how this type of scaling is typically accomplished is by dedicating a particular instance as the master database server. Any time you spin up additional web servers those are still using the master DB IP to connect. At the time in which your database is the bottleneck you then spin up a slave DB instance on one of the boxes (or its own dedicated box). You can then configure replication in either a master to slave direction or a circular replication so that you can write to the slave instance as well.
If you choose the classic master to slave replication then you will have to make sure your writes are only performed on the master DB instance.
You can setup something like Zeus or any other connection load balancer so that you only ever have to connect to a single Database IP which will then round-robin route your read connections to your pool of servers. Otherwise you'd have to manage the connections yourself which is definitely not trivial. Good luck.
Growing Amazon EBS Volume sizes
You can give a try to MySQL clustering on your EBS backed instances. I have similar query, with more requirements, posted here.
EBS Volumes capacity can be scaled up using Snapshot->launch new volume technique, alternatively storage capacity can be scaled out using EBS Striping (RAID 0).
In AWS you cannot mount same EBS Volume to 2 EC2 instances simultaneously, so when you are scaling your application you need to scale out / up your MySQL DB either thru Replication or clustering. AWS RDS is a very good option for MySQL , if your application is read intensive then you can scale out using RDS Read replica's as well. If you need write scaling then functional partition or MySQL Shards can be explored.
AWS has an entire product dedicated to this: RDS.
In all but the rarest and most specialized of circumstances you're going to be better off using RDS than trying to create and tune your own EBS/EC2/MySQL infrastructure.
RDS also directly answers your question - they directly enable the creation of readonly databases to use as query slaves. RDS also performs backups, upgrades, and all sorts of fail-over infrastructure for you.
With EBS there's no way to attach a disk to multiple EC2 instances, so you're not going to be able to build out a failure cluster using that approach. Instead you're going to need replication or backup tools of some type.