I'm looking for a solution to deploy HA MySQL cluster on kubernetes. The main feature must be automatic fail-over (being resilient to crashing master pod).
There is two main types of solutions.
Multi-Master Clusters like Percona XtraDB Cluster. These solutions has some limitations that will make some problems.
Single-Master Cluster with observer pod and a Proxy. The best-known solution I found is Orchestrator, and by now it is a dead project. I didn't found any other solutions that has been widely used.
I'll be so glad to know which solution can solve my problem.
I've tried Percona Operator for kubernetes and it has some limitations. I've also tested mysql operator for kubernetes and it has some bugs. Galera Cluster for MariaDB is not resilient to crashing more than half of MySQL instances.
Related
So I want to deploy a master-slaves MySQL cluster in k8s. I found 2 ways that seem popular:
The first one is to use statefulsets directly from k8s official document: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/
The second one is to use operator, i.e. https://github.com/oracle/mysql-operator
Which way is most commonly used?
Also, in statefulsets, if my MySQL master dies, will k8s automatically promote the slave to be the master?
Lastly, when my logic backend app performs an operation (CRUD) to MySQL cluster, how does k8s know which pod to route to, i.e. write operation can only be sent to master while read is sent to all?
Users can deploy and maintain a set of highly available MySQL services in k8s based on StatefulSets, the process is relatively complex. This process requires users to familiarize themselves with various k8s resource objects, learn many MySQL operation details and maintain a set of complex management scripts. Kubernetes Operators are designed to reduce the threshold for deploying complex applications on k8s.
Operator hides the orchestration details of complex applications and greatly reduces the threshold to use them in k8s. If you need to deploy other complex applications, we recommend that you use the Operator.
Speaking about master election while using StatefulSet.
Electing potential slave to be a master is not an automatic process - you have to configure this manually using Xtrabackup - here is more information - setting_up_replication.
Take a look: cloning-existing-data, starting-replication, mysql-statefulset-operator.
Useful tools: vitess for better MySQL networking management and percona-xtradb-cluster that provides superior performance, scalability and instrumentation.
I’m setting up a new k8s cluster, and I find the concept of BuildConfig and ImageStream quite interesting. But I do not have the incentive to buy Openshift support since the project context does not allow that.
so I was wondering if it is safe to use an OKD cluster in a production environment, and if there is an example of an entity that is already using it in production grade?
I was (still are) in the same situation with projects that could not afford to run AWS or GCE clouds, so we have deployed a 3 node single master, and later a 9 node HA cluster in our own data centre. The HA architecture was based on the reference implementation at http://uncontained.io/.
So yes it is certainly possible and thoroughly worth the effort. Our cluster is running Kafka, Spark, Neo4J, MongoDB, Jenkins and Cassandra and about 100 business application pods. The DevOps in Openshift (OKD) the biggest benefit.
The learning curve is steep though. I have invested enormous amounts of time in reading up on persistent storage (GlusterFS in our case), networking, cluster architecture, etc. It is very important to script the provisioning process in a rigorously repeatable manner. You are going to stand up and tear down the initial cluster close to 100 times before it plays through reliably.
I have been tasked with recommending the VM provisioning for an OpenShift production environment. The OpenShift installation documents don't really detail a lot of different options. I know that we want High Availability (which means multiple masters) but some of the things that I'm a bit confused by are:
separate hosts for etcd
infrastructure nodes
Do I need separate hosts/nodes for etcd? (advantages seem to be performance related but would like to better understand)
Do I need separate hosts/nodes for the infrastructure components (registry, router, etc.) or can these just be hosted on the master nodes?
AFAIK etcd can be on same host as master unless you really have a big cluster and want maintenance of etcd separate of openshift cluster.
Running routers on dedicated nodes help having high availability and reduce chances of nodes running into health issues due to other container work loads running on same machine. applications inside openshift cluster can run even if all masters go down (may be rare) but router nodes need to be available all the time for serving traffic.
There are many reference architectures published by redhat checkout blog.openshift.com and also redhat.com official docs
etcd and masters can be installed in the same node or separately. Here you can find some best practices for etcd. As you see, here is recommended that it is installed separately and this is what I would suggest if you can "afford" more servers. If not, co-locating masters and etcds we can say is symbiotic in that masters are CPU intensive whereas etcd uses a lot of disk IO and memory.
Regarding infrastructure deployments such as routers, docker-registry, EFK stack, metrics and so forth, the recommended deployment configuration (all within your possibilities) is that masters are not schedulable, and they worry only about serving the API and controlling the nodes. Then you can split your schedulable nodes into infrastructure and compute nodes.
Infrastructure nodes will only host applications used by the cluster itself or by other applications (i.e. Gitlab or Nexus)
Worker/Compute nodes will host business applications
Having a multi-master installation with HA routers is of course the best solution, but then you have to decide how you want to provide this HA, is it with an external LoadBalancer or with IP Failover?
As #debianmaster mentioned, there are several reference architecture documents you can read. Like this one here
Is it reasonable to use Kubernetes for a clustered database such as MySQL in production environment?
There are example configurations such as mysql galera example. However, most examples do not make use of persistent volumes. As far as I've understood persistent volumes must reside on some shared file system as defined here Kubernetes types of persistent volumes. A shared file system will not guarantee that the database files of the pod will be local to the machine hosting the pod. It will be accessed over network which is rather slow. Moreover, there are issues with MySQL and NFS, for example.
This might be acceptable for a test environment. However, what should I do in a production environment? Is it better to run the database cluster outside Kubernetes and run only application servers with Kubernetes?
The Kubernetes project introduced PetSets, a new pod management abstraction, intended to run stateful applications. It is an alpha feature at present (as of version 1.4) and moving rapidly. A list of the various issues as we move to beta are listed here. Quoting from the section on when to use petsets:
A PetSet ensures that a specified number of "pets" with unique identities are running at any given time. The identity of a Pet is comprised of:
a stable hostname, available in DNS
an ordinal index
stable storage: linked to the ordinal & hostname
In addition to the above, it can be coupled with several other features which help one deploy clustered stateful applications and manage them. Coupled with dynamic volume provisioning for example, it can be used to provision storage automatically.
There are several YAML configuration files available (such as the ones you referenced) using ReplicaSets and Deployments for MySQL and other databases which may be run in production and are probably being run that way as well. However, PetSets are expected to make it a lot easier to run these types of workloads, while supporting upgrades, maintenance, scaling and so on.
You can find some examples of distributed databases with petsets here.
The advantage of provisioning persistent volumes which are networked and non-local (such as GlusterFS) is realized at scale. However, for relatively small clusters, there is a proposal to allow for local storage persistent volumes in the future.
I'm looking to scale mysql on a swarm that could potentially be involve multiple servers. What is the best way to ensure that the data is in sync between the containers on the different servers?
I realise on a standard configuration without docker I'd have to set up replication. I'm wondering if there is a way to do it which is more suitable and easy to deploy with docker.
Docker-compose and docker swarm are great tool for scaling in docker environments. But Currently
MySQL-DB scaling is not possible in docker-compose or docker-swarm. Reasons:
Scaling fit for stateless containers
Configuration not possible for master-slave in docker-swarm
No replication method available in docker overlay network
May be in future we have such tech to enable RDBMS scaling.
This is what database replication is for.
https://dev.mysql.com/doc/refman/5.7/en/replication.html
You can try mariadb galera cluster, setup under docker, however you need other steps to provision and load balancer, and a node monitors the state of your containers (many works and it's not easy)
And, if you have multiple Nodes on docker swarm, your need to setup NFS server for docker to share files.
There is a tool called Cluster Control, free and paid version
https://severalnines.com/product/clustercontrol/docker-mysql-database-management