I'm launching a big project of mine which took me more than a year.
if this work as I want, this would need a solid database architecture, but I'm a developer and kind of noob for database configuration.
Application is made with Spring/Mysql served with tomcat 8.
So after some research i've concluded this :
I will use mysql cluser with InnoDB engine
To start I need 3 server minimum for mysql clustering
To handle this correctly I'll use ProxySQL to route correctly queries on right node
Left some question:
How does replication works for migrations, if I want to add a column or a table does this replicates on all of nodes ?
I want to take 3 more server on cloud to do this, how much ram is the best for this kind of use ?
Is there better options ?
Related
We have 6 Servers (4 Applications servers and 2 DB Servers)
We are using HAProxy to load balance between the Application and API servers (2/2)
Now the issue I'm having is that the system administrator setup a Master/Slave on the MySQL but it's always failing and until now we cannot use the slave since most data are always corrupted and we always need to fix it and each time we are getting different errors .
We tried to make some sort of load balancing for the read/write (write on master , read on slave) but we were not able to use that since slave data are not always correct .
What I'm wondering is how the big guys proceed when dealing with high load servers where you always need the data to be accurate and cannot take any risk?
Can someone tell me his own experience and what he used ?
What i found : Percona XtraDB Cluster , but before going into this direction need input ...
Thank's !!
You can choose MySQL/MariaDB+ Percona + HAproxy. This combination support Master- Master synchronization and Data sync work really well. The most of the Real-time Data synchronization has Issue with primary and foreign Key. You can avoid those issue too using Percona. Go ahead and Good Luck
The "table is full" error means your slave doesn't have enough space to perform the ALTER TABLE. You need to get larger disks to resolve that error.
But the subtext is that no one is monitoring your database servers, and that's a bigger problem. You need to get a database administrator, or else get a professional service to do it.
What I'm wondering is how the big guys proceed when dealing with high load servers where you always need the data to be accurate and cannot take any risk?
First, get it out of your head that any system has no risk. That's impossible, if you plan to use the system at all. You can't eliminate the possibility of errors, but you can be prepared to recover from them seamlessly.
The big guys do the following:
Hire operations staff including system administrators, network administrators, database administrators to take care of the servers.
Monitor everything. Use software to track system load, disk space, errors, and many other things continuously. The best option is New Relic. For MySQL slave integrity, use a tool like pt-table-checksum.
Redundancy. Create standby systems and data to take over when (not if) the primary system fails.
You probably want to learn about the field of high availability architecture. Check out this talk: Scalable Internet Architectures
Get on amazon ec2. You can launch 4 app server along with 2 db servers on the fly and set up load balancing using aws engineering features.
http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-getting-started.html#define-load-balancer
https://aws.amazon.com/articles/1639
I have linux machine ( ubuntu 14.04 32GB RAM 8 core ...)
I want to run on this machine several Slaves ( currently 5 slave replications but I will need more )
I use master-slave mysql replication
From my point of view there are 2 options to do that
1. use mysqld_multi - set 5 instance ( done that in the past with 2 instance)
2. use Docker container - each one with mysql slave
what is the Best solution ?
which one will be more easy to maintain ( and add ) ?
Tnx for help
The issues I'm trying to solve are:
I have problems with Performance and with the architecture that we use I’m unable to use cluster - so I want to use load balancing and split the read/write
On one of my machines I need to split data from 1 master to different slaves by some column value - and i want all slaves will be on same machine
I would suggest reading the book High Perfromance MySQL.
If you're having performance problems but your server has enough resources to add multiple slaves then you should tweak the configuration of your master MySql instance to better utilize those resources. If you're attempting to split the writes to slaves those changes will never be propagated back to the master. If you want to utilize slaves to increase read performance then you can do that, but I would only suggest this if you've maxed out the box that the master instance is running on.
I would need more information on why you think you need to do this: 'On one of my machines I need to split data from 1 master to different slaves by some column value' to be able to comment about it. At the surface it feels like this is a bad idea, but there could be a reason for it.
Openshift offers scalability. However, it seems to me, if you are using MySql, in the end MySql queries/hits will be the bottleneck (if you are lucky enough to generate traffic which need scalability and considering the max-connections limit on openshift).
Suppose I want to use OpenShift, is it possible to create multiple mysql cartridges to balance the load and create dynamic environmental variables to assign requests to different mySql cartridges? (suppose I send an id or something and the environment variable for the mysql is set to "dbname+lastdigit" of this id").
This is a simplified example which should multiply the database capacity by ten (if this is unrelated data). Can it be done?
I hope some openshift guy or girl will clarify this for me....
cheers
Edit: Thanks mbaird for your info:
To clarify:
I wasn't talking about auto-scaling but using for instance 11 static/persistant db-cartridges which would never scale up or down.
Then you could store user information in any of them depending on (also for instance) their last id-digit.
The 11th database cartride could be used as a table to get the user's id and then redirect that user to the right database (if last digit = 0, db = db0, if last digit = 1, db = db1 etc). This would enable me to call the right database for the right user.
Of course, this is not auto-scaling but it would multiply the database capacity by (roughly) ten.
However, this would require the ability to create multiple mysql cartridges and corresponding environmental variables to gain access to all these mysql cartridges.
It seems to me this not possible right now, so I will investigate your suggestions.
The OpenShift database tier currently doesn't scale. Further, even if you could add a second MySQL cartridge it wouldn't give you a scalable database, it would give you a new, empty database. What you are looking for is the ability to scale the MySQL cartridge across multiple gears, not adding another cartridge.
I've actually seen some comments from OpenShift (although I can't seem to find them now) that the databases on OpenShift are for development only and you should look for another service to host your database if you have a mission-critical application that requires database fail-over and scalability.
Since you are specifically using MySQL, I would look into using Amazon RDS (either MySQL or the new Aurora engine which is MySQL compatible) or ScaleDB.
I'm running MySQL servers load balanced via HAProxy. But I need to perform some action whenever a node (MySQL) goes down or comes up, like we do in PgPool-II.
Example:
When ever MySQL-1 goes down, I also want to shutdown my WebServer-1. (STONITH like)
In PgPool-II for PostgreSQL, I can do this. I'm unaware, if there are similar tools for MySQL.
Is this possible in HAProxy itself:
Say, when my DB-1 goes down, route the web server traffic, to WebServer-2.
MySQL has quite an extensive support for replication, high availability and sharding. Your question is not very clear to me but here are a few things you could try reading:
For failover, MySQL provides a python script called mysqlfailover which is shipped as part of MySQL utilities.
For sharding and high availability you could try out MySQL Fabric. MySQL Fabric is a relatively new product so expect it to not provide all kinds of fancy sophisticated sharding schemes in theory but it is stable enough for you to trust it with what it provides.
By the way both of them are open source so you could tweak it to suit your need!
I'm relatively new to MySQL replication. In a nutshell I have a MySQL 5.1 server instance on 3 Ubuntu Lucid Lynx servers.
ATM I have Server A (MASTER) which replicates a single database to Server C (SLAVE).
This has been running for a couple months without a problem.
I now want to have Server B (as MASTER) replicate a different single database to Server C (SLAVE).
I was looking into implementing this, but my initial reading seems to indicate that a replication slave server can't have two masters.
My question is, is this the case even when only single or select database are being replicated?
Keep in mind I do not wish to replicate the same database from different masters. I simply wish to replicate multiple separate database on a single server, from separate masters.
You can set a different master per database in MySQL.
The book high performance MySQL has a full chapter on this issue. I recommend getting hold of a copy and using the info in that.
http://oreilly.com/catalog/9780596101718/
You can read a copy of the relevant chapter online here:
http://oreilly.com/catalog/hpmysql/chapter/ch07.pdf
This is from the first edition, the second edition is more up to date, but chapter 8, which deals with replication is not freely available online.
UPDATE
The solution I'm talking about is only in the 2nd edition of the book, see this answer to a simular question: Is it possible to do N-master => 1-slave replication with MySQL?
In general you can't do this. You can't replicate from many-to-one (you can from one-to-many).
Also you can't really just replicate one database - cross-database updates in statement-based mode then become unreplicatable, which means your slave will fail (or become out of sync) as soon as someone does one on the master.
The standard solution is to install multiple mysql instances, which is far from ideal, but works.