We use mysql multi-source replication, analysts and developers works with the databases, it's a main task of server(cross-bases queries, etc). Mysql slave replicates data from about 10-15 servers, some of them is realy big (400gb, 500gb, 1.5tb). Slave host - ec2 r4.2xlarge. But have some problems, main of them - stability. Often there is an errors 1236, 1594. We fix one channel - another fails, sometimes it's critical.
Backups of most master databases are performed through ebs snapshot, where datadir located. But here another problem of mysql multi-source - work with backups, unlike multi instance replication, I can not just change datadir, where backup located.
I was trying to find a solution that at least partially solved the problems of stability and work with snapshots, but I found absolutely nothing.
Did I understand correctly that there is no support for multi-source replication in the aws-RDS?
Maybe there is a similar solution, or there is another approach to solving the problem (FEDERATED is bad variant). Thanks for any help and advices.
As Michael said, you shouldn't be encountering these errors. Multi source replication has quite stable now. I personally been using it for more than 2 years now. Yes I have faced some of issues like "slave trying to access binlogs that master has purged", but most of them were easy to sort out. I would recommend you to look at these replication topics:
Errant transactions
GTID hole : this was there because of bug
max allowed packet error
We fix one channel - another fails, sometimes it's critical.
make sure you're not touching GTID set for other channels.
Did I understand correctly that there is no support for multi-source
replication in the aws-RDS?
The feature still not available in any of AWS cloud services. https://forums.aws.amazon.com/thread.jspa?messageID=781416&tstart=0
Multi source replication is not supported in RDS , you can use AWS DMS for same purpose .
Related
I've just setup MariaDB on Ubuntu VMs with Master-Master replication using a combination of these three guides:
http://www.chriscouture.com/setting-up-replication-on-mariadb/
https://www.server-world.info/en/note?os=Ubuntu_16.04&p=mariadb&f=3
https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication
In short, both are masters and both are slaves to the other one. This will be used by separate web hosting servers. One will be "primary master" which all web servers connect to, and in case of a failure on that server they will be manually changed to the "secondary master".
The reason I went for this instead of Master-Slave is
The 3rd party backup job is simpler. Both backups and restores are done on the "secondary" master, instead of backup from the secondary and restoring to the primary.
If the primary fails and I have to switch to the secondary for a while, all changes made during that time should be written to the primary when it comes online again (?).
It's not live yet but everything works fine so far when I'm testing it. However I'm no DBA. I've been reading about how it's not "worth it" and if you don't absolutely need the Master-Master you should go with Master-Slave.
Now I'm afraid I will run into problems like inconsistencies and other headaches.
A maximum of about 100 databases for mostly Wordpress installations will be hosted.
What are your thoughts on the matter? M-M or M-S?
If M-S, how do I "sync back" data changed if the Master goes down and the Slave need to be the active server for a period of time?
I'm also wondering if there's a standard, generally accepted, step-by-step way to upgrade these. I'm thinking when a OS or MariaDB version upgrade is needed.
My general thoughts are something like take down one server, upgrade it, start it. Then take down the other, upgrade it, start it.
But I'm guessing there's more to it than that.
I only found some documentation on M-S upgrading with Mysql.
I am looking at creating a high availability Wordpress installation across two servers with a load balancer to distribute the traffic between these servers, I intend to setup a LAMP stack on these two web servers and configure MySQL master master replication to ensure both servers have the same content similar to the setup at https://www.linode.com/docs/websites/cms/high-availability-wordpress
For anyone created a similar setup before, is there anything to watch out for? Will this work without there being differences in the database assuming both nodes are online? If one node goes down will this synchronise with the other master node successfully?
This is a pretty old post, but I've set up exactly this recently.
I do not recommend setting up WordPress with master-master replication. The nodes went out of sync multiple times and resulted in split-brain. We've reverted back to master-slave replication because of the instability of the setup.
I recently tried this and regret it. I realize your post is now almost 2 years old, but thought I should put in my 2 cents for anyone considering this approach. I recommend you go with MariaDB with MaxScale to separate the reads and writes, and go with master-slave configuration. I'm running into all sorts of deadlocks, and issues with this configuration. Currently using MariaDB with Galera, with HAProxy up front.
Using simple replication settings with one MASTER and one SLAVE, how can one ensure that the SLAVE and MASTER are fully synchronized?
Now yes, they both started from the exact same image and replication is working and reporting that everything is okay BUT:
* It has happened that there were errors stopping the replication and then the replication had to be stopped and later resumed.
* Perhaps a change accidentally occurred on the SLAVE and then it's not the same as the MASTER anymore.
* Other whichever scenarios that might break sync.
While it's possible to do a big mysqldump of both database and compare the files I would be interested in a method that can be implemented more easily and also can be checked automatically to ensure all is in sync.
Thanks
Have you tried Percona Toolkit (formerly known as Maatkit)? You can use one of their tools which is pt-table-checksum for your case. You can check other tools too at their website.
pt-table-checksum performs an online replication consistency check by
executing checksum queries on the master, which produces different
results on replicas that are inconsistent with the master. The
optional DSN specifies the master host. The tool’s exit status is
nonzero if any differences are found, or if any warnings or errors
occur.
The following command will connect to the replication master on
localhost, checksum every table, and report the results on every
detected replica:
If you have MySQL Server versions 5.6.14 or higher, you can use the MySQL Replication Synchronization Checker. It's included in the MySql server package. Is designed to work exclusively for servers that support global transaction identifiers (GTIDs) and have gtid_mode=ON.
This utility permits you to check replication servers for synchronization. It checks data consistency between a master and slaves or between two slaves. The utility reports missing objects as well as missing data.
The utility can operate on an active replication topology, applying a synchronization process to check the data. Those servers where replication is not active can still be checked but the synchronization process will be skipped. In that case, it is up to the user to manually synchronize the servers.
See MySQL Documentation for more information
You are right to be suspicious of a seemingly healthy master/slave replication setup! We were running fine when suddenly we got alerts from check_mk concerning a database that existed on our master that did not exist on our slave... but the master and slave status outputs were good! How unnerving is that? The way to prove integrity of the process is to use checksums to verify the data.
I have seen a lot of chatter on the Internet recommending pt-table-checksum . However, its limitations proved to be too onerous for us to be comfortable with. Most importantly, it requires and even sets statement-based replication (see the pt-table-checksum link). As it says in the mysql 5.6 online documentation, (for row-based replication...) "all changes can be replicated. This is the safest form of replication." There are other disadvantages to statement-based replication that make our developers nervous because some functions cannot be replicated properly; see the doc for a list.
We have already experienced issues with a master and slave using statement-based replication so we're specifically trying to avoid it.
We are going to try mysqlrplsync which specifically mentions that it "works independently of the binary log format (row, statement, or mixed)". It also mentions, however, that gtid-mode must be on and it requires MySQL 5.6.14 and higher... which means, I believe, that the MySQL delivered with RHEL7/CentOS 7 at least is out. You'll need to get the MySQL Community Edition, which is left as an exercise for the reader but you can go here for the packages or here for the repos, including RHEL derivatives and Debian.
I have a MySQL database running on our server at this location.
However, the internet connection at this location is slow (Especially when several users are connected remotely).
We also have a remote web server on a very fast internet connection.
Can I run another MySQL server on the remote server and still be able to run queries and updates on it?
I want to have two servers because
- Users at this location can connect via lan (fast)
- Users working remotely can connect to synced remote server (fast)
Is this possible? From what I understand replication does not work this way. What is replication used for then? Backups?
Thanks for your help!
[Edit]
After doing some more reading, I am a little worried about setting up multi-master replication due to the fact that I had not considered multi-master when designing the database and conflicts could be an issue.
The good news though is that most time consuming operations are queries not updates.
And, I found out that there is a driver that handles master-slave connections.
http://dev.mysql.com/doc/refman/5.1/en/connector-j-reference-replication-connection.html
That way writes will be sent to the master and reads can come from the faster connection.
Has anyone tried doing this before? My one concern is that if I update to the master, then run a query expecting to see the update on the slave, will it be there right away? Or will the slow connection make this solution just as slow as using the master for both read and write?
What you're asking, I believe, is called Multi-Master Replication, by which both servers serve as replication masters to each other. Changes on either server become replicated back to the other as soon as possible. MySQL can be configured to do it, however I'm not sure how the differences in speed would affect your performance and data integrity.
http://dev.mysql.com/doc/refman/5.1/en/mysql-cluster-replication-multi-master.html
Is there any way to get a fault tolerant MySQL replication? I am in an environment that has many networking issues. It appears that replication gets an error and just stops. I need it to continue to work and recover from these faults. There is some wrapper software that checks the state of replication and restarts it in the case of losing its log position. Is there an alternative?
Note:
Replication is done from an embedded computer with MySQL 4.1 to a external computer that has MySQL 5.0.45
What error are you getting? You also haven't described what replication scheme or Mysql version you're using. The errors you're getting are also important.
Replication usually stops when there's a primary/unique key conflict in a Master-Master replication. Other than that on a typical Master-Slave replication setup, networking issues shouldn't cause problems.
Try using Mysql 5.1 or newer, since replication in 5.0 is statement-based and causes problems in Master-Master setups, or when you're using stored-procedures.
(Also, stay away from Mysql Cluster ... noticed the advice on another comment).
Replication errors only happen if the databases get out of sync somehow, having the server simply continue would mean incoherent databases, I really doubt you'd want that.
In my experience, the only time you end up with such errors is if one of the master servers did not complete a query and the slave noticed.
In any case, if you really want to have the slave continue via some sort of chron job, you could always have a query run every few minuts asking the slave "SHOW SLAVE STATUS" then checking the error column, if it's present, send a "STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER; START SLAVE;" command. But it would probably be much more apt to send an email to an admin when mysql encounters an error instead, so he/she can investigate the source of the problem and make sure the databases are actually in sync, otherwise you're likely to see more errors in the near future as the databases become more and more out of sync.
Consider MySQL Cluster using the NDB storage engine, it's meant to be shared-nothing and fault tolerant
MySQL replication will normally detect problems and reconnect anyway, continuing from where it left off.
If you're getting replication errors, it's likely that the source is something else. MySQL replication effectively does a "tail -f" on the query log and replays it on the slave (it's slightly smarter than that, but not much).
If the databases become out of sync, MySQL replication will neither detect nor repair this, but it may eventually cause it to break as a subsequent update cannot proceed due to conflicting data on the slave.
The default timeouts on the replication slave are much too long - it waits hours (or something) - you'll want to reduce this.
Data becoming out of sync is difficult to avoid, mitigation steps are:
Monitor replication using something like mk-table-checksum from Maatkit
Audit all your code for replication-unsafe queries
If using 5.1, switch to row-based replication, which is less likely to suffer from this problem