I'm trying to find a solution for creating a small mysql cloud. I don't need it for storage, it's more for reliability. Is it possible to have 2 identical mysql servers and if one goes down the other is still working and will sync when it comes back online? The idea is kinda like a RAID-1 setup, but intead of HDDs I want to use two whole servers.
Is there a solution like this out there? I was looking at onapp, but it may not work well with mysql.
Using MySQL replication.
using active-pasive with rdbd + heartbeat
In any case you would need use DNS to automatically update the IP or VIP
What you want is MySQL Master-Master replication, which will work as you request.
We did this once with 6 mysql servers doing replication in a circular chain. The replication chain was:
A -> B -> C -> D -> E -> F -> A
When a failover was detected the master was moved from A to B (an so forth) - DNS was updated for the master - but reads where taken from random member of the chain.
You want MySQL replication. There are a number of ways to set it up; have a look at that doc.
Related
I am trying to setup a MySQL database that takes data from 3 other MySQL databases. The data that would be copied would be a query that standardizes the data format. The method would need to either be run daily as a script or synced in real time, either method would be fine for this project.
For example:
The query from source DB:
SELECT order_id, rate, quantity
WHERE date_order_placed = CUR_DATE()
FROM orders
Then I want to take the results of that query to be inserted into a destination DB.
The databases are on separate hosts.
I have tried creating scripts that run CSV and SQL exports/imports without success. I have also tried using Python pymysql library but seemed overkill. I'm pretty lost haha.
Thanks :)
Plan A:
Connect to source. SELECT ... INTO OUTFILE.
Connect to destination. LOAD DATA INFILE from the output above.
Plan B (both MySQL):
Set up Replication from the source (as a Master) and the destination (as a Slave)
Plan C (3 MySQL servers):
Multi-source replication to allow gathering data from two sources into a single, combined, destination.
I think MariaDB 10.0 is when they introduced multi-source repl. Caution: MariaDB's GTIDs are different than MySQL's. But I think there is a way to make the replication you seek to work. (It may be as simple as turning off GTIDs??)
Plan D (as mentioned):
Some ETL software.
Please ponder which Plan you would like to pursue, then ask for help in focusing on one. Meanwhile, your question is too broad.
I have a situation here that will have 2 MySQL servers linked over a WAN. Each location (lets call them Location X and Location Y) will need access to three databases (Lets call them A, B and C.
Location X needs to use Database A as their main database, and database B as a 'lookup' read only database.
Location Y needs to use Database B as their main database, and database A as a 'lookup' read only database.
BOTH locations need read/write access to a replicated database C. (This database contains no auto incrementing columns etc. - fairly simple logging tables).
In essence, I need a combination of master-slave AND master-master replication happening:
X Y
--------
A --> A
B <-- B
C <--> C
Is this feasible to set up on MySQL?
MySQLD-Multi Is a package that can help you accomplish this. Individual MySQL databases can only have one master. If you setup A series mastered in one direction, and your B series mastered in the opposite direction, they would need to run on their own IP or port numbers. Likewise for C.
Multimaster setup becomes arduous if you get data transmission errors, lose table space, or get struck by thundering herds and drop connections. You will likely end up in the un-enviable position of using a log-based recovery tool to re-sync them. Please consider that your application will want to very cautiously use its "write" privileged connection and rely heavily on read connections.
Splitting up MySQL instances in this manner allows for latency between applications to isolated better. In a multi-master situation, you commonly have to watch the latency and begin failing things out of your pool if they differ even by 1 second of latency. If you truly need MM for your C-series, serioiusly consider chaining read-only slaves on each end for your C-series to minimize latency on the write-masters.
I'd suggest using Percona XtraDB Cluster.
PXC is a MySQL derivative server (so everything about using it will seem familiar except for the replication part). It supports synchronous bidirectional replication over a WAN, so both site X and Y can write to its main database, which will then be replicated to the other site. PXC even supports writes to a database on both sites, which satisfies your requirement for database C.
Percona XtraDB Cluster is free and open source software, like all Percona products.
Disclaimer: I worked for Percona 2010-2014.
I want to collect MySQL data from 10 different machines and aggregate into a one big MySQL db on a different machine. All machines are Linux based.
What is the "mysqldump" syntax if I want to do this periodically to collect only the "delta" data?
Are there any other ways to achieve this?
This isn't natively supported in MySQL. You could use replication, but a replica can have only a single master, not 10 masters. I know of two workable options:
1) is to script something up that switches the replica between masters in a round-robin fashion. You might wish to refer to http://code.google.com/p/mysql-mmre/ or http://thenoyes.com/littlenoise/?p=117.
2) is to use an ETL tool.
If you get stuck, we (Percona) can help you. This is a common request, but not an easy one, because each case is different.
mysqldump can't generate incremental backups, as it doesn't have any way of determining which rows (or what parts of the schema!) have changed since the last backup, or indeed even when the last backup was. For that you'd need something which could read the MySQL binlog and convert it into a bunch of INSERT/UPDATE/DELETE statements; I'm not aware of anything that exists quite like that.
The current "state of the art" in MySQL backups is generally considered to be Percona XtraBackup.
Multiple Master Slave? Have each of the 10 as Masters, and the aggregate a slave to all 10. This assumes that the data you are aggregating is different on each of the 10. If the data is the same (or similar) on all 10 and you want to interleave it as well as integrate it then this won't work.
Imagine the setup of 5 myqsl servers and 1 of them has the correct data for some tables which are being updated all the time and I would like to copy over this data to the other mysql servers.
Now I do remember working on a MySQL Replication task once where through the same website I write to the Master DB and read from the Slave DB but in this case, is this possible to do? Also is it feasible to do?
An example of a table would be "Translations". Whatever new translations are entered in one DB, they are copied to the other servers
You have answered your own question.
You need to set up replication using master - slave servers.
Where you only do updates in the master and let the slaves feed on the master.
See:
http://dev.mysql.com/doc/refman/5.0/en/replication-howto.html
http://crazytoon.com/2008/01/29/mysql-how-do-you-set-up-masterslave-replication-in-mysql-centos-rhel-fedora/
If you want a book I'd recommend: High performance MySQL.
Previously, I have experience in doing the following database replication.
(1) I have 2 tables within 1 database in Machine A
(2) I update 2 tables in Machine A
(3) Machine A will replicate 2 tables to Machine B. Machine B will also contain 2 tables within 1 database.
Now, I would like to accomplish the following :
(1) I have Table A, within 1 database in Machine A.
(2) I have Table B, within 1 database in Machine B.
(3) I would like to replicate Table A and Table B to Machine C.
(4) Machine C will have Table A and Table B, within ONE database.
Is it possible this to be accomplished, through database replication?
Unfortunately, you can have only Master per mysql server. So for instance you could run two separate instances of mysql on different ports on Machine C that slaved from Machine A and Machine B respectively, but not from both in one server.
Depending on your situation, doing that might be get you close enough that some other replication technique (like periodically using mysqldump to copy one table across on Machine C) would work. It would just depend on your needs for the slaves (how big are the tables (i.e. how quickly can they be copied via a non-slaving method), how out of date is acceptable, do you really need them in one DB or is one server good enough, etc).
After a second thought, there is one type of multi-master replication which is possible and might serve your needs if you just want the data in one DB and don't really need Machine C. In that case, you could actually have one of the servers be the Master for Table A and Slave for Table B, while the other is Master for Table B (and if necessary, Slave for Table A). Decent looking explanation.
Multi-master replication isn't really possible unless your using Cluster, then I don't think you can use the example your talking about unless the two tables are really the same data simply seperate partitions.