I've been having a hard time to figure out how to configure distributed database system in MySQL. I have 3 servers, each has the same database model. I have configured them for using replication - so there is 1 Master and 2 Slaves. Everything works fine, but what I want to do next is to "filter" replication data.
Let's say that I have two tables: customers and products. I want to replicate all content of products table to both slaves (and I got this part), but I want to replicate customers from Europe to Slave1 and from Asia to Slave2. So master will contain all information about customers, but slaves only some part of it. How can I achive that?
As far as I know replication itself doesn't support this kind of filtering. I am not sure, but it seems like partitioning is not an answer as well.
Are there any build-in mechanizm in MySQL that may help? If not, how would you resolve this situation?
For this you have to use custom way to read data and based on your business logic add data to slave 1 or slave 2.
For instance, you can read change in database entry from binlog and based on your input data, you can store it in different slave databases. However, there might be a small delay in this.
There is a very good system which does the same, ie maxwell(https://github.com/zendesk/maxwell) which uses kafka to push database change event with type of query and old and new data. Now on the basis of that you can write kafka consumer to read data and push it into your slave databases. maxwell uses binlog replicator same as Master/Slave replicator and is reliable.
Related
My problem is I have a website that customers place orders on. That information goes into orders, ordersProducts, ...etc tables. I have a reporting Database on a DIFFERENT server where my staff will be processing the orders from. The tables on this server will need the order information AND additional columns so they can add extra information and update current information
What is the best way to get information from the one server (order website) to the other (reporting website) efficiently without the risk of data loss? Also I do not want the reporting database to be connecting to the website to get information. I would like to implement a solution on the order website to PUSH data.
THOUGHTS
mySQL Replication - Problem - Replicated tables are strictly for reporting and not manipulation. Example what if customer address changes? Need products added to order? This would mess up the replicated table.
Double Inserts - Insert into Local tables and then insert into Reporting Database. Problem - If for whatever reason the reporting database goes down there is a chance I lose data because the mySQL connection wont be able to push the data. Implement some sort of query log?
Both Servers use mySQL and PHP
Mysql replication sounds exactly like what you are looking for, I'm not too sure I understand what you've listed as the disadvantage there.
The solution to me sounds like a master to read-only slave where the slave is the reporting database. If your concern is changes to the master then making the slave out of sync then this shouldn't be too much of an issue, all changes will be synced over. In the situation of a loss of connectivity then the slave would track how many seconds it is behind master and execute the changes until the two are back in sync.
We are moving to a master/multi-slave MySQL setup and I am concerned about how to tackle the issue of data integrity without changing the application code too much.
Our application uses an ORM (Doctrine PHP) and in theory we could extend it to simply send SELECT statements to one of the slaves or to the master and UPDATE/DELETE queries to the master.
where this would fail due to the nature of the ORM is:
Doing writes based on outdated data from a slave, e.g. read a record from the slave that has recently had its column Y changed on the master but not yet on the slave and then saving a change on column X together with the outdated data of column Y on the master, thus overwriting the prior change to column Y.
Doing reads from the slave right after a recent write on the master (scenario where a user just posted a form - say a forum comment - and reloads the page and does not see his comment)
I am thinking that we should build the logic on our application controllers that should let the data layer know whether there is tolerance for getting outdated data and when not (e.g. when there is an intention to write) and when not so that the data layer can connect to master or slave accordingly, however; it seems against good software engineering practices. E.g. controllers should be implementing business logic not having to deal with whether the data should come from a slave or a master.
I am sure this has been dealt with in the past. Maybe not with Doctrine PHP, but with Hibernate or other ORM solutions?
Are there any general good practices or recommendations one could share?
Two machines, each running mysql, each synchronized to the other peer-to-peer. I do not want a master db replicated. Rather, I want two users to be able to work on the data offline (each running a mysql server on his machine) and then when reconnected synchronize to each other. Any way to do this with mysql? Any other database I should be looking at to accomplish this better than mysql?
Two-way replication is provided by various database systems (e.g. SQLServer, Sybase etc.) but there are always problems with such a set up.
For example, if the same row is updated at the same time on the two databases, which update wins?
If your aim is to provide a highly-available MySQL database, then there are better options than using replication. MySQL has a clustering solution (though I've not had much success with it) or you can use things like DRBD and heartbeat to provide automatic failover with no loss of data.
If you mean synchronous writing back and forth, this would cause serious data consistency issues. I think you may be referring to MySQL replication, wherein a master server sends its updates to one or more slave database servers, which can be queried.
As for "Other Database Options" SQLServer supports a fairly advanced "replication" process for synchronizing the data between two or more db's. Looks like MySql has something like this as well though.
I have several databases located in different locations and a central database which is in a data center. All have the same schema. All of them are changed(insert/update/delete) in each location with different data including the central database.
I would like to synchronise all the data in the central database. I would also like all data in the central database synchronise to all locations. What I mean is that database change in location 1 should also be reflected in location 2 database.
Any ideas on how to go about this?
Just look at SymmetricDS. It is a data replication software that supports multiple subscribers and bi-directional synchronization.
You will have to implement a two-way replication scheme between the databases. Every new record created should have a unique identifier (eg. a GUID), so that data from the different databases does not conflict. (See the mysql replication howto).
MySql only supports one-way replication, so you will need to set up each database as a master, and make each database a slave of all the other database instances. Good luck with that.
I went to SymmetricDS
I think it is the top quality also just to mention I found in sourceforge (php mysql sync)
and found many links on the internet.
Unfortunately, MySQL replication capabilities won't allow you to do exactly what you want.
Usually, to synchronize two servers the master-master replication scheme can be used. See http://www.howtoforge.com/mysql_master_master_replication
The problem is that each MySQL server can have ONLY ONE master.
The only way I know to keep several servers synchronized, would be a circular replication (see http://onlamp.com/pub/a/onlamp/2006/04/20/advanced-mysql-replication.html?page=2), but won't exactly fit your needs ("star" configuration)
Maybe this configuration could be close enough : all the "distant" (non central) databases would be only be readable slaves (synchronized though a basic master-slave replication), and writings would only occur on the central server (which would be master).
I've got two rails applications. One is internal and second is external client version.
In client version I have got cutted version of database. So, now I need to replicate my master MySQL db but not all data: only certain columns and certain tables.
How can I implement this job?
If there are some ruby stuff (gem for working with replication in this way), it'll be great.
Replication is typically something you do at the database layer, here is the documentation for Mysql replication:
http://dev.mysql.com/doc/refman/5.0/en/replication.html
That would typically replicate the entire database.
Another solution would be to have a job (perhaps written in ruby), that runs a couple of times a day and copies the desired data.
Perhaps you want to push data from the master to the slaves with as little delay as possible? Then you could make a hook on the save() method in ActiveRecord, that pushes the changes to the slave db.
Haven't looked in to it, but perhaps this is something: http://www.rubyrep.org/