Lets say I have a datbase of Cars. I have Makes and Models (FK to Makes). I plan on having users track their cars. each Car has a FK to Model. Now, I have a lot of users, and I want to split up my database to distribute load. The Makes and Models tables don't change so much, but they need to be shared across shards. My thought is to use MySQL replication from a master DB of makes and models to each slave database. My question is: Can I safely write to the slave databases assuming I don't write to those tables on the master?
And while on the subject, is there anyway to guarantee one slave database has the latest data? For example, someone just added the 'Taurus' make, and then wants to add their car. Can I ensure that the slave database they are using has the latest master data?
Yes, in general you can safely write to a table on the slaves that is not being written on the master. If you do things like insert auto_increment rows on the slaves and on the master, independently, you will of course have problems. You should configure that table to be excluded from replication entirely, really.
For checking whether you have the latest data, SHOW SLAVE STATUS includes a field Seconds_Behind_Master that tells you whether the slave is up to date. Obviously you want it to be zero. To be certain that inserted and replicated data is present, of course, you need to wait a second and then see that Seconds_Behind_Master is zero.
This was a good solution I gleaned while searching
I included the main point as avilable here:
http://erlycoder.com/43/mysql-master-slave-and-master-master-replication-step-by-step-configuration-instructions-
MySQL master-master replication and autoincrement indexes
If you are using master-slave replication, than most likely you will design your application the way to write to master and read from slave or several slaves. But when you are using master-master replication you are going to read and write to any of master servers. So, in this case the problem with autoincremental indexes will raise. When both servers will have to add a record (different one each server simultaneously) to the same table. Each one will assign them the same index and will try to replicate to the salve, this will create a collision. Simple trick will allow to avoid such collisions on MySQL server.
On the Master 1/Slave 2 add to /etc/my.cnf:
auto_increment_increment= 2
auto_increment_offset = 1
On the Master 2/Slave 1 add to /etc/my.cnf:
auto_increment_increment= 2
auto_increment_offset = 2
Related
I'm currently thinking about the following problem:
A customer has set up a simple master/slave replication between two mariaDB systems. For unknown reasons they have set the flag "Replicate_Wild_Ignore_Table" to skip "logdb.%". Obviously, they decided to skip the skipping of that database and want the logdb to be included in the replication again.
I'm curious now, is it possible to somehow remove that flag and have the database in question be replicated as the rest or is there no way to circumvent the "stop slave, dump master, import dump, recreate replication based on current logpos, start slave" procedure?
You can't assume that the master still has all relevant binlogs that once contained updates to the logdb.% tables. That is, even if you could re-apply those updates, do you have enough history to account for all changes to the tables?
Another risk is if you use statement-based replication, if there were ever statements that referenced both a table in logdb.% and a table in another database, the replication filter has skipped that statement. So for example:
INSERT INTO mydb.mytable SELECT * FROM logdb.othertable;
Therefore even the tables that are not in logdb.% might be compromised. The point is you don't know for sure.
The bottom line is that you should definitely reinitialize the replica now by taking a current backup of the master, and avoid using replication filters in the future.
If you use InnoDB tables, you might consider using Percona XtraBackup to make the process easier. See https://www.percona.com/doc/percona-xtrabackup/2.3/howtos/setting_up_replication.html
I have a couple of servers and I want to prepare a Master - Slave Mysql replication for one database. The both servers have many databases and I don't want to altere the secuence of ID generation. For example, after I have prepared the configuration I don't want to have in the tables just even numbers for the all the IDs in one of servers.
The replicated database (slave server) will be not accesed for write.
Is posible to configure that scenario?
Many thanks in advance.
Update: The server_id has nothing to do with id generation. It just needs to be a unique integer greater than 0 on each server in your replica-set.
Below is my original answer, which was my guess about what you were asking about, because it's the only feature I could think of that has to do with both replication and auto-increment id generation.
You don't need to change id generation for simple replication.
The scenario where you might use auto_increment_increment=2 is the master-master replication, where two servers replicate from each other, and you want to minimize the risk of split-brain if an insert occurs on both servers. But this is not the scenario you describe.
If you have one master, and it's the only server you write changes on directly, and the replica(s) that replicate from that master are all read-only, then you don't need to change the auto_increment_increment.
I have 2 Databases
Database 1,
Database 2
Now Each Database has Table say Table 1(IN DATABASE 1) and Table 2(IN DATABASE 2).
Table 1 is Basically a Copy of Table 2(Just for Backup).
How can i Sync Table 2 if Table 1 is Updated?
I am using MYSQL,Storage Engine:InnoDBand in back-end programming i am using php.
Further i can check for update after every 15 minutes using php script but it takes too much time because each table has51000 rows.
So, How can i achieve something like if Administrator/Superuser updates table 1, that update should me immediately updated in Table 2.
Also, is there a way where Bi-Directional Update can work i.e Both can be Masters?
Instead Table 1 as the only master, Both Table 1 and Table 2 can be Master's? if any update is done at Any of the tables other one should update accordingly?
If not wrong, what you are looking for is Replication which does this exact thing for you. If you configure a Transnational Replication then every DML operation will get cascaded automatically to the mirrored DB. So, no need for you to do continuously polling from your application.
Quoted from MySQL Replication document
Replication enables data from one MySQL database server (the master)
to be replicated to one or more MySQL database servers (the slaves).
Replication is asynchronous - slaves need not be connected permanently
to receive updates from the master. This means that updates can occur
over long-distance connections and even over temporary or intermittent
connections such as a dial-up service. Depending on the configuration,
you can replicate all databases, selected databases, or even selected
tables within a database.
Per your comment, Yes Bi-Directional Replication can also be configured.
See Configuring Bi-Directional Replication
As Rahul stated, what you are looking for is replication.
The standard replication of mysql is master -> slave which means that one of the databases is "master", the rest slaves. All changes must be written to the master db and will then be copied to the slaves. More info can be found in the mysql documentation on replication.
There is aslo an excellent guide on the digitaloceans community forums on master <-> master replication setup.
If the requirements for "Administrator/Superuser" weren't in your question, you could use the mysql's Replication functions on the databases.
If you want the data to be synced immediately to the Table2 upon inserting in Table1, you could use a trigger on the table. In that trigger you can check which user (if you have a column in that table specifying which user inserted the data) submitted data. If the user is an admin, configure the trigger to duplicate the data, if the user is a normal user, don't do anything.
Next for normal users entering data, you could keep an counter on each row, increasing by 1 if it's a new 'normal' user's data. Again in the same trigger, you could also check for what number the counter already is. Let's say if you reach 10, then duplicate all the rows to the other table and reset the counter + remove the old counter values from the just-duplicated-rows.
I have a MySQL DB with a quickly growing amount of data.
I'd like to use some web based tool that plugs into the DB so that I can analyze data and create reports.
The idea would be to use replication in order to give R/O access to the slave DB instead of having to worry about security issues on the master (which also contains data not relevant to this project, but just as important).
The master DB contains strings that are hashed (SHA1 128) from the source and, on the slave, they need to go back to their original form using a reverse hash database.
This will allow whatever tool I plug into the slave-DB (living on another server) to work straight out of the box.
My question is: what is the best way to do replication while somehow reshaping the slave-DB with the mentioned strings back into the source format?
example
MASTER DB
a8866ghfde332as
a8fwe43kf3e3t42
SLAVE DB
John Smith
Rose White
The slave DB should already contain the tables reversed and should NOT be reversed when doing a query.
How do you guys think I should approach this?
Is replication the way to go?
Thank you for any help!
EDIT
I should specify some details:
the slave DB would also contain a reverse hash (lookup) table
the amount of source strings is limited so there's little risk of collisions
the best option would be to replicate only certain tables to the slave, where the slave-DB does a reverse hash lookup every time there is an INSERT and saves the reversed hash in another table (or column) ready to be read by the web based tool
This type of setup I am willing to use is mainly focused on NOT having anything connecting to the master other than the source (that creates records in the DB) and the slave DB itself.
This would result in better security by having the reverse lookup table sitting in a DB (the slave) that is NOT in direct contact with the source of data.
So, even in case somebody hacks the source and makes it to the master DB, no useful data could be retrieved being the strings in object hashed.
It is easier, simpler, and most foolproof to replicate everything from master to slave in MySQL, so plan to relicate everything unless you have an extremely compelling reason not to.
That said, MySQL has absolutely no problem with the slave having tables that the master does not have -- tables created directly on the slave will not cause a problem if no tables with conflicting schema+table names exist on the master.
You don't want to try to have the slave "fix" the data on the way in, because that's not something MySQL replication is designed to do, nor is it something readily accomploahed. Triggers will fire on tables on the slave only when the master writes events to its binlog in statement mode, which is not as reliable as row mode nor as flexible as mixed mode, and even if you had this working, you then lose the ability to compare master and slave data sets with table checksums, which is an important part of the ongoing maintenance of master/slave replication.
However... I do see a way to accomplish what you want to do with the reverse hash tables: create the lookup tables on the slave, and then create views that reconstruct the data in its desired form by joining the replicated tables to the lookup tables, and run your queries on the slave against these views.
When a view simply joins properly indexed tables and doesn't include anything unusual like aggregate functions (e.g. COUNT()) or UNION or EXISTS, then, the server will process queries against views as if the underlying tables had been queried directly, using all available and appropriate indexes... so this approach should not cause significant performance penalties. In fact, you can declare indexes on replicated tables on the slave (on replicated tables) that you don't have or need on the master (except for UNIQUE indexes, which wouldn't make sense) and these can be designed as needed for the slave-specific workload.
Hash functions are surjective, so it is possible for two different inputs to have the same output. As such, it would not be possible to accurately rebuild the data on the slave.
On a simple level, and to demonstrate this; consider a hashing function for integers, that happens to return the square of the input; so, -1 => 1, 0 => 0, 1 => 1, 2 => 4, 3 => 9, etc. Now consider the inverse, being the square root, 1 => -1 & 1, 4 => -2 & 2, etc.
It may be better to only replicate the specific data needed for your project to the slaves, and do it without hashing.
Imagine the setup of 5 myqsl servers and 1 of them has the correct data for some tables which are being updated all the time and I would like to copy over this data to the other mysql servers.
Now I do remember working on a MySQL Replication task once where through the same website I write to the Master DB and read from the Slave DB but in this case, is this possible to do? Also is it feasible to do?
An example of a table would be "Translations". Whatever new translations are entered in one DB, they are copied to the other servers
You have answered your own question.
You need to set up replication using master - slave servers.
Where you only do updates in the master and let the slaves feed on the master.
See:
http://dev.mysql.com/doc/refman/5.0/en/replication-howto.html
http://crazytoon.com/2008/01/29/mysql-how-do-you-set-up-masterslave-replication-in-mysql-centos-rhel-fedora/
If you want a book I'd recommend: High performance MySQL.