This is the scenario:
I have a MySQL server with a database, let's call it consolidateddb. This database a consolidation of several tables from various databases
I have another MySQL server, with the original databases, these databases are production databases and are updates daily.
The company wants to copy each update/insert/delete on each table in the production databases to the corresponding tables in consolidateddb.
Would replication accomplish that? I know that replication is done on a databas to database, but not on tables that belong to different databases to one target database.
I hope my explanation was clear. Thanks.
Edit: Would a recursive copy of all tables inn each database to the single slave work? Or is it an ugly solution?
To clear up some things, let's name things accordingly to current mysql practice. A database is a database server. A schema is a database instance. A database server can have multiple schemas. Tables live within a schema.
Replication will help you if you want to duplicate schemas or tables as they are defined on the master/production server. The replication works by shipping a binary log of all the sql statements that are run on the master to the slave which dutifully runs them as if they run sequentially on itself.
You can choose to replicate all data, or you can choose some of the schemas or even just some of the tables.
You can not choose tables from different schemas and have them replicated into one schema, a table belongs to a specific schema.
By the way, important notice. A replication server can not be a slave to multiple masters. You could mimic this using federated tables, but that would never copy the data to the consolidation server, just show them as if the data from different servers were on one server.
The bonus of replication is that your consolidation server will more or less have updated data all the time.
You could take the binary logs from each of the masters, parse them with mysqlbinlog and then run that into the consolidated machine.
Something very approximately like:
mysqlbinlog [binary log files] | mysql -h consolidated
you'd need some kind of simple application (I suspect it could be done in bash if you needed) to wrap the logic.
Check out Replicating Different Databases to Different Slaves, see if it helps you in any way.
MySQL statement-based replication (basic replication) works by running the exact same statements that were run on the master on the slave. This includes information about what database the table was in.
I don't think MySQL provides any built-in way to move replication statements between databases (i.e. "insert into db1.table1 ..." -> "insert into db2.table1"). You may be able to trick it by manually altering the replication logs on the fly, but it wouldn't be out-of-the-bod MySQL replication.
You might be able to pull it off with MySQL Proxy
You may want to check out the maatkit toolkit. It's a free download and has a host of tools that specialize in optimizing things like archiving tables. I've used it on past projects to duplicate certain data to another DB, etc. You can do it based on time or any other number of factors.
To the best of my knowledge you can set up replication (MySQL 4+) and in the my.cnf file have the slave either only process certain tables or have the master log only certain tables, either way will solve your problem.
Here is a guide to some techniques:
http://www.onlamp.com/pub/a/onlamp/2006/04/20/advanced-mysql-replication.html
I have very few problems with replication set-up, all my problems came trying to sync DBs, especially after a reboot etc.
Related
is there a way to replicate mysql while the master server already has a lot of data.I tried the normal way, but I had difficulty getting the MASTER_LOG_POS value. how can the slave server be able to replicate data that previously existed on the master server.
Generally you start with an exact full copy of your existing database. This means creating a real copy of your MySQL data directory (while the server is off), go with a (consistent) snapshot, or use a tool like Percona XtraBackup.
Only after you have 2 identical MySQL servers, you can start replicating. Note that using a tool like mysqldump is not a good idea for consistent snapshots.
If you have a relatively small amount of data you could use mysqldump --master-data=1 --single-transaction. This will create a snapshot with the correct master-binlog and position required. This should not be used for production environments or large amounts of data.
I am trying to configure MySQL databases using the Master-Slave replication. Before I realized that I had to set up my environment using this replication, I already have 2 separate servers running their own MySQL DB. Each of these servers are configured the exact same. The MySQL DB are configured with hundreds of tables.
Is there a way that i can set up (Master-Slave) Replication using the configured DB's? Or will i have to start from scratch and configure the replication first and then load in all the DB tables?
You can delete all data from one of the servers. Remaining one with the data will be your Master. Then use mysqldump to backup all the data and insert it to the slave.
Take a look for the detailed instructions on the page below:
https://livecaller.io/blog/how-to-set-up-mysql-master-slave-replication/
If the data is exactly same in both the MySQL database then you can start master slave replication, but you need to be sure that the data is same. MySQL will not check that, and if there is some discrepancy in the primary key then it will throw error immediately after next DML statement.
To be on a safer side, drop the database from one server, and restore it using the MySQL dump of another server. This will give the surety that database is same on both the server.
Take the reference from the below link to establish replication between two MySQL servers.
https://www.digitalocean.com/community/tutorials/how-to-set-up-master-slave-replication-in-mysql
We have the requirement to synchronize the data in a table across multiple MySQL databases. One of the databases would be the source, and all others need to have the synchronized data for one of the tables.
We have multiple databases used in microservice architecture, and they all need to have a local copy of a specific table in their database, and not the entire database itself, hence read replica or multi-AZ configuration is not the solution.
Database: MySQL hosted on AWS RDS
Is there any managed service by AWS or another vendor that can be used to accomplish this? Or do we have to write a custom script to do that?
It's a simple MySQL replication. But you have to replicate the whole database. So create a MySQL database in AWS and enable Multi-AZ replication and activate the automatic snapshots.
The A-Z Replication is synchronous. When you use a "Read replica" it's asynchronous. So if you have very important data you should enable Multi-AZ replication.
https://aws.amazon.com/rds/details/multi-az/?nc1=h_ls
I think you have to either isolate the table in one database and replicate that database or write a custom script.
If I were writing a custom script I would look at the binlog functionality. Here are some helpful links:
https://dev.mysql.com/doc/refman/5.7/en/mysqlbinlog.html
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html (at the bottom).
I'm setting up MySQL replication via binlog. It's a master / slave setup.
I have a few questions on how to do this the best way.
Should information_schema be replicated, if yes/no, why?
Should mysql db be replicated, if yes/no, why?
If it all should be replicated, no binlog-do-db needs to be set?
Cheers.
No, Information Schema should NEVER be replicated. This is meta data about your tables, dbs, routines, triggers, etc. Let the server populate it's own information_schema.
mysql db is fine to replicate, especially in the instance where you will be setting up a number of users. If your master fails, you will need the same user information available on your slave server in order to fail over to it. If that should occur, you need to have the same privileges available.
List only the database you want replicated with binlog-do-db. If you don't list any, everything will be replicated by default.
No. need to replicate information schema read this for detail https://dev.mysql.com/doc/refman/5.0/en/information-schema.html
Yes you can because mysql database contains all user credentials so in order to have a backup of users you can replicate it.
Yes use binlog-do-db to replicate selected databases otherwise everything will be replicated if nothing mentioned.
When I have two mysql servers that have different jobs (holding different databases) but want to be able to use one of them to slip in when the other one fails, what would you suggest how I keep the data on both of them equal "close to realtime"?
Obviously it's not possible to make a full database dump every x minutes.
I've read about the Binary Log, is that the way that I need to go? Will that not slow down the fallback server a lot? Is there a way to not include some tables in the binary log - where it doesn't matter that the data has changed?
You may want to consider the master-master replication scenario, but with a slight twist. You can specify which databases to replicate and limit the replication for each server.
For server1 I would add --replicate-do-db=server_2_db and on server2 --replicate-do-db=server_1_db to your my.cnf (or my.ini on Windows). This would mean that only statements for the server_1_db would be replicated to server2 and vice verse.
Please also make sure that you perform full backups on a regular basis and not just rely on replication as it does not provide safety from accidental DROP DATABASE statements or their like.
Binary log is definitely the way to go. However, you should be aware that with MySQL you can't just flip back and forth between servers like that.
One server will be the master and the other will be the slave. You write/read to the master, but can only read from the slave server. If you ever write to the slave, they'll be out of sync and there's no easy way to get them to sync up again (basically, you have to swap them so the master is the new slave, but this is a tedious manual process).
If you need true hot-swappable backup databases you might have to go to a system other than MySQL. If all you want is a read-only live backup that you can use instantly in the worst-case scenario (master is permanently destroyed), Binary Log will suit you just fine.