I'm currently thinking about the following problem:
A customer has set up a simple master/slave replication between two mariaDB systems. For unknown reasons they have set the flag "Replicate_Wild_Ignore_Table" to skip "logdb.%". Obviously, they decided to skip the skipping of that database and want the logdb to be included in the replication again.
I'm curious now, is it possible to somehow remove that flag and have the database in question be replicated as the rest or is there no way to circumvent the "stop slave, dump master, import dump, recreate replication based on current logpos, start slave" procedure?
You can't assume that the master still has all relevant binlogs that once contained updates to the logdb.% tables. That is, even if you could re-apply those updates, do you have enough history to account for all changes to the tables?
Another risk is if you use statement-based replication, if there were ever statements that referenced both a table in logdb.% and a table in another database, the replication filter has skipped that statement. So for example:
INSERT INTO mydb.mytable SELECT * FROM logdb.othertable;
Therefore even the tables that are not in logdb.% might be compromised. The point is you don't know for sure.
The bottom line is that you should definitely reinitialize the replica now by taking a current backup of the master, and avoid using replication filters in the future.
If you use InnoDB tables, you might consider using Percona XtraBackup to make the process easier. See https://www.percona.com/doc/percona-xtrabackup/2.3/howtos/setting_up_replication.html
Related
In MySQL master-slave synchronization, we configured the ignore table because of the performance problem of the slave. However, since the configuration is configured on the slave, when the host writes data in large quantities, will the data be synchronized to the slave and then filtered by the slave? In fact, if you ignore the table configuration, will the performance improvement be minimal? Or does it mean that when this data is written to the host, the ignore table information of the slave is read? Data transmission will not occur, which can greatly reduce the load of the slave? After all, I configure the ignore table to avoid the performance problems of the slave. If a large amount of data is written, the slave can't carry it.
Create a new database; I'll call it NoRepl
Put binlog-ignore-db=NoRepl on the Primary.
Move that big table into the database NoRepl.
That way, the filtering will occur much sooner -- thereby not clogging up replication either on the Replica or in the Primary's binlog.
If you are using Primary-Primary or you have another Replica that does want to see this table, then my suggestion will not work.
I need to encrypt some data on MySQL slave and using AES_Encrypt for the same. Is it possible to encrypt it while replicating it from the Master database?
Using MySQL 5.6
In MySQL replication, when using binlog_format=STATEMENT, triggers fire on the master and again on the slave, but there is no specific requirement that the triggers be defined identically on the master and slave, or that triggers be provisioned on both.
I say there is no specific requirement, because it is not enforced. If you want the data to be identical on master and slave -- as is almost always the case in a replicated environment -- then yes, the triggers have to be defined identically on both servers... but it is possible to customise the slave triggers -- and one way of doing it would be to define BEFORE INSERT and BEFORE UPDATE triggers on the slave that encrypted the columns before inserts and updates.
This does require STATEMENT replication, which I don't use because it is more delicate and some statements cannot be deterministically replicated in STATEMENT mode.
This would require extra care to ensure that master and slave are always consistent, and would require custom tools since standard table comparison tools would consider master and slave tables to be different.
With binlog_format=ROW, any triggers defined on the slave are ignored for replicated events, because they would be redundant -- the changes done by the triggers on the master will already be replicated to the slave as row events, and executing the triggers on the slave would not have the correct result. Row-based replication uses raw row images, so is always deterministic, unlike statement based replication.
The MariaDB team recognizes that there may be cases where identical data sets are not the objective, such as where you might want to build denormalized query tables on the slave, so in MariaDB 10.1, they introduced a feature to allow slaves to fire triggers for row-based events. This might also be a viable solution, if your master is not using statement-based replication. It does not appear to require the master to run MariaDB, only the slave.
https://mariadb.com/kb/en/mariadb/running-triggers-on-the-slave-for-row-based-events/
It is also possible to create accounts that cannot access all columns, since the MySQL permissions model allows grants to be made at the server, database, table, and column level. Without permission to access a column, you won't see it, in spite of it being there.
Arguably, though, if the data is sensitive, the most correct solution would be to encrypt it on the master.
I have been looking for a way to prevent MySQL delete statements from getting processed by the slave, I'm working on data warehousing project, and I would like to delete data from production server after having data replicated to slave.
what is the best way to get this done?
Thank you
There are several ways to do this.
Run SET SQL_LOG_BIN=0; for the relevant session on the master before executing your delete. That way it is not written to the binary log
Implement a BEFORE DELETE trigger on the slave to ignore the deletes.
I tend to use approach #1 for statements that I don't want to replicate. It requires SUPER privilege.
I have not tried #2, but it should be possible.
You'll only be able to achieve this with a hack, and it will likely cause problems. MySQL replication isn't designed for this.
Imagine you insert a record in your master, it replicates to the slave. You then delete from the master, but it doesn't delete from the slave. If someone adds a record with the same unique key, there will be a conflict on the slave.
Some alternatives:
If you are looking to make a backup, I would do this by another means. You could do a periodic backup with a cronjob that runs mysqldump, but this assumes you don't want to save EVERY record, only create periodic restore points.
Triggers to update a second, mirror database. This can't cross servers though, you'd have to recreate each table with a different name. Also, the computational cost would be high and restoring from this backup would be difficult.
Don't actually delete anything, simply create a Status field which is Active or Disabled, then hide Disabled from the users. This has issues as well, for example, ON DELETE CASCADE couldn't be used, it would have to be all manually done in code.
Perhaps if you provide the reason you want this mirror database without deletes, I could give you a more targeted solution.
I wonder if there is any easy way to keep the scheme consistent in two different MySQL clusters. Apart from classic replication, I would like to have a special "replication" which would reproduce all DDL queries (CREATE, ALTER, DROP, ...) on another cluster (namely the master of that cluster).
I don't need the actual data to be replicated.
Has anyone ever done or tried anything like this?
You can filter replication in MySQL based upon the database in which a query was executed. That doesn't prevent you making changes in other databases, however! So you can do;
USE ddl_repl_db;
ALTER TABLE other_db.foo ADD COLUMN <etc>
This relies on you configuring your servers correctly. I haven't set up MySQL replication for a while, but IIRC you can both filter what you send out from the master for replication and what you accept on the slave.
Old but still high in search.
So, on you DDL replica set all tables engine to BLACKHOLE
Lets say I have a datbase of Cars. I have Makes and Models (FK to Makes). I plan on having users track their cars. each Car has a FK to Model. Now, I have a lot of users, and I want to split up my database to distribute load. The Makes and Models tables don't change so much, but they need to be shared across shards. My thought is to use MySQL replication from a master DB of makes and models to each slave database. My question is: Can I safely write to the slave databases assuming I don't write to those tables on the master?
And while on the subject, is there anyway to guarantee one slave database has the latest data? For example, someone just added the 'Taurus' make, and then wants to add their car. Can I ensure that the slave database they are using has the latest master data?
Yes, in general you can safely write to a table on the slaves that is not being written on the master. If you do things like insert auto_increment rows on the slaves and on the master, independently, you will of course have problems. You should configure that table to be excluded from replication entirely, really.
For checking whether you have the latest data, SHOW SLAVE STATUS includes a field Seconds_Behind_Master that tells you whether the slave is up to date. Obviously you want it to be zero. To be certain that inserted and replicated data is present, of course, you need to wait a second and then see that Seconds_Behind_Master is zero.
This was a good solution I gleaned while searching
I included the main point as avilable here:
http://erlycoder.com/43/mysql-master-slave-and-master-master-replication-step-by-step-configuration-instructions-
MySQL master-master replication and autoincrement indexes
If you are using master-slave replication, than most likely you will design your application the way to write to master and read from slave or several slaves. But when you are using master-master replication you are going to read and write to any of master servers. So, in this case the problem with autoincremental indexes will raise. When both servers will have to add a record (different one each server simultaneously) to the same table. Each one will assign them the same index and will try to replicate to the salve, this will create a collision. Simple trick will allow to avoid such collisions on MySQL server.
On the Master 1/Slave 2 add to /etc/my.cnf:
auto_increment_increment= 2
auto_increment_offset = 1
On the Master 2/Slave 1 add to /etc/my.cnf:
auto_increment_increment= 2
auto_increment_offset = 2