I wonder if there is any easy way to keep the scheme consistent in two different MySQL clusters. Apart from classic replication, I would like to have a special "replication" which would reproduce all DDL queries (CREATE, ALTER, DROP, ...) on another cluster (namely the master of that cluster).
I don't need the actual data to be replicated.
Has anyone ever done or tried anything like this?
You can filter replication in MySQL based upon the database in which a query was executed. That doesn't prevent you making changes in other databases, however! So you can do;
USE ddl_repl_db;
ALTER TABLE other_db.foo ADD COLUMN <etc>
This relies on you configuring your servers correctly. I haven't set up MySQL replication for a while, but IIRC you can both filter what you send out from the master for replication and what you accept on the slave.
Old but still high in search.
So, on you DDL replica set all tables engine to BLACKHOLE
Related
Since MySQL 8.0 has depreciated utf8mb3 (and will subsequently completely remove utf8mb3 support in future), what is the proper approach to upgrade/convert complete database (in production) from utf8mb3 (utf-8) to utf8mb4 with no (or minimal) downtime?
The problem is not converting, I know there exist multiple scripts for this conversion. My major concern is downtime. Is it possible to achieve this with no downtime?
Assuming no Slaves, 8.0, no Triggers, and a few other things, I would recommend
pt-online-schema-change
It will CREATE TABLE with the new schema (utf8mb4), then copy the rows from the existing table (utf8 aka utf8mb3), and use Triggers to keep things in sync. There is a brief lock on the table at the end to finish up and swap tables.
You would do that one table at a time.
Doc: https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html
Also search around for it; sometimes abbreviated pt-osc.
With a Slave
Investigate the "failover" options and speed with RDS. If you can control the failover, then this method may be fast and 'simple' and possible. I assume it is really Master-Slave, not Master-Master?
With the Slave, leave it replicating from the Master. (Potential problem: RBR may squawk at utf8 replicating to utf8mb4. This needs investigating.)
Do ALTER TABLE ... CONVERT TO ... for each table on the Slave.
Failover.
Depending on what tools RDS has, it may be better to rebuild the Slave(s) from the new Master instead of doing ALTERs on the new Slave (which is the old Master).
Suggest you spin up some tiny instances with the same Master-Slave topology and version and charsets. Then try the steps. It does not need may rows, but it does need FOREIGN KEYs, Triggers, etc, in case any of them cause trouble. (Note how pt-osc has some issues with FKs.)
With phpmyadmin,
Select the database, then you'll get a view of all table structures,
Select all tables,then select Operations at the top menu, you'll find a Collation feature, at the end of the page, select the utf8mb4_unicode_ci,then mark the "Change all tables collations" & "Change all tables columns collations" checkboxes, then click go.
The operation takes utmost 7 seconds, and you are done.
I'm currently thinking about the following problem:
A customer has set up a simple master/slave replication between two mariaDB systems. For unknown reasons they have set the flag "Replicate_Wild_Ignore_Table" to skip "logdb.%". Obviously, they decided to skip the skipping of that database and want the logdb to be included in the replication again.
I'm curious now, is it possible to somehow remove that flag and have the database in question be replicated as the rest or is there no way to circumvent the "stop slave, dump master, import dump, recreate replication based on current logpos, start slave" procedure?
You can't assume that the master still has all relevant binlogs that once contained updates to the logdb.% tables. That is, even if you could re-apply those updates, do you have enough history to account for all changes to the tables?
Another risk is if you use statement-based replication, if there were ever statements that referenced both a table in logdb.% and a table in another database, the replication filter has skipped that statement. So for example:
INSERT INTO mydb.mytable SELECT * FROM logdb.othertable;
Therefore even the tables that are not in logdb.% might be compromised. The point is you don't know for sure.
The bottom line is that you should definitely reinitialize the replica now by taking a current backup of the master, and avoid using replication filters in the future.
If you use InnoDB tables, you might consider using Percona XtraBackup to make the process easier. See https://www.percona.com/doc/percona-xtrabackup/2.3/howtos/setting_up_replication.html
I need to encrypt some data on MySQL slave and using AES_Encrypt for the same. Is it possible to encrypt it while replicating it from the Master database?
Using MySQL 5.6
In MySQL replication, when using binlog_format=STATEMENT, triggers fire on the master and again on the slave, but there is no specific requirement that the triggers be defined identically on the master and slave, or that triggers be provisioned on both.
I say there is no specific requirement, because it is not enforced. If you want the data to be identical on master and slave -- as is almost always the case in a replicated environment -- then yes, the triggers have to be defined identically on both servers... but it is possible to customise the slave triggers -- and one way of doing it would be to define BEFORE INSERT and BEFORE UPDATE triggers on the slave that encrypted the columns before inserts and updates.
This does require STATEMENT replication, which I don't use because it is more delicate and some statements cannot be deterministically replicated in STATEMENT mode.
This would require extra care to ensure that master and slave are always consistent, and would require custom tools since standard table comparison tools would consider master and slave tables to be different.
With binlog_format=ROW, any triggers defined on the slave are ignored for replicated events, because they would be redundant -- the changes done by the triggers on the master will already be replicated to the slave as row events, and executing the triggers on the slave would not have the correct result. Row-based replication uses raw row images, so is always deterministic, unlike statement based replication.
The MariaDB team recognizes that there may be cases where identical data sets are not the objective, such as where you might want to build denormalized query tables on the slave, so in MariaDB 10.1, they introduced a feature to allow slaves to fire triggers for row-based events. This might also be a viable solution, if your master is not using statement-based replication. It does not appear to require the master to run MariaDB, only the slave.
https://mariadb.com/kb/en/mariadb/running-triggers-on-the-slave-for-row-based-events/
It is also possible to create accounts that cannot access all columns, since the MySQL permissions model allows grants to be made at the server, database, table, and column level. Without permission to access a column, you won't see it, in spite of it being there.
Arguably, though, if the data is sensitive, the most correct solution would be to encrypt it on the master.
I have been looking for a way to prevent MySQL delete statements from getting processed by the slave, I'm working on data warehousing project, and I would like to delete data from production server after having data replicated to slave.
what is the best way to get this done?
Thank you
There are several ways to do this.
Run SET SQL_LOG_BIN=0; for the relevant session on the master before executing your delete. That way it is not written to the binary log
Implement a BEFORE DELETE trigger on the slave to ignore the deletes.
I tend to use approach #1 for statements that I don't want to replicate. It requires SUPER privilege.
I have not tried #2, but it should be possible.
You'll only be able to achieve this with a hack, and it will likely cause problems. MySQL replication isn't designed for this.
Imagine you insert a record in your master, it replicates to the slave. You then delete from the master, but it doesn't delete from the slave. If someone adds a record with the same unique key, there will be a conflict on the slave.
Some alternatives:
If you are looking to make a backup, I would do this by another means. You could do a periodic backup with a cronjob that runs mysqldump, but this assumes you don't want to save EVERY record, only create periodic restore points.
Triggers to update a second, mirror database. This can't cross servers though, you'd have to recreate each table with a different name. Also, the computational cost would be high and restoring from this backup would be difficult.
Don't actually delete anything, simply create a Status field which is Active or Disabled, then hide Disabled from the users. This has issues as well, for example, ON DELETE CASCADE couldn't be used, it would have to be all manually done in code.
Perhaps if you provide the reason you want this mirror database without deletes, I could give you a more targeted solution.
We have a moderately large production MySQL database. Periodically, we will run commands, usually via a rails migration, that while running, bog down the database. As a specific example, we might add an index to a large table.
Is there any method that can reduce the priority MySQL gives to a specific task. A sort of "nice" within MySQL itself? I found this, which is what inspired the question:
PostgreSQL tips and tricks
Since adding an index causes the work to be done within the DB and MySQL process, lowering the priority of the Rails migration process seems like it won't help. Are there other ways we can lower the priority?
We use multiple, replicated database servers to make changes like this.
In our case, db1 is the master, replicated to db2. (db1->db2).
Start by making the change to db2. If things lock, replication will stall, but that's OK.
Move your traffic to db2. Any remnant traffic going to db1 will replicate over, so you won't lose anything.
Once there's no traffic on db1, rebuild it as a slave of db2 (db2->db1).
That's the general idea and you get very little downtime and you don't have to pull an all-nighter! We actually have three servers, so it's a little more complicated, but not much.
Good luck.
Unfortunately, there is no simple way to do this: commands that alter the database structure don't have a priority option.
If your tables are MyISAM, you can try this:
mysqlhotcopy to make a backup of the table
import that backup it into a different database server (one that's not under load)
make the changes there
make a mysqlhotcopy backup of the altered table
import it into the live server
Note that this may or may not be faster than adding the index on the live server, depending on the time it takes you to transfer the table back and forth.