Can anyone tell me the exact difference in replicate_wild_do_table and replicate_do_table while creating Slave.
Thanks
replicate-wild-do-table allows you to restrict replication statements to databases and tables using % and _ wildcard characters, that is perform pattern matching.
Where replicate-do-table uses concrete databases and tables names.
Here's a manual reference: link
UPDATE:
Seems like you are having trouble with USE statement or fully qualified table names (those with database name specified) in your setup. First of all, there are 2 types of binary logging:
Statement-based
Row-based
In short, statement-based replication logs every statement that could modify data and row-based logs modified rows.
Here's a manual reference: link
Here's a comparison from the manual: link
Those two types imply different behaviour for replication options. The key one here is that for statement-based replication MySQL replicates statements that are performed on default (selected via USE) database only.
So the question is: do you need cross-database updates? If so, you can't use replicate-do-db, because it restricts replication on database level. In terms of cross-database updates, there is no difference between replicate-wild-do-table and replicate-do-table.
Related
Since MySQL 8.0 has depreciated utf8mb3 (and will subsequently completely remove utf8mb3 support in future), what is the proper approach to upgrade/convert complete database (in production) from utf8mb3 (utf-8) to utf8mb4 with no (or minimal) downtime?
The problem is not converting, I know there exist multiple scripts for this conversion. My major concern is downtime. Is it possible to achieve this with no downtime?
Assuming no Slaves, 8.0, no Triggers, and a few other things, I would recommend
pt-online-schema-change
It will CREATE TABLE with the new schema (utf8mb4), then copy the rows from the existing table (utf8 aka utf8mb3), and use Triggers to keep things in sync. There is a brief lock on the table at the end to finish up and swap tables.
You would do that one table at a time.
Doc: https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html
Also search around for it; sometimes abbreviated pt-osc.
With a Slave
Investigate the "failover" options and speed with RDS. If you can control the failover, then this method may be fast and 'simple' and possible. I assume it is really Master-Slave, not Master-Master?
With the Slave, leave it replicating from the Master. (Potential problem: RBR may squawk at utf8 replicating to utf8mb4. This needs investigating.)
Do ALTER TABLE ... CONVERT TO ... for each table on the Slave.
Failover.
Depending on what tools RDS has, it may be better to rebuild the Slave(s) from the new Master instead of doing ALTERs on the new Slave (which is the old Master).
Suggest you spin up some tiny instances with the same Master-Slave topology and version and charsets. Then try the steps. It does not need may rows, but it does need FOREIGN KEYs, Triggers, etc, in case any of them cause trouble. (Note how pt-osc has some issues with FKs.)
With phpmyadmin,
Select the database, then you'll get a view of all table structures,
Select all tables,then select Operations at the top menu, you'll find a Collation feature, at the end of the page, select the utf8mb4_unicode_ci,then mark the "Change all tables collations" & "Change all tables columns collations" checkboxes, then click go.
The operation takes utmost 7 seconds, and you are done.
I need to encrypt some data on MySQL slave and using AES_Encrypt for the same. Is it possible to encrypt it while replicating it from the Master database?
Using MySQL 5.6
In MySQL replication, when using binlog_format=STATEMENT, triggers fire on the master and again on the slave, but there is no specific requirement that the triggers be defined identically on the master and slave, or that triggers be provisioned on both.
I say there is no specific requirement, because it is not enforced. If you want the data to be identical on master and slave -- as is almost always the case in a replicated environment -- then yes, the triggers have to be defined identically on both servers... but it is possible to customise the slave triggers -- and one way of doing it would be to define BEFORE INSERT and BEFORE UPDATE triggers on the slave that encrypted the columns before inserts and updates.
This does require STATEMENT replication, which I don't use because it is more delicate and some statements cannot be deterministically replicated in STATEMENT mode.
This would require extra care to ensure that master and slave are always consistent, and would require custom tools since standard table comparison tools would consider master and slave tables to be different.
With binlog_format=ROW, any triggers defined on the slave are ignored for replicated events, because they would be redundant -- the changes done by the triggers on the master will already be replicated to the slave as row events, and executing the triggers on the slave would not have the correct result. Row-based replication uses raw row images, so is always deterministic, unlike statement based replication.
The MariaDB team recognizes that there may be cases where identical data sets are not the objective, such as where you might want to build denormalized query tables on the slave, so in MariaDB 10.1, they introduced a feature to allow slaves to fire triggers for row-based events. This might also be a viable solution, if your master is not using statement-based replication. It does not appear to require the master to run MariaDB, only the slave.
https://mariadb.com/kb/en/mariadb/running-triggers-on-the-slave-for-row-based-events/
It is also possible to create accounts that cannot access all columns, since the MySQL permissions model allows grants to be made at the server, database, table, and column level. Without permission to access a column, you won't see it, in spite of it being there.
Arguably, though, if the data is sensitive, the most correct solution would be to encrypt it on the master.
I am trying to set up mysql replication. I saw both the mentioned options used in different places. I think replicate-rewrite-db is used when the names of the database in the master and the slave are different. Is that the only difference between these two options. Also is there similar option for rewriting/renaming master to slave tables during replication?
They're not really related.
replicate-rewrite-db is used to change the database specified when a query includes a "USE" statement (that is, if a query to the master was "USE foo", it might be translated to "USE bar" in the query executed on the replicant).
replicate-do-table tells MySQL to restrict the replication stream to the specified table(s), so that only queries made against that (or those) table(s) are replicated on the slave.
This article has in-depth details on replication options you can set: http://dev.mysql.com/doc/refman/5.0/en/replication-options-slave.html
I wonder if there is any easy way to keep the scheme consistent in two different MySQL clusters. Apart from classic replication, I would like to have a special "replication" which would reproduce all DDL queries (CREATE, ALTER, DROP, ...) on another cluster (namely the master of that cluster).
I don't need the actual data to be replicated.
Has anyone ever done or tried anything like this?
You can filter replication in MySQL based upon the database in which a query was executed. That doesn't prevent you making changes in other databases, however! So you can do;
USE ddl_repl_db;
ALTER TABLE other_db.foo ADD COLUMN <etc>
This relies on you configuring your servers correctly. I haven't set up MySQL replication for a while, but IIRC you can both filter what you send out from the master for replication and what you accept on the slave.
Old but still high in search.
So, on you DDL replica set all tables engine to BLACKHOLE
I am looking about the best way to implement a multi-sites replication with MySDQL. As I am mainly a MS-SQL user, I feel quite unconfortable with MySQL replication terminology, and the multiple MySQL versions that do not do exactly the same thing the same way.
My objective is, as per SQL terminology, to have a publisher and many subscribers. Suscribers are open to user updates. Changes are replicated with the publisher, which will then distribute these changes to other suscribers.
So my objective is here to determine the correct primary key rule to be used for our tables. As we want exclusively surrogate keys, we have the possibility to use either integer\autoincrement or uuid/uuid_short fields. In order to avoid replication conflicts, the integer\autoincrement does not fits our needs, as it can create replication conflicts when, between two synchronisations, both servers insert a record in the same table. So, according to me, the correct solution to avoid replication\primary key conflicts would be to:
use uuid or uuid-short fields as primary keys
have the corresponding uuid values set by the server at INSERT time
set the replication to RBR (Row Based Replication - sounds equivalent to MS-SQl merge replication) mode, and not SBR (Statement Based Replication - sounds like transactional replication). As I understand it, RBR will insert the calculated uuid value 'as is' in the other servers at replication time, while SBR will recall the uuid() function and generate a new value on each server ... thus causing a major problem!
Will it work?
I think there are two unrelated issues here:
1.
Choosing a primary key in a way which is probably unique - UUID is an acceptable approach. You can use it with SBR or RBR, provided the client app generates the UUID. If you happen to be calling the mysql function to generate it, then you need to use row-based replication. Row-based replication is generally much better, so the only reason you'd want to use statement-based replication is backwards compatibility with a MySQL <5.1 slave, or legacy application which relies on some of its behaviour.
2.
Secondly, you appear to want to do multi-master replication. IT WILL NOT WORK.
MySQL replication is extremely simplistic - it writes changes to a log, pushes them from one server to another, reading the log as needed.
You can't do multi-master replication, because it will fail. No only does MySQL not actually support it (for arbitrary toplologies), but it also doesn't work.
MySQL replication has no "conflict resolution" algorithm, it will simply stop and break if a conflict is discovered.
Even assuming that your database contains NO UNIQUE INDEXES except primary keys which are allocated by UUIDs, your application can still have rules which make multi-master fail.
Any constraints, invariants or assumptions within your application, probably only work if the database has immediate-consistency. Trying to use multi-master replication breaks this assumption and will cause your application to enter unexpected (i.e. normally impossible) states.