MySQL/MariaDB: selected synchronize of tables between instances - mysql

Question

I have to synchronize some tables from a MySql-Database to another (different server on different machine).
That transfer should only include special tables and only rows of that tables with a special characteristics (e.g. if a column named transfer is set to 1).
And it should be automatically/transparent, fast and work within short cycles (at least every 20s).
I tried different ways but none of them matched all requirements.
DB-synchronize with Galera works fine but does not exclude tables/rows.
mysqldump is not automatically (must be started) and does not exclude.
Is there no other way for that job beside doing it with some own code that runs permanently?

Answer 1

Those partial sync must be performed with specially created scheme.
Possible realization:
Test does your server instances supports The FEDERATED Storage Engine. By default this is allowed.
Test does destination server may access the data stored on source server using CREATE SERVER.
Create server attached to remote source server and needed remote database. Check that remote data is accessible.
On the destination server create an event procedure which is executed each 20s (and make it disabled firstly). I recommend you to create it in separate service database. In this event procedure execute the queries like
SET #event_start_timestamp = CURRENT_TIMESTAMP;
INSERT localDB.local_table ( {columns} )
SELECT {columns}
FROM serviceDB.federated_table
WHERE must_be_transferred
AND created_at < #event_start_timestamp;
UPDATE serviceDB.federated_table
SET must_be_transferred = FALSE
WHERE must_be_transferred
AND created_at < #event_start_timestamp;
Destination server sends according SELECT query to remote source server. It executes this query and sends the output back. Received rows are inserted. Then destination server sends UPDATE which drops the flag.
Enable Event scheduler.
Enable event procedure.
Ensure that your event procedure is executed fast enough. It must finish its work before the next firing. I.e. execute your code as regular stored procedure and check its execution time. Maybe you'd increase sheduling regularity time.
You may exclude such parallel firings using static flag in some service table created in your service database. If it is set (previous event have not finished its work) then event procedure 'd exit. I recommend you to perform this check anycase.
You must handle the next situation:
destination receives a row;
source updates this row;
destination marks this row as synchronized.
Possible solution.
The flag must_be_transferred should be not boolean but (unsigned tiny)integer, with the next values: 0 - not needed in sync; 1 - needed in sync; 2 - selected for copying, 3 - selected for copying but altered after selection.
Algo:
dest. updates the rows marked with non-zero value and set them to 2;
dest. copies the rows using the condition flag OR 2 (or flag IN (2,3));
dest. clears the flag using the expression flag XOR 2 and above condition.
src. marks altered rows as needed in sync using the expression flag OR 1.