I have a system where there's a mysql database to which changes are done. Then I have other machines that connect to this mysql database every ten minutes or so, and re-download tables concerning them (for example, one machine might download tables A, B, C, while another machine might download table A, D, E).
Without using Debezium or Kafka, is there a way to get all MySQL changes done after a certain timestamp, so that only those changes are sent to a machine requesting the updates, instead of the whole tables ? ... For example, machine X might want all mysql changes done since it last contacted the mysql database, and then apply those changes to its own old data to update it.
Is there some way to do this ?
MySQL can be setup to automatically replicate databases, tables etc. automatically. If the connection is lost, it will catch up when the connection is restored.
Take a look at this page MySQL V5.5 Replication, or this one MySQL V8.0 Replication
You can use Debezium as a library embedded into your application, if you don't want or can deploy a Kafka cluster.
Alternatively, you could directly use the MySQL Binlog Connector (it's used by the Debezium connector underneath, too), it lets you read the binlog from given offset positions. But then you'd have to deal with many things yourself, which are handled by solutions such as Debezium already, e.g. the correct handling of schema metadata and changes (e.g. additions of columns to existing tables). Usually this involves parsing the MySQL DDL, which by itself is quite complex.
Disclaimer: I'm the lead of Debezium
Related
Google says NO triggers, NO stored procedures, No views. This means the only thing I can dump (or import) is just a SHOW TABLES and SELECT * FROM XXX? (!!!).
Which means for a database with 10 tables and 100 triggers, stored procedures and views I have to recreate, by hand, almost everything? (either for import or for export).
(My boss thinks I am tricking him. He cannot understand how previous, to me, employers did that replication to a bunch of computers using two clicks and I personally need hours (or even days) to do this with an internet giant like Google.)
EDIT:
We have applications which are being created in local computers, where we use our local MySQL. These applications use MySQL DB's which consist, say, from n tables and 10*n triggers. For the moment we cannot even check google-cloud-sql since that means almost everything (except the n almost empty tables) must be "uploaded" by hand. And we cannot also check using google-cloud-sql DB since that means almost everything (except the n almost empty tables) must be "downloaded" by hand.
Until now we do these "up-down"-loads by taking a decent mysqldump from the local or the "cloud" MySQL.
It's unclear what you are asking for. Do you want "replication" or "backups" because these are different concepts in MySQL.
If you want to replicate data to another MySQL instance, you can set up replication. This replication can be from a Cloud SQL instance, or to a Cloud SQL instance using the external master feature.
If you want to backup data to or from the server, checkout these pages on importing data and exporting data.
As far as I understood, you want to Create Cloud SQL Replicas. There are a bunch of replica options found in the doc, use the one that fits the best to you.
However, if you said "replica" as Cloning a Cloud SQL instance, you can follow the steps to clone your instance in a new and independent instance.
Some of these tutorials are done by using the GCP Console and can be scheduled.
I have a need to maintain a copy of an external database (including some additional derived data). With the same set of hardware, which one of the following solutions would give me faster consistency (low lag) with high availability? Assume updates to external database happen at 1000 records per second.
a) Create local mysql replica of an external db using mysql 5.7 replication (binary log file mechanism).
OR
b) Get real time Kafka events from external system, doing HTTP GET to fetch updated object details and use these details to maintain a local mysql replica.
The first will almost certainly give you lower lag (since there are just two systems and not three). Availability is about same - Kafka is high availability, but you have two databases on both sides anyway.
The second is better if you think you'll want to send the data in real-time to additional system. That is:
MySQL1 -> Kafka -> (MySQL2 + Elastic Search + Cassandra + ...)
I hate to answer questions with a 'just use this oddball thing instead' but I do worry you're gearing up a bit too heavy than you may need to -- or maybe you do, and I mis-read.
Consider a gossipy tool like serf.io . It's almost finished, and could give you exactly what you may need with something lighter than a kafka cluster or mysql pair.
Two machines, each running mysql, each synchronized to the other peer-to-peer. I do not want a master db replicated. Rather, I want two users to be able to work on the data offline (each running a mysql server on his machine) and then when reconnected synchronize to each other. Any way to do this with mysql? Any other database I should be looking at to accomplish this better than mysql?
Two-way replication is provided by various database systems (e.g. SQLServer, Sybase etc.) but there are always problems with such a set up.
For example, if the same row is updated at the same time on the two databases, which update wins?
If your aim is to provide a highly-available MySQL database, then there are better options than using replication. MySQL has a clustering solution (though I've not had much success with it) or you can use things like DRBD and heartbeat to provide automatic failover with no loss of data.
If you mean synchronous writing back and forth, this would cause serious data consistency issues. I think you may be referring to MySQL replication, wherein a master server sends its updates to one or more slave database servers, which can be queried.
As for "Other Database Options" SQLServer supports a fairly advanced "replication" process for synchronizing the data between two or more db's. Looks like MySql has something like this as well though.
i has two ubuntu server,and installed Mysql 4.0, ServerA and ServerB
i want to synchronize ServerA's mysql table to ServerB (copy ServerATable to ServerBTable, keep ServerATable and ServerBTable equals)
how to do it?
(shell script,or java, or linux mysql tools)
thanks :)
There's a few options. One is to set up replication within MySQL, which will automatically copy data back and forth between the servers and keep them synchronized automatically, with a granularity of a few seconds as the data flies around. Downside is you have to expose at least the master server to the net to allow TCP connections.
Externally you can do regular mysqldumps on server A, copy to server B, load into mysql, and off you go. This will have a granularity of whatever time interval you run the dump/copy/load sequence in. Downside is that mysqldump locks tables as it's working. If you have a large database, you'll be locked out of serverA's tables while the dump progresses, and the locked out of serverB as the data's loaded. And loading a dump is much slower than doing the dump in the first place, as mysql does all kinds of internal housekeeping (key updates, table metadata updates, etc...) during the load.
For the third option, you could write a synchronizer that compares rows between the two databases and does updates as necessary. However, then you're slurping the contents of one server over to the other and doing row-by-row operations. Plus having to handle any dependent foreign key/child relation updates as well, and this will get complicated in a hurry. At that point you might be better off using the mysqldump option instead.
Of course, you should upgrade your MySQL to something more modern. v4.x is very outdated. You should be running at least a 5.0, preferably a 5.1 version instead.
I'd like to populate the MySQL timezone tables with the database provided by MySQL. I am using a cloud DB and can't overwrite DB tables and restart the server.
Can someone help me understand how to load these files manually?
Rational
I loaded the tz tables from the OS, but the OS has a ton of timezone names. I'd like a more concise set of names that I can query for forms. I think the set provided by MySQL might be a better fit. No other apps are running on the database, thus timezone conflicts aren't an issue.
The database provided by mysql comes as a bunch of myISAM container files; I don't think you're going to be able to safely drop them into the mysql data base directory without bouncing your mysqld.
Do you own this mysqld, or are you one of many tenants in a vendor-owned system?
If you own it, you can load a subset of the /usr/share/zoneinfo time zones. A useful subset might be /usr/share/zoneinfo/posix.
If you're using the mysql.time_zone_name.Name to populate a pick list (a good use for it) you could select an appropriate subset of the admittedly enormous list of names,
or create some aliases right in that table.
I ended up loading the tables into a SQL server on my on local machine, then exporting insert statements and manually loading those onto the server for which I don't have direct control of. Not a glamors solution, it it appears to be the only reasonable way to go about it.