I am trying to create multiple connector same datebase, but getting exception
org.apache.kafka.connect.errors.ConnectException: A slave with the same server_uuid/server_id as this slave has connected to the master; the first event 'mysql-bin.000004' at 1088, the last event read from './mysql-bin.000004' at 1310, the last byte read from './mysql-bin.000004' at 1310. Error code: 1236; SQLSTATE: HY000.\n\tat io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
In each connector configuration, the field 'database.server.id' must be unique.
Delete each connector (except one) and recreate it with a new 'database.server.id'.
Related
I wanted to use the new fully managed Blue/Green deployments to upgrade Aurora 1 databases (MySQL 5.6) to Aurora 2 (MySQL 5.7). Though it worked great on one pre-production environment the replication fails on other environments, including production. The replication fails with errors like:
Read Replica Replication Error - SQLError: 1032, reason: Could not execute Delete_rows event on table my_schema.my_table; Can't find record in 'my_table', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin-changelog.000122, end_log_pos 2040
On other instances there were duplicate entries for the primary key (Error code 1062). I tried ROW and MIXED as binlog_format. AWS support's recommendation is to solve those problems manually, which does seem inpractical. For debugging purposes, I tried to read the binlog with the mysqlbinlog utility. This led to the following error:
ERROR: Could not construct log event object: Found invalid event in binary log
Did anybody encounter similar issues? Is there a way to get more insight into these errors and solve them?
I'm trying to create an MySQL 5.7.35 read-only replica on GCP from an external origin. The database is enourmous, with aproximately 800G of data.
I have already ajusted the definer on the triggers, views and functions in a way that GCP accepts (root#%) and thefore the full-dump that the Database Migration Service makes worked. Also got the replication working with the schema of this database (no data).
So far made just one attempt with data. On this attempt the full-dump was sucesful (took 2 days and 10 hours) and failed some time after the replication started with the following error:
2021-09-05T06:09:33.293123Z 2 [ERROR] Slave SQL for channel '': Could not execute Write_rows event on table pacsdb.content_item; Duplicate entry '1441957' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin.000005, end_log_pos 78621021, Error_code: 1062
Selecting this row on the replica returned the same data of the origin (the row was already there).
Since I can't stop slave, skip_counter and start slave or something like that on GCP I have to figure out why this is happening.
My next step would be try to make the dump manualy using the flags that Google recommends.
Someone had a similar problem or have a clue why this is happening?
Any tips are apreciated, thx!
Activating the consistency warnings and GTID-based replication should work. There is information relating replication with Global Transaction Identifiers for MySQL 5.7 here [1].
[1] - https://dev.mysql.com/doc/refman/5.7/en/replication-gtids.html
When I check the status of my debezium connector via the kakfa-connect's REST API, I see this error message for the connector:
org.apache.kafka.connect.errors.ConnectException: The slave is
connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the
master has purged binary logs containing GTIDs that the slave
requires. Error code: 1236; SQLSTATE: HY000.\n\tat
io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)\n\tat
io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:197)\n\tat
io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogReader.java:997)\n\tat
com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:950)\n\tat
com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)\n\tat
com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)\n\tat
java.lang.Thread.run(Thread.java:748)\nCaused by:
com.github.shyiko.mysql.binlog.network.ServerException: The slave is
connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the
master has purged binary logs containing GTIDs that the slave
requires.\n\tat
com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:914)\n\t...
3 more\n
Is this an issue with how I am configuring my debezium connector or an issue with MySQL? Whats crazy is that even when I tried setting the option snapshot.mode to never and this error is still being thrown! According to the documentation, when snapshot.mode is set to either never or when_needed it should not require the GTID so I am super confused as to what is happening
The problem is that Debezium was probably down for some time and some of the transactions it has not seen are no longer available on the server.
That could be an issue with the wrong offsets for the connector.
So I've deleted the connector, deleted all related Kafka topics (like schema history, etc), and cleaned the offsets using the following guide https://debezium.io/documentation/faq/#how_to_remove_committed_offsets_for_a_connector
And it helped! After re-creation - the connector works as expected now.
I have a running debezium setup for doing CDC from MySQL. Now I want to create one more MySQL connector for another MySQL server. But I don't want snapshot for existing data, I want to start the debezium new connector from a specific file and position.
I read some questions from stackoverflow, they told to manually insert the record to connect-offsets topic. But if I do this what will happen to my existing setup?
On a test server, I tried to set the above solution, but it was not working.
kafka-console-producer --broker-list localhost:9092 --topic connect-offsets
>{"file":"mysql-bin.000002","pos":2012}
>[2019-12-30 05:43:52,666] WARN [Producer clientId=console-producer] Got error produce response with correlation id 4 on topic-partition connect-offsets-5. Error: CORRUPT_MESSAGE (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-30 05:43:52,767] WARN [Producer clientId=console-producer] Got error produce response with correlation id 5 on topic-partition connect-offsets-5, Error: CORRUPT_MESSAGE (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-30 05:43:52,870] WARN [Producer clientId=console-producer] Got error produce response with correlation id 6 on topic-partition connect-offsets-5, Error: CORRUPT_MESSAGE (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-30 05:43:52,975] ERROR Error when sending message to topic connect-offsets with key: null, value: 38 bytes with error: (org.apache.kafka.clients.ingCallback)
org.apache.kafka.common.errors.CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, has a null key for a compacted to.
Im not sure how to achieve this. Can somebody help me on this?
Connect-offsets topic records are used for connectors offset management, For Debezium MySQL connector the records in that topic has the key which contains the connector name and the MySQL server name which you have configured under your connector configurations.
So for this, you will require that key as well if you want to produce record in that topic only then the Debezium connector will be able to read those offsets.
I had my debezium mysql source connector working on Kafka. I added another debezium mysql source connector using the same database but with different data formats. As a result, my first connector started showing the following error :
[2019-07-11 10:29:09,125] ERROR WorkerSourceTask{id=debezium-connector-0} Task threw an uncaught and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask:177)
org.apache.kafka.connect.errors.ConnectException: Encountered change event for table db.user whose schema isn't known to this connector
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:208)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:508)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table db.user whose schema isn't known to this connector
at io.debezium.connector.mysql.BinlogReader.informAboutUnknownTableIfRequired(BinlogReader.java:758)
at io.debezium.connector.mysql.BinlogReader.handleUpdateTableMetadata(BinlogReader.java:733)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:492)
... 5 more
[2019-07-11 10:29:09,125] ERROR WorkerSourceTask{id=debezium-
connector-krazybee-0} Task is being killed and will not recover
until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask:178)
[2019-07-11 10:29:09,125] INFO Stopping MySQL connector task
(io.debezium.connector.mysql.MySqlConnectorTask:430)
[2019-07-11 10:29:09,125] INFO ChainedReader: Stopping the binlog
reader (io.debezium.connector.mysql.ChainedReader:121)
[2019-07-11 10:29:09,126] INFO Discarding 0 unsent record(s) due
to the connector shutting down
(io.debezium.connector.mysql.BinlogReader:129)
[2019-07-11 10:29:09,126] INFO Discarding 0 unsent record(s) due to the connector shutting down (io.debezium.connector.mysql.BinlogReader:129)
I have restarted the debezium connector using REST API.
Though I understood to the best of my knowledge that the debezium connector is having a mismatch in database history schema, but unable to figure out how to correct it without deleting the existing connector.
I also reloaded the existing connector with previous values using PUT request but of no use.
I believe you are using the same database.history.kafka.topic for both connectors. You should use unqiue topic for each instance.