plz someone help me resolve this errors. im trying to build an application which will achive cdc for5 this i have used debezium connector
2022-01-25 12:52:02.806 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : including 'world.countrylanguage' among known tables
2022-01-25 12:52:02.806 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : 'world.countrylanguage' is filtered out of capturing
2022-01-25 12:52:02.806 WARN 108 --- [server-snapshot] io.debezium.config.Configuration : Using configuration property "table.whitelist" is deprecated and will be removed in future versions. Please use "table.include.list" instead.
2022-01-25 12:52:02.807 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : snapshot continuing with database(s): []
2022-01-25 12:52:02.808 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Step 6: generating DROP and CREATE statements to reflect current database schemas:
2022-01-25 12:52:04.879 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Step 7: committing transaction
2022-01-25 12:52:05.033 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Step 8: releasing global read lock to enable MySQL writes
2022-01-25 12:52:05.051 INFO 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Writes to MySQL tables prevented for a total of 00:00:04.843
2022-01-25 12:52:05.119 ERROR 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Failed due to error: Aborting snapshot due to error when last running 'UNLOCK TABLES': tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
org.apache.kafka.connect.errors.ConnectException: tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:218) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:846) [debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_281]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_281]
Caused by: java.lang.IllegalAccessError: tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.extractCharset(MySqlAntlrDdlParser.java:403) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.antlr.listener.CreateAndAlterDatabaseParserListener.enterCreateDatabaseOption(CreateAndAlterDatabaseParserListener.java:49) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.ddl.parser.mysql.generated.MySqlParser$CreateDatabaseOptionContext.enterRule(MySqlParser.java:5754) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.antlr.ProxyParseTreeListenerUtil.delegateEnterRule(ProxyParseTreeListenerUtil.java:46) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.enterEveryRule(MySqlAntlrDdlParserListener.java:89) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:41) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:85) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:527) [debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
... 3 common frames omitted
2022-01-25 12:52:05.123 INFO 108 --- [pool-2-thread-1] io.debezium.jdbc.JdbcConnection : Connection gracefully closed
2022-01-25 12:52:05.361 INFO 108 --- [pool-1-thread-1] i.d.connector.common.BaseSourceTask : Stopping down connector
2022-01-25 12:52:05.362 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.MySqlConnectorTask : Stopping MySQL connector task
2022-01-25 12:52:05.362 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.ChainedReader : ChainedReader: Stopping the snapshot reader
2022-01-25 12:52:05.362 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.SnapshotReader : Discarding 0 unsent record(s) due to the connector shutting down
2022-01-25 12:52:05.362 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.SnapshotReader : Discarding 0 unsent record(s) due to the connector shutting down
2022-01-25 12:52:05.363 INFO 108 --- [pool-1-thread-1] i.d.connector.mysql.MySqlConnectorTask : Connector task finished all work and is now shutdown
2022-01-25 12:52:05.364 INFO 108 --- [pool-1-thread-1] o.a.k.c.storage.FileOffsetBackingStore : Stopped FileOffsetBackingStore
2022-01-25 12:52:05.367 ERROR 108 --- [pool-1-thread-1] io.debezium.embedded.EmbeddedEngine : Error while trying to run connector class 'io.debezium.connector.mysql.MySqlConnector'
org.apache.kafka.connect.errors.ConnectException: tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:218) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:846) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_281]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_281]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_281]
Caused by: java.lang.IllegalAccessError: tried to access field com.mysql.cj.CharsetMapping.COLLATION_INDEX_TO_COLLATION_NAME from class io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser
at io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.extractCharset(MySqlAntlrDdlParser.java:403) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.antlr.listener.CreateAndAlterDatabaseParserListener.enterCreateDatabaseOption(CreateAndAlterDatabaseParserListener.java:49) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.ddl.parser.mysql.generated.MySqlParser$CreateDatabaseOptionContext.enterRule(MySqlParser.java:5754) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.antlr.ProxyParseTreeListenerUtil.delegateEnterRule(ProxyParseTreeListenerUtil.java:46) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.enterEveryRule(MySqlAntlrDdlParserListener.java:89) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:41) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28) ~[antlr4-runtime-4.7.2.jar:4.7.2]
at io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:85) ~[debezium-ddl-parser-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:527) ~[debezium-connector-mysql-1.4.2.Final.jar:1.4.2.Final]
... 3 common frames omitted
As per the error looks like the you don't have enough permissions to have this running. What privileges do you have for the debezium user that connects to your mysql database ? Does it have replication client/slave and super privileges ?
d.connector.mysql.SnapshotReader : Writes to MySQL tables prevented for a total of 00:00:04.843 2022-01-25 12:52:05.119 ERROR 108 --- [server-snapshot] i.d.connector.mysql.SnapshotReader : Failed due to error: Aborting snapshot due to error when last running 'UNLOCK TABLES': tried to access field
Related
I have just created the spring boot application from spring initializer and execute it in intellij.But I am facing some error. Can anyone please help me to solve the error?
package com.demo.grocery;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class GroceryApplication {
public static void main(String[] args) {
SpringApplication.run(GroceryApplication.class, args);
}
}
And this is the error I am facing
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.7.2)
2022-07-29 22:12:46.331 INFO 14136 --- [ main] com.demo.grocery.GroceryApplication : Starting GroceryApplication using Java 11.0.13 on DESKTOP-MQ2O3FR with PID 14136 (C:\Users\keerthi\Downloads\grocery\target\classes started by keerthi in C:\Users\keerthi\Downloads\grocery)
2022-07-29 22:12:46.336 INFO 14136 --- [ main] com.demo.grocery.GroceryApplication : No active profile set, falling back to 1 default profile: "default"
2022-07-29 22:12:47.288 INFO 14136 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2022-07-29 22:12:47.306 INFO 14136 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 7 ms. Found 0 JPA repository interfaces.
2022-07-29 22:12:48.437 INFO 14136 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2022-07-29 22:12:48.463 INFO 14136 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2022-07-29 22:12:48.463 INFO 14136 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.65]
2022-07-29 22:12:48.721 INFO 14136 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2022-07-29 22:12:48.721 INFO 14136 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2267 ms
2022-07-29 22:12:48.832 WARN 14136 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataSourceScriptDatabaseInitializer' defined in class path resource [org/springframework/boot/autoconfigure/sql/init/DataSourceInitializationConfiguration.class]: Unsatisfied dependency expressed through method 'dataSourceScriptDatabaseInitializer' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Hikari.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.zaxxer.hikari.HikariDataSource]: Factory method 'dataSource' threw exception; nested exception is org.springframework.boot.autoconfigure.jdbc.DataSourceProperties$DataSourceBeanCreationException: Failed to determine a suitable driver class
2022-07-29 22:12:48.839 INFO 14136 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2022-07-29 22:12:48.868 INFO 14136 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2022-07-29 22:12:48.897 ERROR 14136 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.
Reason: Failed to determine a suitable driver class
Action:
Consider the following:
If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).
Process finished with exit code 1
I have tried most of the solutions from stackoverflow, help me to solve this error
spring.jpa.hibernate.ddl-auto=update
spring.datasource.url=jdbc:mysql://${MYSQL_HOST:localhost}:3306/db_example
spring.datasource.username=springuser
spring.datasource.password=ThePassword
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
#spring.jpa.show-sql: true
Paste this in your application properties file in the resource folder.
Change the database name in the second line and also change the username and password according to your database.
All the domain classes are being mapped as tables in MySQL instead of one domain class named USER (Name of the table is changed to db_user in mapping to avoid keyword violations) as shown below:
And application keeps showing error on startup :
Caused by: java.sql.SQLSyntaxErrorException: Table 'umm.db_user' doesn't exist
Versions of stack:
| Grails Version: 3.2.3
| Groovy Version: 2.4.7
| JVM Version: 1.8.0_242
| Gradle: 3.0
| MySQL: 8.0.26-cluster
| Hibernate: 5.1.1.Final
application.groovy
application.yml
Logs:
2021-09-02 02:43:09.722 ERROR 26370 --- [ main] o.h.engine.jdbc.spi.SqlExceptionHelper : Table 'umm.db_user' doesn't exist
2021-09-02 02:43:09.783 ERROR 26370 --- [ main] grails.app.init.umm1.BootStrap : Exception occured in bootstrap:
org.hibernate.exception.SQLGrammarException: could not extract ResultSet
at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:63)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2122)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1905)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1881)
at org.hibernate.loader.Loader.doQuery(Loader.java:925)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342)
at org.hibernate.loader.Loader.doList(Loader.java:2622)
at org.hibernate.loader.Loader.doList(Loader.java:2605)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2434)
at org.hibernate.loader.Loader.list(Loader.java:2429)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:109)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1787)
at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:363)
at org.grails.orm.hibernate.query.AbstractHibernateQuery.singleResultViaListCall(AbstractHibernateQuery.java:785)
at org.grails.orm.hibernate.query.AbstractHibernateQuery.singleResult(AbstractHibernateQuery.java:775)
Caused by: java.sql.SQLSyntaxErrorException: Table 'umm.db_user' doesn't exist
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:970) at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1020)
at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498)
org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1426) org.apache.tomcat.jdbc.pool.interceptor.StatementDecoratorInterceptor$StatementProxy.invoke(StatementDecoratorInterceptor.java:261)
at com.sun.proxy.$Proxy102.executeQuery(Unknown Source) org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70)
... 134 common frames omitted
Grails application running at http://localhost:9091/umm in environment: development
Do three things,
Change dialect to
org.hibernate.dialect.MySQL8Dialect
and/update mysql driver dependency
mysql:mysql-connector-java:8.0.17
Make sure you are using the correct type in configs (Holders.config.ProfilePictureType) (longtext, maybe)
We have a Java 11 Spring boot app integrated with Kafka/Debezium.
On a Mysql (5.7) database database change is made (user dropped/ tables recreated with data) Debezium's binlogreader starts to read the bin logs, but then logs the following Exception
2021-07-22 10:05:08.584 INFO 24570 --- [debyte.com:3306] i.d.r.history.DatabaseHistoryMetrics : Already applied 10969 database changes
2021-07-22 10:05:10.134 ERROR 24570 --- [debyte.com:3306] i.debezium.connector.mysql.BinlogReader : Error while deserializing binlog event at offset {ts_sec=1626941231, file=mysql-bin.000532, pos=75496603, server_id=2001, event=876}.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=82610388 --stop-position=82618497 --verbose mysql-bin.000532
2021-07-22 10:05:10.135 ERROR 24570 --- [debyte.com:3306] i.debezium.connector.mysql.BinlogReader : Error during binlog processing. Last offset stored = {ts_sec=1626941227, file=mysql-bin.000532, pos=71066122, row=272, server_id=2001, event=518}, binlog reader near position = mysql-bin.000532/82610388
2021-07-22 10:05:10.150 ERROR 24570 --- [debyte.com:3306] i.debezium.connector.mysql.BinlogReader : Failed due to error: Error processing binlog event
org.apache.kafka.connect.errors.ConnectException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1626941231000, eventType=EXT_WRITE_ROWS, serverId=2001, headerLength=19, dataLength=8090, nextPosition=82618497, flags=0}
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230) ~[debezium-connector-mysql-1.0.0.Final.jar:1.0.0.Final]
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207) ~[debezium-connector-mysql-1.0.0.Final.jar:1.0.0.Final]
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:536) ~[debezium-connector-mysql-1.0.0.Final.jar:1.0.0.Final]
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1158) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1005) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connectWithTimeout(BinaryLogClient.java:517) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at com.github.shyiko.mysql.binlog.BinaryLogClient.access$1100(BinaryLogClient.java:90) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:881) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: java.lang.RuntimeException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1626941231000, eventType=EXT_WRITE_ROWS, serverId=2001, headerLength=19, dataLength=8090, nextPosition=82618497, flags=0}
at io.debezium.connector.mysql.BinlogReader.handleServerIncident(BinlogReader.java:604) ~[debezium-connector-mysql-1.0.0.Final.jar:1.0.0.Final]
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:519) ~[debezium-connector-mysql-1.0.0.Final.jar:1.0.0.Final]
... 6 common frames omitted
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1626941231000, eventType=EXT_WRITE_ROWS, serverId=2001, headerLength=19, dataLength=8090, nextPosition=82618497, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:300) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:223) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:239) ~[debezium-connector-mysql-1.0.0.Final.jar:1.0.0.Final]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:984) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
... 4 common frames omitted
Caused by: java.io.EOFException: null
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:190) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at java.base/java.io.InputStream.read(InputStream.java:271) ~[na:na]
at java.base/java.io.InputStream.skip(InputStream.java:531) ~[na:na]
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.skipToTheEndOfTheBlock(ByteArrayInputStream.java:216) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:296) ~[mysql-binlog-connector-java-0.21.0.jar:0.21.0]
... 7 common frames omitted
2021-07-22 10:05:10.150 INFO 24570 --- [debyte.com:3306] i.debezium.connector.mysql.BinlogReader : Error processing binlog event, and propagating to Kafka Connect so it stops this connector. Future binlog events read before connector is shutdown will be ignored.
2021-07-22 10:05:10.152 INFO 24570 --- [debyte.com:3306] i.debezium.connector.mysql.BinlogReader : Stopped reading binlog after 167936 events, last recorded offset: {ts_sec=1626941227, file=mysql-bin.000532, pos=71066122, row=272, server_id=2001, event=518}
2021-07-22 10:05:37.521 INFO 24570 --- [pool-1-thread-1] i.d.connector.mysql.MySqlConnectorTask : Stopping MySQL connector task
This also is logged on application startup.
The suggested mysqlbinlog command:
mysqlbinlog --start-position=82610388 --stop-position=82618497 --verbose mysql-bin.000532
Returns
/*!50530 SET ##SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!50003 SET #OLD_COMPLETION_TYPE=##COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#210722 8:06:26 server id 2001 end_log_pos 123 CRC32 0xc7bff6dc Start: binlog v 4, server v 5.7.19-log created 210722 8:06:26
BINLOG '
Aif5YA/RBwAAdwAAAHsAAAAAAAQANS43LjE5LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAXwAEGggAAAAICAgCAAAACgoKKioAEjQA
Adz2v8c=
'/*!*/;
# at 82610388
#210722 8:07:11 server id 2001 end_log_pos 82618497 CRC32 0xd2561ccc Write_rows: table id 43469
WARNING: The range of printed events ends with a row event or a table map event that does not have the STMT_END_F flag set. This might be because the last statement was not fully written to the log, or because you are using a --stop-position or --stop-datetime that refers to an event in the middle of a statement. The event(s) from the partial statement have not been written to output.
SET ##SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=#OLD_COMPLETION_TYPE*/;
/*!50530 SET ##SESSION.PSEUDO_SLAVE_MODE=0*/;
I have no idea why this is happening or how to code around it. The error is not thrown up to the app.
Has anyone else had a similar issue?
I found a solution on debezium's doc:
What is causing intermittent EventDataDeserializationExceptions with the MySQL connector?
When you run into intermittent deserialization exceptions around 1 minute after starting connector, with a root cause of type EOFException or java.net.SocketException: Connection reset:
Caused by: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializati>onException: Failed to deserialize data of EventHeaderV4{timestamp=1542193955000, eventType=GTID, serverId=91111, headerLength=19, dataLength=46, nextPosition=1058898202, flags=0}
Caused by: java.lang.RuntimeException: com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1542193955000, eventType=GTID, serverId=91111, headerLength=19, dataLength=46, nextPosition=1058898202, flags=0}
Caused by: java.io.EOFException
or
Caused by: java.net.SocketException: Connection reset
Then updating these MySQL server global properties like this will fix it:
set global slave_net_timeout = 120; (default was 30sec)
set global thread_pool_idle_timeout = 120;
Configured MySQL-Debezium for CDC. It was capturing DDL changes like create/Drop table, but not capturing DML events.
Using MySQL 8.0.11 and Embedded debezium version 0.8.3.Final.
No additional configurations were done in MySQL server while creating table.
Configuration bean is created with below code
#Bean
public io.debezium.config.Configuration customerConnector() {
return io.debezium.config.Configuration.create()
.with(EmbeddedEngine.CONNECTOR_CLASS, "io.debezium.connector.mysql.MySqlConnector")
.with(EmbeddedEngine.OFFSET_STORAGE, "org.apache.kafka.connect.storage.FileOffsetBackingStore")
.with(EmbeddedEngine.OFFSET_STORAGE_FILE_FILENAME, "path-to-file")
.with("offset.flush.interval.ms", 60000)
.with(EmbeddedEngine.ENGINE_NAME, "customer-mysql-connector")
.with(MySqlConnectorConfig.SERVER_NAME, databaseServer)
.with(MySqlConnectorConfig.HOSTNAME, databaseServer)
.with(MySqlConnectorConfig.PORT, databasePort)
.with(MySqlConnectorConfig.USER, databaseUser)
.with(MySqlConnectorConfig.PASSWORD, databasePassword)
.with(MySqlConnectorConfig.DATABASE_WHITELIST, databaseSchemaName)
.with(MySqlConnectorConfig.TABLE_WHITELIST, databaseTable)
.with(MySqlConnectorConfig.DATABASE_HISTORY,
MemoryDatabaseHistory.class.getName()).build();
}
Below is the log when starting it as Springboot application
2020-05-29 21:24:28.028 INFO 5576 --- [pool-1-thread-1] i.d.connector.mysql.MySqlConnectorTask : MySQL has the binlog file 'binlog.000009' required by the connector
2020-05-29 21:24:28.072 INFO 5576 --- [pool-1-thread-1] io.debezium.util.Threads : Requested thread factory for connector MySqlConnector, id = localhost named = binlog-client
2020-05-29 21:24:28.074 INFO 5576 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2020-05-29 21:24:28.074 INFO 5576 --- [pool-1-thread-1] io.debezium.util.Threads : Creating thread debezium-mysqlconnector-localhost-binlog-client
2020-05-29 21:24:28.090 INFO 5576 --- [-localhost:3306] io.debezium.util.Threads : Creating thread debezium-mysqlconnector-localhost-binlog-client
2020-05-29 21:24:28.121 INFO 5576 --- [-localhost:3306] c.g.shyiko.mysql.binlog.BinaryLogClient : Connected to localhost:3306 at binlog.000009/3786 (sid:6293, cid:36)
2020-05-29 21:24:28.121 INFO 5576 --- [-localhost:3306] i.debezium.connector.mysql.BinlogReader : Connected to MySQL binlog at localhost:3306, starting at binlog file 'binlog.000009', pos=3786, skipping 8 events plus 0 rows
2020-05-29 21:24:28.121 INFO 5576 --- [-localhost:3306] io.debezium.util.Threads : Creating thread debezium-mysqlconnector-localhost-binlog-client
2020-05-29 21:24:28.183 INFO 5576 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Context refreshed
2020-05-29 21:24:28.199 INFO 5576 --- [ main] d.s.w.p.DocumentationPluginsBootstrapper : Found 1 custom documentation plugin(s)
2020-05-29 21:24:28.199 INFO 5576 --- [ main] s.d.s.w.s.ApiListingReferenceScanner : Scanning for api listing references
Any Clue?
Thanks!
table.whitelist should be set to to <schema>.<table> so in your case source.customer
I am working with a Spring Boot application . I have faced some issue related to org.hibernate.HibernateException: Missing column: auth_password in medxlabpro.appsettings
I have column in my PoJo class authpassword not auth_password. So why this type of issue coming in my application . Anyone can help me please. I have added JPA configuration and Log.
As per my concern i guess it is coming due to configuration mistake.
generate-ddl: false
hibernate:
ddl-auto: validate
:: Spring Boot :: (v1.2.1.RELEASE)
2016-06-21 19:31:52.733 INFO 8196 --- [ main] org.sam.application.Application : Starting Application on DESKTOP-JIBLNR1 with PID 8196 (C:\LabSystem\Scheduler\target\classes started by Sitansu in C:\LabSystem\Scheduler)
2016-06-21 19:31:52.805 INFO 8196 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext#bd4dc25: startup date [Tue Jun 21 19:31:52 IST 2016]; root of context hierarchy
2016-06-21 19:31:53.912 INFO 8196 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.scheduling.annotation.SchedulingConfiguration' of type [class org.springframework.scheduling.annotation.SchedulingConfiguration$$EnhancerBySpringCGLIB$$960de3bc] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2016-06-21 19:31:53.948 INFO 8196 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.scheduling.annotation.ProxyAsyncConfiguration' of type [class org.springframework.scheduling.annotation.ProxyAsyncConfiguration$$EnhancerBySpringCGLIB$$3cdcd8e2] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2016-06-21 19:31:54.435 INFO 8196 --- [ main] j.LocalContainerEntityManagerFactoryBean : Building JPA container EntityManagerFactory for persistence unit 'default'
2016-06-21 19:31:54.463 INFO 8196 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [
name: default
...]
2016-06-21 19:31:54.560 INFO 8196 --- [ main] org.hibernate.Version : HHH000412: Hibernate Core {4.3.7.Final}
2016-06-21 19:31:54.561 INFO 8196 --- [ main] org.hibernate.cfg.Environment : HHH000206: hibernate.properties not found
2016-06-21 19:31:54.563 INFO 8196 --- [ main] org.hibernate.cfg.Environment : HHH000021: Bytecode provider name : javassist
2016-06-21 19:31:54.900 INFO 8196 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {4.0.5.Final}
2016-06-21 19:31:55.677 INFO 8196 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQLDialect
2016-06-21 19:31:55.687 INFO 8196 --- [ main] o.h.e.jdbc.internal.LobCreatorBuilder : HHH000423: Disabling contextual LOB creation as JDBC driver reported JDBC version [3] less than 4
2016-06-21 19:31:56.056 INFO 8196 --- [ main] o.h.h.i.ast.ASTQueryTranslatorFactory : HHH000397: Using ASTQueryTranslatorFactory
2016-06-21 19:31:56.592 INFO 8196 --- [ main] o.h.tool.hbm2ddl.SchemaValidator : HHH000229: Running schema validator
2016-06-21 19:31:56.593 INFO 8196 --- [ main] o.h.tool.hbm2ddl.SchemaValidator : HHH000102: Fetching database metadata
2016-06-21 19:31:56.628 INFO 8196 --- [ main] o.hibernate.tool.hbm2ddl.TableMetadata : HHH000261: Table found: medxlabpro.appsettings
2016-06-21 19:31:56.629 INFO 8196 --- [ main] o.hibernate.tool.hbm2ddl.TableMetadata : HHH000037: Columns: [hl7ftphostname, hl7ftppassword, clientcode, desktopheight1, desktopheight2, newordereligipdfneeded, billingdays, receiptno, defaultpagesize, desktopheight3, postingbatchno, desktopheight4, desktopheight5, id, edibatchno, desktopheight6, datarootfolder, desktopheight7, desktopheight8, claimno, hl7ftpfolder, hl7ftpusername, neworderorderpdfneeded, authpassword, desktoppagesize8, hl7ftpport, desktoppagesize6, neworderresultpdfneeded, desktoppagesize7, desktoppagesize4, desktoppagesize5, versionno, desktoppagesize2, desktoppagesize3, desktoppagesize1, invoiceno]
2016-06-21 19:31:56.636 WARN 8196 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaAutoConfiguration.class]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: Missing column: auth_password in abcdpro.appsettings
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1566)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:956)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:747)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:691)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:321)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:961)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:950)
at org.sam.application.Application.main(Application.java:33)
Caused by: org.hibernate.HibernateException: Missing column: auth_password in abcdpro.appsettings
at org.hibernate.mapping.Table.validateColumns(Table.java:365)
at org.hibernate.cfg.Configuration.validateSchema(Configuration.java:1338)
at org.hibernate.tool.hbm2ddl.SchemaValidator.validate(SchemaValidator.java:155)
at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:525)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1859)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:852)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:845)
at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.withTccl(ClassLoaderServiceImpl.java:398)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:844)
at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:60)
at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:343)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:318)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1625)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1562)
JPA configuration:
jpa:
generate-ddl: false
hibernate:
ddl-auto: validate
database: MYSQL
show-sql: true
How can i fix this anyone can help me please ?
Thanks
Sitansu
By default Spring Boot uses SpringNamingStrategy. This strategy generates underscores in table and field names. So for field authPassword the strategy generates auth_password column name.
If you want to map entity fields to camel case column names you can use org.hibernate.cfg.EJB3NamingStrategy or implement your own.
You can specify a naming strategy in application.properties
spring.jpa.hibernate.naming_strategy=org.hibernate.cfg.EJB3NamingStrategy
I have column in my PoJo class authpassword not auth_password.
Easy solution could be:
add #Column(name="auth_password") to your authpassword field.
Then it should work