I have to send records from Aurora/Mysql to MSK and from there to Elastic search service
Aurora -->Kafka-connect--->AWS MSK--->kafka connect --->Elastic search
The record in Aurora table structure is something like this
I think record will go to AWS MSK in this format.
"o36347-5d17-136a-9749-Oe46464",0,"NEW_CASE","WRLDCHK","o36347-5d17-136a-9749-Oe46464","<?xml version=""1.0"" encoding=""UTF-8"" standalone=""yes""?><caseCreatedPayload><batchDetails/>","CASE",08-JUL-17 10.02.32.217000000 PM,"TIME","UTC","ON","0a348753-5d1e-17a2-9749-3345,MN4,","","0a348753-5d1e-17af-9749-FGFDGDFV","EOUHEORHOE","2454-5d17-138e-9749-setwr23424","","","",,"","",""
So in order to consume by elastic search i need to use proper schema so schema registry i have to use.
My question
Question 1
How should i use schema registry for above type of message schema registry is required ?.
Do i have to create JSON structure for this and if yes where i have keep that.
More help required here to understand this ?
I have edited
vim /usr/local/confluent/etc/schema-registry/schema-registry.properties
Mentioned zookeper but i did not what is kafkastore.topic=_schema
How to link this to custom schema .
Even i started and got this error
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic _schemas not present in metadata after 60000 ms.
Which i was expecting because i did not do anything about schema .
I do have jdbc connector installed and when i start i get below error
Invalid value java.sql.SQLException: No suitable driver found for jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123 for configuration Couldn't open connection to jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123
Invalid value java.sql.SQLException: No suitable driver found for jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123 for configuration Couldn't open connection to jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
Question 2
Can i create two onnector on one ec2 (jdbc and elastic serach one ).If yes do i have to start both in sepearte cli ?
Question 3
When i open vim /usr/local/confluent/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties
I see only propeties value like below
name=test-source-sqlite-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123
mode=incrementing
incrementing.column.name=id
topic.prefix=trf-aurora-fspaudit-
In the above properties file where i can mention schema name and table name?
Based on answer i am updating my configuration for Kafka connect JDBC
---------------start JDBC connect elastic search -----------------------------
wget /usr/local http://packages.confluent.io/archive/5.2/confluent-5.2.0-2.11.tar.gz -P ~/Downloads/
tar -zxvf ~/Downloads/confluent-5.2.0-2.11.tar.gz -C ~/Downloads/
sudo mv ~/Downloads/confluent-5.2.0 /usr/local/confluent
wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz
tar -xzf mysql-connector-java-5.1.48.tar.gz
sudo mv mysql-connector-java-5.1.48 mv /usr/local/confluent/share/java/kafka-connect-jdbc
And then
vim /usr/local/confluent/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties
Then i modified below properties
connection.url=jdbc:mysql://fdgfgdfgrter.us-east-1.rds.amazonaws.com:3306/trf
mode=incrementing
connection.user=admin
connection.password=Welcome123
table.whitelist=PANStatementInstanceLog
schema.pattern=dbo
Last i modified
vim /usr/local/confluent/etc/kafka/connect-standalone.properties
and here i modified below properties
bootstrap.servers=b-3.205147-ertrtr.erer.c5.ertert.us-east-1.amazonaws.com:9092,b-6.ertert-riskaudit.ertet.c5.kafka.us-east-1.amazonaws.com:9092,b-1.ertert-riskaudit.ertert.c5.kafka.us-east-1.amazonaws.com:9092
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/usr/local/confluent/share/java
When i list topic i do not see any topic listed for table name .
Stack trace for the error message
[2020-01-03 07:40:57,169] ERROR Failed to create job for /usr/local/confluent/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties (org.apache.kafka.connect.cli.ConnectStandalone:108)
[2020-01-03 07:40:57,169] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:119)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 2 error(s):
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://****.us-east-1.rds.amazonaws.com:3306/trf
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://****.us-east-1.rds.amazonaws.com:3306/trf
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:116)
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 2 error(s):
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://****.us-east-1.rds.amazonaws.com:3306/trf
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://****.us-east-1.rds.amazonaws.com:3306/trf
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.runtime.AbstractHerder.maybeAddConfigErrors(AbstractHerder.java:423)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:188)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:113)
curl -X POST -H "Accept:application/json" -H "Content-Type:application/json" IPaddressOfKCnode:8083/connectors/ -d '{"name": "emp-connector", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": "1", "connection.url": "jdbc:mysql://IPaddressOfLocalMachine:3306/test_db?user=root&password=pwd","table.whitelist": "emp","mode": "timestamp","topic.prefix": "mysql-" } }'
schema registry is required ?
No. You can enable schemas in json records. JDBC source can create them for you based on the table information
value.converter=org.apache.kafka...JsonConverter
value.converter.schemas.enable=true
Mentioned zookeper but i did not what is kafkastore.topic=_schema
If you want to use Schema Registry, you should be using kafkastore.bootstrap.servers.with the Kafka address, not Zookeeper. So remove kafkastore.connection.url
Please read the docs for explanations of all properties
i did not do anything about schema .
Doesn't matter. The schemas topic gets created when the Registry first starts
Can i create two onnector on one ec2
Yes (ignoring available JVM heap space). Again, this is detailed in the Kafka Connect documentation.
Using standalone mode, you first pass the connect worker configuration, then up to N connector properties in one command
Using distributed mode, you use the Kafka Connect REST API
https://docs.confluent.io/current/connect/managing/configuring.html
When i open vim /usr/local/confluent/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties
First of all, that's for Sqlite, not Mysql/Postgres. You don't need to use the quickstart files, they are only there for reference
Again, all properties are well documented
https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html#connect-jdbc
I do have jdbc connector installed and when i start i get below error
Here's more information about how you can debug that
https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/
As stated before, I would personally suggest using Debezium/CDC where possible
Debezium Connector for RDS Aurora
I'm guessing that you're planning to use AVRO in order to transfer data so don't forget to specify AVROConverter as the default converter when you start up your Kafka Connect workers. If you will use JSON then Schema Registry is not needed.
1.1 kafkastore.topic=_schema
Have you started up your own schema registry? When you start Schema Registry you'll have to specify the "schemas" topic. Basically, this topic will be used by Schema Registry to store the schemas registered by it and in case of a failure, it can recover them from there.
1.2 jdbc connector installed and when i start i get below error
By default, JDBC Connector only works with SQLite and PostgreSQL. If you would like it to work with a MySQL database then you should add the MySQL Driver to the classpath as well.
2.It depends on how you are deploying your Kafka Connect workers. If you go for Distributed mode ( recommended ) then you don't really need separate CLI's. You can deploy your connectors through the Kafka Connect REST API.
3.There is another property called table.whitelist on which you can specify your schemas and tables. e.g: table.whitelistusers,products,transactions
I am trying to generateChangeLog for a db on a percona server, i get the below error when i try to do so.
Starting Liquibase at Wed, 05 Dec 2018 22:34:37 EST (version 3.6.2 built at
2018-07-03 11:28:09)
Unexpected error running Liquibase: liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing for
MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata information
for temporary table 'TMP_XDBOCVCKWHSQYXKP'
liquibase.exception.LiquibaseException:
liquibase.command.CommandExecutionException:
liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing for
MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata information
for temporary table 'TMP_XDBOCVCKWHSQYXKP'
at liquibase.integration.commandline.Main.doMigration(Main.java:1043)
at liquibase.integration.commandline.Main.run(Main.java:191)
at liquibase.integration.commandline.Main.main(Main.java:129)
Caused by: liquibase.command.CommandExecutionException:
liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing
for MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata
information for temporary table 'TMP_XDBOCVCKWHSQYXKP'
I am trying to use it via command line using below statement
liquibase --driver=com.mysql.cj.jdbc.Driver --classpath=C:/liquibase-3.6.2-
bin/jars/mysql-connector-java-8.0.13.jar --changeLogFile=db.changelog-
1.0.xml --url="jdbc:mysql://REMOTE_SERVER_IP:3306/DB_NAME?
autoReconnect=true" --username=USER_NAME --password=PASSWORD --logLevel=info
generateChangeLog
Has anyone encountered this issue before? I tried using the old and new JDBC drivers but did not help.
I hit the same issue. Setting the logLevel to debug you could see that Liquibase was creating and dropping a temporary table. That temporary table seemed to cause the issue.
I was able to work-around this issue by using MySQL Workbench to copy the schema to a regular MySQL instance then running generateChangeLog from there.
I'm trying to convert my MySQL database to PostgreSQL using EnterpriseDB Migration Tool. My setup is as follows:
Windows 10 x64
MySQL 5.5.52 Community Edition GPL
EDB PostgreSQL 9.6.2-2
EnterpriseDB Migration Toolkit 50.0.1
The issue is - it connects successfully to to source MySQL and target PostgreSQL but then fails with the error "unrecognized configuration parameter db_dialect":
C:\Program Files (x86)\edb\mtk\bin>runMTK -sourcedbtype mysql s2c_data
Running EnterpriseDB Migration Toolkit (Build 50.0.1) ...
Source database connectivity info...
conn=jdbc:mysql://localhost:3306/s2c_data?zeroDateTimeBehavior=convertToNull
user =s2c_data_user
password=******
Target database connectivity info...
conn =jdbc:edb://localhost:5432/s2c_data
user =s2c_data_user
password=******
Connecting with source MySQL database server...
Connected to MySQL, version '5.5.52-log'
Connecting with target EDB Postgres database server...
Connected to PostgreSQL, version '9.6.2'
Importing mysql schema s2c_data...
MTK-17003:Error in copy tables: java.sql.SQLException: com.edb.util.PSQLException: ERROR: unrecognized configuration parameter: "db_dialect"
MTK-03000: General Error DB-null: java.sql.SQLException: java.sql.SQLException: java.sql.SQLException: com.edb.util.PSQLException: ERROR: unrecognized configuration parameter: "db_dialect"
One or more schema objects could not be imported during the migration process. Please review the migration output for more details.
Migration logs have been saved to C:\Users\Alex_R\.enterprisedb\migration-toolkit\logs
******************** Migration Summary ********************
Total objects: 0 Successful count: 0 Failed count: 0 Invalid count: 0
*************************************************************
C:\Program Files (x86)\edb\mtk\bin>
Anybody faced the same issue? Seems like a bug in EDB MTK but may be I'm missing something. Would be grateful for any help.
According to EDB's documentation, this is a parameter of their Postgres PLUS fork of PostgreSQL.
So it looks like this migration tool only works when you migrate to their proprietary version of PostgreSQL, not the standard version.
I think you should refer to
this tool's documentation.
According to this document, maybe you can try to add one more command line option:
C:\Program Files (x86)\edb\mtk\bin>runMTK -sourcedbtype mysql -targetdbtype postgresql s2c_data
I found the solution - DBTransfer. The original version was pretty old and buggy so I had to fix few things. The source is here. Feel free to use it for your projects.
I have installed Jre6 update 37 in my product but my JDK version is still 1.5.
Since my product is running as server JVM in old versions(i didn't find server folder under jre/bin), i just copied Client jvm.dll to Server jvm.dll and started service. While connecting to MySql it throws "Cannot run program "33366": CreateProcess error=2, The system cannot find the file specified".
JRE7 breaks some Oracle applications. You'll need to revert to JRE6
Source:
http://oracleformsinfo.wordpress.com/2012/06/17/frm-92095-oracle-jinitiator-version-too-low-disable-jre-1-7-auto-update-for-all-e-business-suite-and-oracle-forms-end-users/
I am utilising Liquibase (www.liquibase.org) into our MVC3 SQL Server 2008 project to manage database migration/changes. However I'm stumbling on the first hurdle: Connecting to Microsoft SQL Server instance.
I am looking at the quick start tutorial on the liquibase site, but exchanging the mysql for sql server DB
I run this command:
liquibase --driver=sqljdbc.jar --changeLogFile="C:\Temp\ChangeLog.xml" --url="jdbc:sqlserver://localhost;databaseName=test" --username=user --password=pass migrate
And receive this error:
Liquibase Update Failed: Cannot find database driver: sqljdbc.jar
I have tried adding --classpath pointing to the sqljdbc driver with no luck.
How can I create or update an MS-SQL Server database with liquibase?
Create a properties file called liquibase.properties containing the following:
classpath=C:\\Program Files\\Microsoft SQL Server 2005 JDBC Driver\\sqljdbc_1.2\\enu\\sqljdbc.jar
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost:1433;databaseName=test
username=myuser
password=mypass
changeLogFile=C:\\Temp\\ChangeLog.xml
liquibase will use this file when located in the same directory. Useful to simplify the command-line.
Database is updated as follows:
liquibase update
Notes:
I'm not a SQL server user, I picked up the JDBC driver and URL details from Microsoft doco
The "migrate" command has been deprecated.