Liquibase MySQL JDBC Driver connectivity error using liquibase.properties - mysql

I am currently evaluating Liquibase as a Database Version Control solution. I am running on Ubuntu 16.04 LTS with java version "1.8.0_181". I have installed Liquibase 3.6.2, I have setup two MySQL 5.7 docker containers on ports 4408 and 4409 for testing, and I have downloaded MySQL Connector/J 8.0 which is recommended for use with MySQL Server 5.7. I placed mysql-connector-java-8.0.12.jar in /usr/local/liquibase/lib/ per the liquibase readme:
The "lib" directory is automatically scanned by the liquibase scripts and all .jar files are added to the classpath.
If you have JDBC drivers or Liquibase extensions that you want to be automatically included, add them to this directory.
In my project test directory I have the following liquibase.properties file:
classpath=/usr/local/liquibase/lib/mysql-connector-java-8.0.12.jar
driver=com.mysql.cj.jdbc.Driver
changeLogFile=db.changelog-1.0.xml
url="jdbc:mysql://localhost:4409/myDatabase"
username=myUser
password=myPassword
I want to generate a changelog from the current state of my development database on port 4409 and then (after integrating routines and triggers) run that changelog on an empty database on port 4408, then do a diff to see if the two are identical and then from there perform a diff with the production database to test how well the changes are integrated. So my first step is to run the following command from the same directory that my liquibase.porperties file is:
liquibase generateChangeLog
The result is the following output:
Starting Liquibase at Thu, 23 Aug 2018 07:37:21 PDT (version 3.6.2 built at 2018-07-03 11:28:09)
Unexpected error running Liquibase: liquibase.exception.DatabaseException: Connection could not be created to "jdbc:mysql://localhost:4409/myDatabase" with driver com.mysql.cj.jdbc.Driver. Possibly the wrong driver for the given database URL
liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Connection could not be created to "jdbc:mysql://localhost:4409/myDatabase" with driver com.mysql.cj.jdbc.Driver. Possibly the wrong driver for the given database URL
at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:132)
at liquibase.integration.commandline.Main.doMigration(Main.java:953)
at liquibase.integration.commandline.Main.run(Main.java:191)
at liquibase.integration.commandline.Main.main(Main.java:129)
Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Connection could not be created to "jdbc:mysql://localhost:4409/myDatabase" with driver com.mysql.cj.jdbc.Driver. Possibly the wrong driver for the given database URL
at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:254)
at liquibase.database.DatabaseFactory.openDatabase(DatabaseFactory.java:149)
at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:97)
... 3 common frames omitted
Caused by: liquibase.exception.DatabaseException: Connection could not be created to "jdbc:mysql://localhost:4409/myDatabase" with driver com.mysql.cj.jdbc.Driver. Possibly the wrong driver for the given database URL
at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:249)
... 5 common frames omitted
Note: Since I have included the connector file in the liquibase lib directory, per the quote from the readme above, I shouldn't have to specify the classpath. I have tried it both ways and received the same error. Further, I have tried with mysql-connector-java-5.1.32.jar which is what is in my maven local repo, and adjusted the driver in liquibase properties file to driver=com.mysql.jdbc.Driver, and still the same error.
Now, here is where it gets confusing: If instead of using the liquibase properties file if I execute the following command:
liquibase \
--classpath=/usr/local/liquibase/lib/mysql-connector-java-8.0.12.jar \
--driver=com.mysql.cj.jdbc.Driver \
--changeLogFile=db.changelog-1.0.xml \
--url="jdbc:mysql://localhost:4409/myDatabase" \
--username=myUser \
--password=myPassword \
generateChangeLog
It successfully generates the changelog file with the following output:
Starting Liquibase at Thu, 23 Aug 2018 09:54:41 PDT (version 3.6.2 built at 2018-07-03 11:28:09)
Thu Aug 23 09:54:43 PDT 2018 WARN: Establishing SSL connection without
server's identity verification is not recommended. According to MySQL
5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be
established by default if explicit option isn't set. For compliance
with existing applications not using SSL the verifyServerCertificate
property is set to 'false'. You need either to explicitly disable SSL
by setting useSSL=false, or set useSSL=true and provide truststore for
server certificate verification.
Liquibase command 'generateChangeLog' was executed successfully.
Now, I have other issues that I would like to resolve like why the changelog that is generated includes remarks but when applying the changelog to an empty database the remarks on the first column of every table is not included or when I do a diffChangeLog between the two and try to update the database with the resulting changelog I get and error that changes have validation failures: setColumnRemarks is not supported on mysql, or the the views that the diff generated have replaceIfExists enabled yet the update throws an error that the table already exists. Anyhow, before I post a separate thread for these issues, I thought why not work from the first issue I could not resolve to see if somewhere along the way this may be causing other issues.

So the solution was really simple, I cant believe I didn't see it. In the liquibase.properties file the url should not have quotes. Editing the file to this:
classpath=/usr/local/liquibase/lib/mysql-connector-java-8.0.12.jar
driver=com.mysql.cj.jdbc.Driver
changeLogFile=db.changelog-1.0.xml
url=jdbc:mysql://localhost:4409/myDatabase
username=myUser
password=myPassword
successfully generates the changelog file. It must be the way that the arguments are parse in the properties file vs the command line.

Related

Kafka connect setup to send record from Aurora using AWS MSK

I have to send records from Aurora/Mysql to MSK and from there to Elastic search service
Aurora -->Kafka-connect--->AWS MSK--->kafka connect --->Elastic search
The record in Aurora table structure is something like this
I think record will go to AWS MSK in this format.
"o36347-5d17-136a-9749-Oe46464",0,"NEW_CASE","WRLDCHK","o36347-5d17-136a-9749-Oe46464","<?xml version=""1.0"" encoding=""UTF-8"" standalone=""yes""?><caseCreatedPayload><batchDetails/>","CASE",08-JUL-17 10.02.32.217000000 PM,"TIME","UTC","ON","0a348753-5d1e-17a2-9749-3345,MN4,","","0a348753-5d1e-17af-9749-FGFDGDFV","EOUHEORHOE","2454-5d17-138e-9749-setwr23424","","","",,"","",""
So in order to consume by elastic search i need to use proper schema so schema registry i have to use.
My question
Question 1
How should i use schema registry for above type of message schema registry is required ?.
Do i have to create JSON structure for this and if yes where i have keep that.
More help required here to understand this ?
I have edited
vim /usr/local/confluent/etc/schema-registry/schema-registry.properties
Mentioned zookeper but i did not what is kafkastore.topic=_schema
How to link this to custom schema .
Even i started and got this error
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic _schemas not present in metadata after 60000 ms.
Which i was expecting because i did not do anything about schema .
I do have jdbc connector installed and when i start i get below error
Invalid value java.sql.SQLException: No suitable driver found for jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123 for configuration Couldn't open connection to jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123
Invalid value java.sql.SQLException: No suitable driver found for jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123 for configuration Couldn't open connection to jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
Question 2
Can i create two onnector on one ec2 (jdbc and elastic serach one ).If yes do i have to start both in sepearte cli ?
Question 3
When i open vim /usr/local/confluent/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties
I see only propeties value like below
name=test-source-sqlite-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:mysql://123871289-eruyre.cluster-ceyey.us-east-1.rds.amazonaws.com:3306/trf?user=admin&password=Welcome123
mode=incrementing
incrementing.column.name=id
topic.prefix=trf-aurora-fspaudit-
In the above properties file where i can mention schema name and table name?
Based on answer i am updating my configuration for Kafka connect JDBC
---------------start JDBC connect elastic search -----------------------------
wget /usr/local http://packages.confluent.io/archive/5.2/confluent-5.2.0-2.11.tar.gz -P ~/Downloads/
tar -zxvf ~/Downloads/confluent-5.2.0-2.11.tar.gz -C ~/Downloads/
sudo mv ~/Downloads/confluent-5.2.0 /usr/local/confluent
wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz
tar -xzf mysql-connector-java-5.1.48.tar.gz
sudo mv mysql-connector-java-5.1.48 mv /usr/local/confluent/share/java/kafka-connect-jdbc
And then
vim /usr/local/confluent/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties
Then i modified below properties
connection.url=jdbc:mysql://fdgfgdfgrter.us-east-1.rds.amazonaws.com:3306/trf
mode=incrementing
connection.user=admin
connection.password=Welcome123
table.whitelist=PANStatementInstanceLog
schema.pattern=dbo
Last i modified
vim /usr/local/confluent/etc/kafka/connect-standalone.properties
and here i modified below properties
bootstrap.servers=b-3.205147-ertrtr.erer.c5.ertert.us-east-1.amazonaws.com:9092,b-6.ertert-riskaudit.ertet.c5.kafka.us-east-1.amazonaws.com:9092,b-1.ertert-riskaudit.ertert.c5.kafka.us-east-1.amazonaws.com:9092
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/usr/local/confluent/share/java
When i list topic i do not see any topic listed for table name .
Stack trace for the error message
[2020-01-03 07:40:57,169] ERROR Failed to create job for /usr/local/confluent/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties (org.apache.kafka.connect.cli.ConnectStandalone:108)
[2020-01-03 07:40:57,169] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:119)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 2 error(s):
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://****.us-east-1.rds.amazonaws.com:3306/trf
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://****.us-east-1.rds.amazonaws.com:3306/trf
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:116)
Caused by: org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector configuration is invalid and contains the following 2 error(s):
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://****.us-east-1.rds.amazonaws.com:3306/trf
Invalid value com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. for configuration Couldn't open connection to jdbc:mysql://****.us-east-1.rds.amazonaws.com:3306/trf
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.runtime.AbstractHerder.maybeAddConfigErrors(AbstractHerder.java:423)
at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:188)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:113)
curl -X POST -H "Accept:application/json" -H "Content-Type:application/json" IPaddressOfKCnode:8083/connectors/ -d '{"name": "emp-connector", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": "1", "connection.url": "jdbc:mysql://IPaddressOfLocalMachine:3306/test_db?user=root&password=pwd","table.whitelist": "emp","mode": "timestamp","topic.prefix": "mysql-" } }'
schema registry is required ?
No. You can enable schemas in json records. JDBC source can create them for you based on the table information
value.converter=org.apache.kafka...JsonConverter
value.converter.schemas.enable=true
Mentioned zookeper but i did not what is kafkastore.topic=_schema
If you want to use Schema Registry, you should be using kafkastore.bootstrap.servers.with the Kafka address, not Zookeeper. So remove kafkastore.connection.url
Please read the docs for explanations of all properties
i did not do anything about schema .
Doesn't matter. The schemas topic gets created when the Registry first starts
Can i create two onnector on one ec2
Yes (ignoring available JVM heap space). Again, this is detailed in the Kafka Connect documentation.
Using standalone mode, you first pass the connect worker configuration, then up to N connector properties in one command
Using distributed mode, you use the Kafka Connect REST API
https://docs.confluent.io/current/connect/managing/configuring.html
When i open vim /usr/local/confluent/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties
First of all, that's for Sqlite, not Mysql/Postgres. You don't need to use the quickstart files, they are only there for reference
Again, all properties are well documented
https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html#connect-jdbc
I do have jdbc connector installed and when i start i get below error
Here's more information about how you can debug that
https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/
As stated before, I would personally suggest using Debezium/CDC where possible
Debezium Connector for RDS Aurora
I'm guessing that you're planning to use AVRO in order to transfer data so don't forget to specify AVROConverter as the default converter when you start up your Kafka Connect workers. If you will use JSON then Schema Registry is not needed.
1.1 kafkastore.topic=_schema
Have you started up your own schema registry? When you start Schema Registry you'll have to specify the "schemas" topic. Basically, this topic will be used by Schema Registry to store the schemas registered by it and in case of a failure, it can recover them from there.
1.2 jdbc connector installed and when i start i get below error
By default, JDBC Connector only works with SQLite and PostgreSQL. If you would like it to work with a MySQL database then you should add the MySQL Driver to the classpath as well.
2.It depends on how you are deploying your Kafka Connect workers. If you go for Distributed mode ( recommended ) then you don't really need separate CLI's. You can deploy your connectors through the Kafka Connect REST API.
3.There is another property called table.whitelist on which you can specify your schemas and tables. e.g: table.whitelistusers,products,transactions

Trying to use liquibase on Percona gives error

I am trying to generateChangeLog for a db on a percona server, i get the below error when i try to do so.
Starting Liquibase at Wed, 05 Dec 2018 22:34:37 EST (version 3.6.2 built at
2018-07-03 11:28:09)
Unexpected error running Liquibase: liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing for
MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata information
for temporary table 'TMP_XDBOCVCKWHSQYXKP'
liquibase.exception.LiquibaseException:
liquibase.command.CommandExecutionException:
liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing for
MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata information
for temporary table 'TMP_XDBOCVCKWHSQYXKP'
at liquibase.integration.commandline.Main.doMigration(Main.java:1043)
at liquibase.integration.commandline.Main.run(Main.java:191)
at liquibase.integration.commandline.Main.main(Main.java:129)
Caused by: liquibase.command.CommandExecutionException:
liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing
for MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata
information for temporary table 'TMP_XDBOCVCKWHSQYXKP'
I am trying to use it via command line using below statement
liquibase --driver=com.mysql.cj.jdbc.Driver --classpath=C:/liquibase-3.6.2-
bin/jars/mysql-connector-java-8.0.13.jar --changeLogFile=db.changelog-
1.0.xml --url="jdbc:mysql://REMOTE_SERVER_IP:3306/DB_NAME?
autoReconnect=true" --username=USER_NAME --password=PASSWORD --logLevel=info
generateChangeLog
Has anyone encountered this issue before? I tried using the old and new JDBC drivers but did not help.
I hit the same issue. Setting the logLevel to debug you could see that Liquibase was creating and dropping a temporary table. That temporary table seemed to cause the issue.
I was able to work-around this issue by using MySQL Workbench to copy the schema to a regular MySQL instance then running generateChangeLog from there.

Migrate MySQL database to PostgreSQL using EnterpriseDB Migration Toolkit

I'm trying to convert my MySQL database to PostgreSQL using EnterpriseDB Migration Tool. My setup is as follows:
Windows 10 x64
MySQL 5.5.52 Community Edition GPL
EDB PostgreSQL 9.6.2-2
EnterpriseDB Migration Toolkit 50.0.1
The issue is - it connects successfully to to source MySQL and target PostgreSQL but then fails with the error "unrecognized configuration parameter db_dialect":
C:\Program Files (x86)\edb\mtk\bin>runMTK -sourcedbtype mysql s2c_data
Running EnterpriseDB Migration Toolkit (Build 50.0.1) ...
Source database connectivity info...
conn=jdbc:mysql://localhost:3306/s2c_data?zeroDateTimeBehavior=convertToNull
user =s2c_data_user
password=******
Target database connectivity info...
conn =jdbc:edb://localhost:5432/s2c_data
user =s2c_data_user
password=******
Connecting with source MySQL database server...
Connected to MySQL, version '5.5.52-log'
Connecting with target EDB Postgres database server...
Connected to PostgreSQL, version '9.6.2'
Importing mysql schema s2c_data...
MTK-17003:Error in copy tables: java.sql.SQLException: com.edb.util.PSQLException: ERROR: unrecognized configuration parameter: "db_dialect"
MTK-03000: General Error DB-null: java.sql.SQLException: java.sql.SQLException: java.sql.SQLException: com.edb.util.PSQLException: ERROR: unrecognized configuration parameter: "db_dialect"
One or more schema objects could not be imported during the migration process. Please review the migration output for more details.
Migration logs have been saved to C:\Users\Alex_R\.enterprisedb\migration-toolkit\logs
******************** Migration Summary ********************
Total objects: 0 Successful count: 0 Failed count: 0 Invalid count: 0
*************************************************************
C:\Program Files (x86)\edb\mtk\bin>
Anybody faced the same issue? Seems like a bug in EDB MTK but may be I'm missing something. Would be grateful for any help.
According to EDB's documentation, this is a parameter of their Postgres PLUS fork of PostgreSQL.
So it looks like this migration tool only works when you migrate to their proprietary version of PostgreSQL, not the standard version.
I think you should refer to
this tool's documentation.
According to this document, maybe you can try to add one more command line option:
C:\Program Files (x86)\edb\mtk\bin>runMTK -sourcedbtype mysql -targetdbtype postgresql s2c_data
I found the solution - DBTransfer. The original version was pretty old and buggy so I had to fix few things. The source is here. Feel free to use it for your projects.

Unable to start mysql in JRE6 update 37

I have installed Jre6 update 37 in my product but my JDK version is still 1.5.
Since my product is running as server JVM in old versions(i didn't find server folder under jre/bin), i just copied Client jvm.dll to Server jvm.dll and started service. While connecting to MySql it throws "Cannot run program "33366": CreateProcess error=2, The system cannot find the file specified".
JRE7 breaks some Oracle applications. You'll need to revert to JRE6
Source:
http://oracleformsinfo.wordpress.com/2012/06/17/frm-92095-oracle-jinitiator-version-too-low-disable-jre-1-7-auto-update-for-all-e-business-suite-and-oracle-forms-end-users/

Setting up Liquibase with MS-SQL Server

I am utilising Liquibase (www.liquibase.org) into our MVC3 SQL Server 2008 project to manage database migration/changes. However I'm stumbling on the first hurdle: Connecting to Microsoft SQL Server instance.
I am looking at the quick start tutorial on the liquibase site, but exchanging the mysql for sql server DB
I run this command:
liquibase --driver=sqljdbc.jar --changeLogFile="C:\Temp\ChangeLog.xml" --url="jdbc:sqlserver://localhost;databaseName=test" --username=user --password=pass migrate
And receive this error:
Liquibase Update Failed: Cannot find database driver: sqljdbc.jar
I have tried adding --classpath pointing to the sqljdbc driver with no luck.
How can I create or update an MS-SQL Server database with liquibase?
Create a properties file called liquibase.properties containing the following:
classpath=C:\\Program Files\\Microsoft SQL Server 2005 JDBC Driver\\sqljdbc_1.2\\enu\\sqljdbc.jar
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost:1433;databaseName=test
username=myuser
password=mypass
changeLogFile=C:\\Temp\\ChangeLog.xml
liquibase will use this file when located in the same directory. Useful to simplify the command-line.
Database is updated as follows:
liquibase update
Notes:
I'm not a SQL server user, I picked up the JDBC driver and URL details from Microsoft doco
The "migrate" command has been deprecated.