Trying to use liquibase on Percona gives error - mysql

I am trying to generateChangeLog for a db on a percona server, i get the below error when i try to do so.
Starting Liquibase at Wed, 05 Dec 2018 22:34:37 EST (version 3.6.2 built at
2018-07-03 11:28:09)
Unexpected error running Liquibase: liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing for
MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata information
for temporary table 'TMP_XDBOCVCKWHSQYXKP'
liquibase.exception.LiquibaseException:
liquibase.command.CommandExecutionException:
liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing for
MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata information
for temporary table 'TMP_XDBOCVCKWHSQYXKP'
at liquibase.integration.commandline.Main.doMigration(Main.java:1043)
at liquibase.integration.commandline.Main.run(Main.java:191)
at liquibase.integration.commandline.Main.main(Main.java:129)
Caused by: liquibase.command.CommandExecutionException:
liquibase.exception.DatabaseException:
liquibase.exception.UnexpectedLiquibaseException: Error during testing
for MySQL/MariaDB JDBC driver bug: could not retrieve JDBC metadata
information for temporary table 'TMP_XDBOCVCKWHSQYXKP'
I am trying to use it via command line using below statement
liquibase --driver=com.mysql.cj.jdbc.Driver --classpath=C:/liquibase-3.6.2-
bin/jars/mysql-connector-java-8.0.13.jar --changeLogFile=db.changelog-
1.0.xml --url="jdbc:mysql://REMOTE_SERVER_IP:3306/DB_NAME?
autoReconnect=true" --username=USER_NAME --password=PASSWORD --logLevel=info
generateChangeLog
Has anyone encountered this issue before? I tried using the old and new JDBC drivers but did not help.

I hit the same issue. Setting the logLevel to debug you could see that Liquibase was creating and dropping a temporary table. That temporary table seemed to cause the issue.
I was able to work-around this issue by using MySQL Workbench to copy the schema to a regular MySQL instance then running generateChangeLog from there.

Related

Debezium CDC Connector task having error : javax.management.InstanceAlreadyExistsException

I had my debezium mysql source connector working on Kafka. I added another debezium mysql source connector using the same database but with different data formats. As a result, my first connector started showing the following error :
[2019-07-11 10:29:09,125] ERROR WorkerSourceTask{id=debezium-connector-0} Task threw an uncaught and unrecoverable exception
(org.apache.kafka.connect.runtime.WorkerTask:177)
org.apache.kafka.connect.errors.ConnectException: Encountered change event for table db.user whose schema isn't known to this connector
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:208)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:508)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.ConnectException: Encountered change event for table db.user whose schema isn't known to this connector
at io.debezium.connector.mysql.BinlogReader.informAboutUnknownTableIfRequired(BinlogReader.java:758)
at io.debezium.connector.mysql.BinlogReader.handleUpdateTableMetadata(BinlogReader.java:733)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:492)
... 5 more
[2019-07-11 10:29:09,125] ERROR WorkerSourceTask{id=debezium-
connector-krazybee-0} Task is being killed and will not recover
until manually restarted
(org.apache.kafka.connect.runtime.WorkerTask:178)
[2019-07-11 10:29:09,125] INFO Stopping MySQL connector task
(io.debezium.connector.mysql.MySqlConnectorTask:430)
[2019-07-11 10:29:09,125] INFO ChainedReader: Stopping the binlog
reader (io.debezium.connector.mysql.ChainedReader:121)
[2019-07-11 10:29:09,126] INFO Discarding 0 unsent record(s) due
to the connector shutting down
(io.debezium.connector.mysql.BinlogReader:129)
[2019-07-11 10:29:09,126] INFO Discarding 0 unsent record(s) due to the connector shutting down (io.debezium.connector.mysql.BinlogReader:129)
I have restarted the debezium connector using REST API.
Though I understood to the best of my knowledge that the debezium connector is having a mismatch in database history schema, but unable to figure out how to correct it without deleting the existing connector.
I also reloaded the existing connector with previous values using PUT request but of no use.
I believe you are using the same database.history.kafka.topic for both connectors. You should use unqiue topic for each instance.

Liquibase MySQL JDBC Driver connectivity error using liquibase.properties

I am currently evaluating Liquibase as a Database Version Control solution. I am running on Ubuntu 16.04 LTS with java version "1.8.0_181". I have installed Liquibase 3.6.2, I have setup two MySQL 5.7 docker containers on ports 4408 and 4409 for testing, and I have downloaded MySQL Connector/J 8.0 which is recommended for use with MySQL Server 5.7. I placed mysql-connector-java-8.0.12.jar in /usr/local/liquibase/lib/ per the liquibase readme:
The "lib" directory is automatically scanned by the liquibase scripts and all .jar files are added to the classpath.
If you have JDBC drivers or Liquibase extensions that you want to be automatically included, add them to this directory.
In my project test directory I have the following liquibase.properties file:
classpath=/usr/local/liquibase/lib/mysql-connector-java-8.0.12.jar
driver=com.mysql.cj.jdbc.Driver
changeLogFile=db.changelog-1.0.xml
url="jdbc:mysql://localhost:4409/myDatabase"
username=myUser
password=myPassword
I want to generate a changelog from the current state of my development database on port 4409 and then (after integrating routines and triggers) run that changelog on an empty database on port 4408, then do a diff to see if the two are identical and then from there perform a diff with the production database to test how well the changes are integrated. So my first step is to run the following command from the same directory that my liquibase.porperties file is:
liquibase generateChangeLog
The result is the following output:
Starting Liquibase at Thu, 23 Aug 2018 07:37:21 PDT (version 3.6.2 built at 2018-07-03 11:28:09)
Unexpected error running Liquibase: liquibase.exception.DatabaseException: Connection could not be created to "jdbc:mysql://localhost:4409/myDatabase" with driver com.mysql.cj.jdbc.Driver. Possibly the wrong driver for the given database URL
liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Connection could not be created to "jdbc:mysql://localhost:4409/myDatabase" with driver com.mysql.cj.jdbc.Driver. Possibly the wrong driver for the given database URL
at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:132)
at liquibase.integration.commandline.Main.doMigration(Main.java:953)
at liquibase.integration.commandline.Main.run(Main.java:191)
at liquibase.integration.commandline.Main.main(Main.java:129)
Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Connection could not be created to "jdbc:mysql://localhost:4409/myDatabase" with driver com.mysql.cj.jdbc.Driver. Possibly the wrong driver for the given database URL
at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:254)
at liquibase.database.DatabaseFactory.openDatabase(DatabaseFactory.java:149)
at liquibase.integration.commandline.CommandLineUtils.createDatabaseObject(CommandLineUtils.java:97)
... 3 common frames omitted
Caused by: liquibase.exception.DatabaseException: Connection could not be created to "jdbc:mysql://localhost:4409/myDatabase" with driver com.mysql.cj.jdbc.Driver. Possibly the wrong driver for the given database URL
at liquibase.database.DatabaseFactory.openConnection(DatabaseFactory.java:249)
... 5 common frames omitted
Note: Since I have included the connector file in the liquibase lib directory, per the quote from the readme above, I shouldn't have to specify the classpath. I have tried it both ways and received the same error. Further, I have tried with mysql-connector-java-5.1.32.jar which is what is in my maven local repo, and adjusted the driver in liquibase properties file to driver=com.mysql.jdbc.Driver, and still the same error.
Now, here is where it gets confusing: If instead of using the liquibase properties file if I execute the following command:
liquibase \
--classpath=/usr/local/liquibase/lib/mysql-connector-java-8.0.12.jar \
--driver=com.mysql.cj.jdbc.Driver \
--changeLogFile=db.changelog-1.0.xml \
--url="jdbc:mysql://localhost:4409/myDatabase" \
--username=myUser \
--password=myPassword \
generateChangeLog
It successfully generates the changelog file with the following output:
Starting Liquibase at Thu, 23 Aug 2018 09:54:41 PDT (version 3.6.2 built at 2018-07-03 11:28:09)
Thu Aug 23 09:54:43 PDT 2018 WARN: Establishing SSL connection without
server's identity verification is not recommended. According to MySQL
5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be
established by default if explicit option isn't set. For compliance
with existing applications not using SSL the verifyServerCertificate
property is set to 'false'. You need either to explicitly disable SSL
by setting useSSL=false, or set useSSL=true and provide truststore for
server certificate verification.
Liquibase command 'generateChangeLog' was executed successfully.
Now, I have other issues that I would like to resolve like why the changelog that is generated includes remarks but when applying the changelog to an empty database the remarks on the first column of every table is not included or when I do a diffChangeLog between the two and try to update the database with the resulting changelog I get and error that changes have validation failures: setColumnRemarks is not supported on mysql, or the the views that the diff generated have replaceIfExists enabled yet the update throws an error that the table already exists. Anyhow, before I post a separate thread for these issues, I thought why not work from the first issue I could not resolve to see if somewhere along the way this may be causing other issues.
So the solution was really simple, I cant believe I didn't see it. In the liquibase.properties file the url should not have quotes. Editing the file to this:
classpath=/usr/local/liquibase/lib/mysql-connector-java-8.0.12.jar
driver=com.mysql.cj.jdbc.Driver
changeLogFile=db.changelog-1.0.xml
url=jdbc:mysql://localhost:4409/myDatabase
username=myUser
password=myPassword
successfully generates the changelog file. It must be the way that the arguments are parse in the properties file vs the command line.

Migrate MySQL database to PostgreSQL using EnterpriseDB Migration Toolkit

I'm trying to convert my MySQL database to PostgreSQL using EnterpriseDB Migration Tool. My setup is as follows:
Windows 10 x64
MySQL 5.5.52 Community Edition GPL
EDB PostgreSQL 9.6.2-2
EnterpriseDB Migration Toolkit 50.0.1
The issue is - it connects successfully to to source MySQL and target PostgreSQL but then fails with the error "unrecognized configuration parameter db_dialect":
C:\Program Files (x86)\edb\mtk\bin>runMTK -sourcedbtype mysql s2c_data
Running EnterpriseDB Migration Toolkit (Build 50.0.1) ...
Source database connectivity info...
conn=jdbc:mysql://localhost:3306/s2c_data?zeroDateTimeBehavior=convertToNull
user =s2c_data_user
password=******
Target database connectivity info...
conn =jdbc:edb://localhost:5432/s2c_data
user =s2c_data_user
password=******
Connecting with source MySQL database server...
Connected to MySQL, version '5.5.52-log'
Connecting with target EDB Postgres database server...
Connected to PostgreSQL, version '9.6.2'
Importing mysql schema s2c_data...
MTK-17003:Error in copy tables: java.sql.SQLException: com.edb.util.PSQLException: ERROR: unrecognized configuration parameter: "db_dialect"
MTK-03000: General Error DB-null: java.sql.SQLException: java.sql.SQLException: java.sql.SQLException: com.edb.util.PSQLException: ERROR: unrecognized configuration parameter: "db_dialect"
One or more schema objects could not be imported during the migration process. Please review the migration output for more details.
Migration logs have been saved to C:\Users\Alex_R\.enterprisedb\migration-toolkit\logs
******************** Migration Summary ********************
Total objects: 0 Successful count: 0 Failed count: 0 Invalid count: 0
*************************************************************
C:\Program Files (x86)\edb\mtk\bin>
Anybody faced the same issue? Seems like a bug in EDB MTK but may be I'm missing something. Would be grateful for any help.
According to EDB's documentation, this is a parameter of their Postgres PLUS fork of PostgreSQL.
So it looks like this migration tool only works when you migrate to their proprietary version of PostgreSQL, not the standard version.
I think you should refer to
this tool's documentation.
According to this document, maybe you can try to add one more command line option:
C:\Program Files (x86)\edb\mtk\bin>runMTK -sourcedbtype mysql -targetdbtype postgresql s2c_data
I found the solution - DBTransfer. The original version was pretty old and buggy so I had to fix few things. The source is here. Feel free to use it for your projects.

Failing to upgrade sonarqube using mysql

I am trying to configure/start SonarQube with MySQL.
I installed everything including sonar DB and sonar user with proper permission.
when I browse 127.0.0.1/sonar, I get 'under maintenance'.
when I browse to /setup to upgrade, I get this message
Database connection cannot be established. Please check database status and JDBC settings.
here is my config
jdbc.default.driverClassName=com.mysql.jdbc.Driver
sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance&useSSL=false
jdbc.default.username=sonar
jdbc.default.password=hpP#ssw0rd
Fail to execute database migration: org.sonar.db.version.v60.AddUuidPathColumnToProjects
java.lang.IllegalStateException: Fail to execute ALTER TABLE projects ADD (uuid_path VARCHAR (1500) NULL)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'uuid_path'
2017.02.15 10:43:05 ERROR web[][o.s.s.p.d.m.PlatformDatabaseMigration] DB Migration or container restart failed. Process ended with an exception
org.jruby.exceptions.RaiseException: (StandardError) An error has occurred, all later migrations canceled:
Fail to execute database migration: org.sonar.db.version.v60.AddUuidPathColumnToProjects

Setting up Liquibase with MS-SQL Server

I am utilising Liquibase (www.liquibase.org) into our MVC3 SQL Server 2008 project to manage database migration/changes. However I'm stumbling on the first hurdle: Connecting to Microsoft SQL Server instance.
I am looking at the quick start tutorial on the liquibase site, but exchanging the mysql for sql server DB
I run this command:
liquibase --driver=sqljdbc.jar --changeLogFile="C:\Temp\ChangeLog.xml" --url="jdbc:sqlserver://localhost;databaseName=test" --username=user --password=pass migrate
And receive this error:
Liquibase Update Failed: Cannot find database driver: sqljdbc.jar
I have tried adding --classpath pointing to the sqljdbc driver with no luck.
How can I create or update an MS-SQL Server database with liquibase?
Create a properties file called liquibase.properties containing the following:
classpath=C:\\Program Files\\Microsoft SQL Server 2005 JDBC Driver\\sqljdbc_1.2\\enu\\sqljdbc.jar
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost:1433;databaseName=test
username=myuser
password=mypass
changeLogFile=C:\\Temp\\ChangeLog.xml
liquibase will use this file when located in the same directory. Useful to simplify the command-line.
Database is updated as follows:
liquibase update
Notes:
I'm not a SQL server user, I picked up the JDBC driver and URL details from Microsoft doco
The "migrate" command has been deprecated.