Spring Boot & Hibernate: MySQL TEXT with local h2 and Flyway - mysql

I am building a Spring Boot application which has several long texts in it's entities.
To ensure I can handle my database migrations well I included Flyway. In production I'm using a MySQL database, for local testing I want to implement the default h2 database.
An entity might have the following property
#Column(columnDefinition = "TEXT")
val startText: String?
For my MySQL database, this works fine and looks like this in my flyway schema:
start_text TEXT,
When I now start my tests with the default h2 in-memory database in Spring, I receive the following error:
Schema-validation: wrong column type encountered in column [start_text] in table [t_table]; found [clob (Types#CLOB)], but expecting [text (Types#VARCHAR)]
I understand that h2 does not support the MySQL specific type TEXT but actually I have no clue how to fix this.
Any help is appreciated.
Thank you.

I found a workaround for this.
In my application.yaml I have the following:
spring:
flyway:
placeholders:
text-datatype: 'TEXT' #defines a placeholder that is available in flyway
In my application.yaml in my test folder I have the following
spring:
flyway:
placeholders:
text-datatype: 'VARCHAR(255)'
Now I can use the placeholder in my Flyway scripts and it works fine:
start_text ${text-datatype}

Related

JPA Hibernate - Multiple Database Dialects and nvarchar(length) data type

I have to do a project using JPA + Hibernate in which I'm using 3 dialects: MySQL5InnoDBDialect, MSSQL2012Dialect and Oracle12cDialect.
Right now I have a specification which is telling me that for some column from:
Oracle database, I have to use NVARCHAR2(LENGTH) data type
MySql database, I have to use VARCHAR(LENGTH) data type
MSSQL database, I have to use NVARCHAR(LENGTH) data type
... and here is my problem..
If I use:
#Column(name="columnName" length = 255)
private String columnName;
hibernate generates varchar(255) and this is good just for MySQL
If I use:
#Column(name="columnName", columnDefinition="nvarchar2(255)")
private String columnName;
it's not possible in MySQL, i get error because of columnDefinition, but in oracle is okay
I tried to customize MySQL dialect creating
public class CustomMySQL5InnoDBDialect extends MySQL5InnoDBDialect{
public CustomMySQL5InnoDBDialect() {
super();
registerColumnType(Types.NVARCHAR, "nvarchar2($l)");//$l not $1
registerHibernateType(Types.NVARCHAR, StandardBasicTypes.STRING.getName());
}
}
and giving this class in hibernate configuration for MySQL dialect.
I have the same problem in MySQL if I'm using columnDefinition property.
Can you help with this problem please?
The solution is to make use of the feature that the JPA API spec provides you with for just this situation. Define a file orm.xml for each datastore that you need to support, and enable the requisite one when using each database. See this link for details of the file format. That way you don't need to think about hacking the internal features of whichever JPA provider you are using, and you also retain JPA provider portability, as well as database portability
The idea of putting schema specific information info (static) Java annotations is an odd one, even more so when wanting database portability.

For django testing, how do I use keepdb with mariadb

I have a database with a lot of nonmanaged tables which I'm using for a django app. For testing I'm wanting to use the --keepdb option so that I don't have to repopulate these tables every time. I'm using MariaDB for my database. If I don't use the keepdb option everything works fine, the test database gets created and destroyed properly.
But when I try to run the test keeping the database:
$ python manage.py test --keepdb
I get the following error:
Using existing test database for alias 'default'...
Got an error creating the test database: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'CREATE DATABASE IF NOT EXISTS test_livedb ;\n SET sql_note' at line 2")
I assume that this is an issue with a different syntax between MariaDB and MySQL. Is there anyway to get the keepdb option to work with MariaDB?
thanks very much!
For what it's worth: This bug was introduced in Django 2.0.0 and fixed in Django 2.1.3 (https://code.djangoproject.com/ticket/29827)
Two things - check out Factory Boy (for creating test data) and I would suggest checking out Pytest as well. With non-managed tables, the issue I think you'll run into is that (at least in my experience) django won't create them in the test environment and you end up running into issues because there is no migration file to create those tables (since they're unmanaged). Django runs the migration files when creating the test environment.
With Pytest you can run with a --nomigrations flag which builds your test database directly off the models (thus creating the tables you need for your unmanaged models).
If you combine Pytest and Factory Boy you should be able to come up with the ability to setup your test data so it works as expected, is repeatable and testable without issue.
I actually approach it like this (slightly hacky, but it works with our complex setup):
On my model:
class Meta(object):
db_table = 'my_custom_table'
managed = getattr(settings, 'UNDER_TEST', False)
I create the UNDER_TEST variable in settings.py like this:
# Create global variable that will tell if our application is under test
UNDER_TEST = (len(sys.argv) > 1 and sys.argv[1] == 'test')
That way - when the application is UNDER_TEST the model is marked as managed (and Pytest will create the appropriate DB table). Then FactoryBoy handles putting all my test data into that table (either in setUp of the test or elsewhere) so I can test against it.
That's my suggestion - others might have something a little more clear or cleaner.

How to force dialect in Flyway test extensions?

I'm using Flyway test extensions with H2 database and MySQL dialect.
Unfortunately, #FlywayTest annotation executes database clean with H2 dialect and ends with error:
org.flywaydb.core.internal.dbsupport.FlywaySqlException:
Unable to drop "PUBLIC"."site"
------------------------------
SQL State : 42S02
Error Code : 42102
Message : Table "site" not found; SQL statement:
DROP TABLE "PUBLIC"."site" CASCADE [42102-193]
at org.flywaydb.core.internal.dbsupport.SchemaObject.drop(SchemaObject.java:82)
at org.flywaydb.core.internal.dbsupport.h2.H2Schema.doClean(H2Schema.java:69)
at org.flywaydb.core.internal.dbsupport.Schema.clean(Schema.java:148)
at org.flywaydb.core.internal.command.DbClean$4.call(DbClean.java:184)
at org.flywaydb.core.internal.command.DbClean$4.call(DbClean.java:181)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.exec
However, when I manually run DROP TABLE PUBLIC.site CASCADE (no quotes) from console it ends successfully.
How to force dialect in Flyway test extensions?
This is no problem of the annotation #FlywayTest.
The annotation #FlywayTest call flyway.clean() method.
And Flyway supports for H2 only quoted names for schema and tables.
See class source code on Github https://github.com/flyway/flyway/blob/master/flyway-core/src/main/java/org/flywaydb/core/internal/dbsupport/h2/
I think there will be no plan to support H2 feature modes, such like Mysql, Oracle, ....
You can open a issue here https://github.com/flyway/flyway/issues

Alternate solution to save as JSON datatype in postgres spring-boot + eclipselink

I am using eclipselink 2.6 with spring-boot JPA for persistance in postgres.
I am persisting a List of objects as a JSON column in database.Acording to this solution: eclipselink + #convert(json) + postgres + list property
I am able to save the data in postgres.
When the column is null, I get this exception:
Caused by: org.postgresql.util.PSQLException: ERROR: column "sample_column" is of type json but expression is of type character varying
Hint: You will need to rewrite or cast the expression.
I can solve this issue by this answer:
Writing to JSON column of Postgres database using Spring / JPA
Q1: Is there an alternate solution other than setting this property stringtype=unspecified int url spring.datasource.url=jdbc:postgresql://localhost:5432/dbnam‌​e?stringtype=unspeci‌​fied
Q2: If not, How can I set stringtype=unspecified in application.prooerties of spring-boot rather than embedding it in the spring.datasource.url
The answer is yes, but is implementation-specific.
For example, in Tomcat, this attribute is called connectionProperties, you would therefore write:
spring.datasource.tomcat.connection-properties: stringtype=unspecified
From the Spring-Boot documentation.
It is also possible to fine-tune implementation-specific settings using their respective prefix (spring.datasource.tomcat., spring.datasource.hikari., spring.datasource.dbcp.* and spring.datasource.dbcp2.*). Refer to the documentation of the connection pool implementation you are using for more details.
If you are using spring-boot 1.4.1 or above,
add a data.sql file in resources folder,
-- IN H2 database, create a column as 'OTHER' data type,
-- if H2 fails to create a column as 'JSON' data type.
CREATE DOMAIN IF NOT EXISTS JSON AS OTHER;
This .sql file will be executed during startup of your application and will create a column with data type others in table for the 'json' data type columns in H2 database.

Spring-Session with JDBC configuration: Table 'test.spring_session' doesn't exist

I try to run this example but without using Redis, instead with my local MySQL server.
I have edited this spring boot app like this:
Gradle:
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath("org.springframework.boot:spring-boot-gradle-plugin:$springBootVersion")
}
}
apply plugin: 'spring-boot'
apply from: JAVA_GRADLE
//this 'if' statement is because I was getting error: Execution failed for task ':samples:findbyusername:findMainClass'.
//> Could not find property 'main' on task ':samples:findbyusername:run'.
if (!hasProperty('mainClass')) {
ext.mainClass = 'sample.FindByUsernameApplication'
}
tasks.findByPath("artifactoryPublish")?.enabled = false
group = 'samples'
dependencies {
compile("org.springframework.boot:spring-boot-starter-jdbc:$springBootVersion")
compile group: 'mysql', name: 'mysql-connector-java', version: '6.0.2'
compile group: 'org.springframework.session', name: 'spring-session', version: '1.2.0.RELEASE'
compile project(':spring-session'),
"org.springframework.boot:spring-boot-starter-web",
"org.springframework.boot:spring-boot-starter-thymeleaf",
"nz.net.ultraq.thymeleaf:thymeleaf-layout-dialect",
"org.springframework.security:spring-security-web:$springSecurityVersion",
"org.springframework.security:spring-security-config:$springSecurityVersion",
"com.maxmind.geoip2:geoip2:2.3.1",
"org.apache.httpcomponents:httpclient"
testCompile "org.springframework.boot:spring-boot-starter-test",
"org.assertj:assertj-core:$assertjVersion"
integrationTestCompile gebDependencies,
"org.spockframework:spock-spring:$spockVersion"
}
def reservePort() {
def socket = new ServerSocket(0)
def result = socket.localPort
socket.close()
result
}
application.properties
spring.datasource.url = jdbc:mysql://localhost:3306/TEST?characterEncoding=UTF-8&useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC
spring.datasource.driverClassName = com.mysql.jdbc.Driver
spring.datasource.username=root
spring.datasource.password=but
spring.thymeleaf.cache=false
spring.template.cache=false
HttpSessionConfig.java
#EnableJdbcHttpSession // <1>
public class HttpSessionConfig {
}
Application starts on tomcat but when I hit localhost in my browser I get:
Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.
Mon May 23 21:14:31 CEST 2016
There was an unexpected error (type=Internal Server Error, status=500).
PreparedStatementCallback; bad SQL grammar [INSERT INTO SPRING_SESSION(SESSION_ID, CREATION_TIME, LAST_ACCESS_TIME, MAX_INACTIVE_INTERVAL, PRINCIPAL_NAME) VALUES (?, ?, ?, ?, ?)]; nested exception is java.sql.SQLSyntaxErrorException: Table 'test.spring_session' doesn't exist
I don't remember reading anything about manually creating this table so I assumed that spring will handle it for me...
EDIT:
I actually tried to manually create tables and then application runs OK. But I guess I shouldn't be doing this manually.
Spring Session ships with database schema scripts for most major RDBMS's (located in org.springframework.session.jdbc package), but the creation of database tables for Spring Session JDBC supports needs to be taken care of by the users themselves.
The provided scripts can be used untouched, however some users may choose to modify them to fit their specific needs, using the provided scripts as a reference.
An option would be to use a database migration tool, such as Flyway, to handle the creation of database tables.
Since you're using Spring Boot, it might be of your interest that there is a pending PR to add support for automatic initialization of Spring Session JDBC schema: https://github.com/spring-projects/spring-boot/pull/5879
If the documentation misled you into thinking the tables should be created automatically by Spring Session itself, consider reporting the issue so we can update the documentation if necessary: https://github.com/spring-projects/spring-session/issues
At least as of now, you don't have to create tables manually.
In my test, tables were created automatically after adding the following line into the file application.properties when this file appears like shown above.
spring.session.jdbc.initialize-schema=always
I found this beautiful line from the following stackoverflow page.
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'test.spring_session' doesn't exist - Spring Boot
I am sorry that I don't know if it was necessary to create tables manually in 2016 or 2017. I will update this answer when I get to know this or get to have some more fruitful related information. I am just wishing that nobody will be led to an idea that automatic creation of session tables is impossible with the lastest Spring Framework version of 2019 or later.
spring.session.jdbc.initialize-schema=always worked for me