in my scenario, I need to automate the flyway migration testing in the pipelines.
before starting flyway for executing DB migration I need to import a bak file to init my database structure
how can I import the SQL server backup file to JUnit testContainer?
this is my code:
#Testcontainers
#ActiveProfiles("test-containers-flyway")
class TestFlyway {
#Container
private static final MSSQLServerContainer MY_SQL_CONTAINER = new MSSQLServerContainer().acceptLicense();
#BeforeAll
static void before() {
ClassicConfiguration configuration = new ClassicConfiguration();
configuration.setDataSource(MY_SQL_CONTAINER.getJdbcUrl(), MY_SQL_CONTAINER.getUsername(), MY_SQL_CONTAINER.getPassword());
configuration.setLocations(new Location("classpath:db/migration"));
Flyway flyway = new Flyway(configuration);
flyway.migrate();
}
#Test
void test() throws Exception {
assertTrue(MY_SQL_CONTAINER.isRunning());
Connection connection = DriverManager.getConnection(MY_SQL_CONTAINER.getJdbcUrl(), MY_SQL_CONTAINER.getUsername(), MY_SQL_CONTAINER.getPassword());
Statement statement = connection.createStatement();
ResultSet result = statement.executeQuery("SELECT value FROM table");
while (result.next()) {
String value = result.getString("value");
System.out.println(value);
}
}
}
I think one solution can be create a backup file as SQL flyway db migration and execute it as version 1
Related
I am creating a quick project using R2DBC and H2 to familiarize myself with this new reactive stuff. Made a repository that extends ReactiveCrudRepository and all is well with the world, as long as i use the DatabaseClient to issue a CREATE TABLE statement that matches my entity first...
I understand spring data R2DBC is not as fully featured as spring data JPA (yet?) but is there currently a way to generate the schema from the entity classes?
Thanks
No, there is currently no way to generate schema from entities with Spring Data R2DBC.
I'm using it in a project with Postgres DB and it's complicated to manage database migrations, but I managed to wire in Flyway with synchronous Postgre driver (Flyway doesn't work with reactive drivers yet) at startup to handle schema migrations.
Even though you still have to write your own CREATE TABLE statements which shouldn't be that hard and you could even modify your entities in some simple project to create JPA entities and let Hibernate create schema then copy-paste it into a migration file in your R2DBC project.
It is possible for tests and for production.
I production make sure your user has no access to change schema otherwise you may delete tables by mistake!!! or use a migration tool like flyway.
You need to put your schema.sql in the main resources and add the relevant properties
spring.r2dbc.initialization-mode=always
h2 for test and postgres for prod
I use gradle and the versions of driver are:
implementation 'org.springframework.boot.experimental:spring-boot-actuator-autoconfigure-r2dbc'
runtimeOnly 'com.h2database:h2'
runtimeOnly 'io.r2dbc:r2dbc-h2'
runtimeOnly 'io.r2dbc:r2dbc-postgresql'
runtimeOnly 'org.postgresql:postgresql'
testImplementation 'org.springframework.boot.experimental:spring-boot-test-autoconfigure-r2dbc'
The BOM version is
dependencyManagement {
imports {
mavenBom 'org.springframework.boot.experimental:spring-boot-bom-r2dbc:0.1.0.M3'
}
}
That's how I solved this problem:
Controller:
#PostMapping(MAP + PATH_DDL_PROC_DB) //PATH_DDL_PROC_DB = "/database/{db}/{schema}/{table}"
public Flux<Object> createDbByDb(
#PathVariable("db") String db,
#PathVariable("schema") String schema,
#PathVariable("table") String table) {
return ddlProcService.createDbByDb(db,schema,table);
Service:
public Flux<Object> createDbByDb(String db,String schema,String table) {
return ddl.createDbByDb(db,schema,table);
}
Repository:
#Autowired
PostgresqlConnectionConfiguration.Builder connConfig;
public Flux<Object> createDbByDb(String db,String schema,String table) {
return createDb(db).thenMany(
Mono.from(connFactory(connConfig.database(db)).create())
.flatMapMany(
connection ->
Flux.from(connection
.createBatch()
.add(sqlCreateSchema(db))
.add(sqlCreateTable(db,table))
.add(sqlPopulateTable(db,table))
.execute()
)));
}
private Mono<Void> createDb(String db) {
PostgresqlConnectionFactory
connectionFactory = connFactory(connConfig);
DatabaseClient ddl = DatabaseClient.create(connectionFactory);
return ddl
.execute(sqlCreateDb(db))
.then();
}
Connection Class:
#Slf4j
#Configuration
#EnableR2dbcRepositories
public class Connection extends AbstractR2dbcConfiguration {
/*
**********************************************
* Spring Data JDBC:
* DDL: does not support JPA.
*
* R2DBC
* DDL:
* -does no support JPA
* -To achieve DDL, uses R2dbc.DataBaseClient
*
* DML:
* -it uses R2dbcREpositories
* -R2dbcRepositories is different than
* R2dbc.DataBaseClient
* ********************************************
*/
#Bean
public PostgresqlConnectionConfiguration.Builder connectionConfig() {
return PostgresqlConnectionConfiguration
.builder()
.host("db-r2dbc")
.port(5432)
.username("root")
.password("root");
}
#Bean
public PostgresqlConnectionFactory connectionFactory() {
return
new PostgresqlConnectionFactory(
connectionConfig().build()
);
}
}
DDL Scripts:
#Getter
#NoArgsConstructor(access = AccessLevel.PRIVATE)
public final class DDLScripts {
public static final String SQL_GET_TASK = "select * from tasks";
public static String sqlCreateDb(String db) {
String sql = "create database %1$s;";
String[] sql1OrderedParams = quotify(new String[]{db});
String finalSql = format(sql,(Object[]) sql1OrderedParams);
return finalSql;
}
public static String sqlCreateSchema(String schema) {
String sql = "create schema if not exists %1$s;";
String[] sql1OrderedParams = quotify(new String[]{schema});
return format(sql,(Object[]) sql1OrderedParams);
}
public static String sqlCreateTable(String schema,String table) {
String sql1 = "create table %1$s.%2$s " +
"(id serial not null constraint tasks_pk primary key, " +
"lastname varchar not null); ";
String[] sql1OrderedParams = quotify(new String[]{schema,table});
String sql1Final = format(sql1,(Object[]) sql1OrderedParams);
String sql2 = "alter table %1$s.%2$s owner to root; ";
String[] sql2OrderedParams = quotify(new String[]{schema,table});
String sql2Final = format(sql2,(Object[]) sql2OrderedParams);
return sql1Final + sql2Final;
}
public static String sqlPopulateTable(String schema,String table) {
String sql = "insert into %1$s.%2$s values (1, 'schema-table-%3$s');";
String[] sql1OrderedParams = quotify(new String[]{schema,table,schema});
return format(sql,(Object[]) sql1OrderedParams);
}
private static String[] quotify(String[] stringArray) {
String[] returnArray = new String[stringArray.length];
for (int i = 0; i < stringArray.length; i++) {
returnArray[i] = "\"" + stringArray[i] + "\"";
}
return returnArray;
}
}
It is actually possible to load a schema by defining a specific class in this way:
import io.r2dbc.spi.ConnectionFactory
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.core.io.ClassPathResource
import org.springframework.data.r2dbc.repository.config.EnableR2dbcRepositories
import org.springframework.r2dbc.connection.init.ConnectionFactoryInitializer
import org.springframework.r2dbc.connection.init.ResourceDatabasePopulator
#Configuration
#EnableR2dbcRepositories
class DbConfig {
#Bean
fun initializer(connectionFactory: ConnectionFactory): ConnectionFactoryInitializer {
val initializer = ConnectionFactoryInitializer()
initializer.setConnectionFactory(connectionFactory)
initializer.setDatabasePopulator(
ResourceDatabasePopulator(
ClassPathResource("schema.sql")
)
)
return initializer
}
}
Pay attention that IntelliJ gives an error "Could not autowire. No beans of 'ConnectionFactory' type found" but it is actually a false positive. So ignore it and build again your project.
The schema.sql file has to be put in resources folder.
I am building an workflow for Corda. I want to use the Hikari connection pool library for connecting to MySql database. I AM NOT trying to replace the, ledger H2 database. This database is for storing/retrieving some information, which is not needed in the ledger. I am able to connect to MySql WITHOUT Hikari. However when I use Hikari, I get an error.
java.lang.ClassNotFoundException: com.mysql.cj.jdbc.MysqlConnectionPoolDataSource
I have tested the Hikari code, as a standalone jar file. It works fine. It is a combination of the way corda loads and runs the jar files, inside cordapps directory, which is causing the issue. Since the class is part of the jar. This seems it a little off
I have added the MySql dependency, inline with what is mentioned in https://docs.corda.net/cordapp-build-systems.html#setting-your-dependencies
I am also able to connect to the MySql DB, if I am not using Hikari.
I explored the cordapp jar .
And I could see that the requisite jar is present inside the cordapp jar.
Gradle dependencies for the cordapp
dependencies {
testCompile "junit:junit:$junit_version"
// Corda dependencies.
cordaCompile "$corda_release_group:corda-core:$corda_release_version"
cordaRuntime "$corda_release_group:corda:$corda_release_version"
testCompile "$corda_release_group:corda-node-driver:$corda_release_version"
runtime "mysql:mysql-connector-java:8.0.11"
cordaCompile "com.zaxxer:HikariCP:2.5.1"
// CorDapp dependencies.
cordapp project(":contracts")
}
Sample code
public class DataSource {
private static final Logger logger = LoggerFactory.getLogger(DataSource.class);
private static HikariConfig config = new HikariConfig();
private static HikariDataSource ds;
static {
try {
logger.info("Connecting with connection pool datasource");
config.setDataSourceClassName("com.mysql.cj.jdbc.MysqlConnectionPoolDataSource");
config.addDataSourceProperty("useSSL", "false");
config.addDataSourceProperty("user", "username");
config.addDataSourceProperty("password", "password");
config.addDataSourceProperty("serverName", "localhost");
config.addDataSourceProperty("useSSL", "false");
config.addDataSourceProperty("port",Integer.parseInt("3306"));
config.addDataSourceProperty("cachePrepStmts", "true");
config.addDataSourceProperty("prepStmtCacheSize", "250");
config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048");
config.addDataSourceProperty("requireSSL", "false");
config.addDataSourceProperty("serverTimezone", "UTC");
config.addDataSourceProperty("useServerPrepStmts", "true");
config.addDataSourceProperty("allowPublicKeyRetrieval", "true");
config.addDataSourceProperty("databaseName", "database");
config.setPoolName("Hikari-MySql Pool Name");
logger.error("-- Create Hikari Datasource with config {} --", config);
ds = new HikariDataSource(config);
} catch (Throwable t) {
logger.error("Error Occurred during Datasource Initializaiton", t);
throw t;
}
}
private DataSource() {
}
public static Connection getConnection() throws SQLException {
return ds.getConnection();
}
}
public class MySqlConnection {
static private final Logger logger = LoggerFactory.getLogger(MySqlConnection.class);
public Connection getMySqlConnection() {
Connection conn = null;
try {
conn = DataSource.getConnection();
logger.info("------------> Got conne :: " + conn);
} catch (SQLException e) {
logger.error("SQLException :: " + e);
}
return conn;
}
}
public class DataSourceTest {
static private final Logger logger = LoggerFactory.getLogger(DataSourceTest.class);
public static void main(String[] args) {
MySqlConnection mySqlConnection = new MySqlConnection();
Connection conn = null;
try {
conn = mySqlConnection.getMySqlConnection();
logger.info("------------> Got connection :: " + conn);
Statement statement = conn.createStatement();
ResultSet rs = statement.executeQuery("{some select statement}");
} catch (SQLException e) {
logger.error("SQLException :: " + e);
}
}
}
Exception:
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.MysqlConnectionPoolDataSource
at com.zaxxer.hikari.util.UtilityElf.createInstance(UtilityElf.java:90) ~[HikariCP-2.5.1.jar:?]
at com.zaxxer.hikari.pool.PoolBase.initializeDataSource(PoolBase.java:314) ~[HikariCP-2.5.1.jar:?]
at com.zaxxer.hikari.pool.PoolBase.<init>(PoolBase.java:108) ~[HikariCP-2.5.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:99) ~[HikariCP-2.5.1.jar:?]
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71) ~[HikariCP-2.5.1.jar:?]
If I run the code from a main class inside a jar, it works. But it does not work from inside a cordapp
In MySQL for HikariCP use setJdbcUrl instead of setDataSourceClassName
config.setJdbcUrl("jdbc:mysql://localhost:3306/simpsons");
The MySQL DataSource is known to be broken with respect to network timeout support. Use jdbcUrl configuration instead.
Check your dependency tree. I think you have some collision with "mysql-connector-java" dependency, which cause mess in class loader.
I need to create a database named test in my local mysql server which i will use to setup my datasource bean. I am using the following spring configuration for setting up the datasource and jdbctemplate for my testing
#Configuration
class Config {
#Bean(initMethod = "setupDatabase")
public DataSource getDataSource() {
String url = "jdbc:mysql://localhost:3306/test";
DriverManagerDataSource dataSource = new DriverManagerDataSource(
url, settings.getUsername(), settings.getPassword());
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
return dataSource;
}
#Bean
public JdbcTemplate getJdbcTemplate() {
JdbcTemplate jdbcTemplate = new JdbcTemplate();
jdbcTemplate.setDataSource(((DataSourceTransactionManager)transactionManager()).getDataSource());
return jdbcTemplate;
}
#Bean
public PlatformTransactionManager transactionManager() {
return new DataSourceTransactionManager(getDataSource());
}
#Bean
public DataSourceInitializer dataSourceInitializer(final DataSource dataSource) {
final DataSourceInitializer initializer = new DataSourceInitializer();
initializer.setDataSource(dataSource);
initializer.setDatabasePopulator(databasePopulator());
return initializer;
}
private DatabasePopulator databasePopulator() {
final ResourceDatabasePopulator populator = new ResourceDatabasePopulator();
populator.addScript(schemaScript);
return populator;
}
#PostConstruct
public void setupDatabase() {
String url = "jdbc:mysql://localhost:3306";
try {
Connection connection = DriverManager.getConnection(url, settings.getUsername(), settings.getPassword());
Statement statement = connection.createStatement();
statement.execute("create database test");
} catch (SQLException exception) {
LOGGER.error("Could not setup database for test", exception);
throw new RuntimeException(exception);
}
}
}
I am getting the following error Caused by:
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown database 'test'
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.datasource.init.DatabasePopulatorUtils.execute(DatabasePopulatorUtils.java:46)
Can someone explain what is going wrong with this configuration?
Did you create the test database at your local mysql? You need to create the db first to connect from your application. If you want to create a database by executing sql query connect to the database as root user and create the db.
The source of the exception: your setupDatabase() method is executed actually by Spring AFTER getDataSource() cause #Bean(initMethod = "setupDatabase") forces Spring to execute initMetod AFTER a bean creation for the bean initialization. But it's not you expect here.
You need somehow define your getDataSource() is depended on setupDatabase(). E.g. with a help of #DependsOn annotation.
Also see Spring 3 bean instantiation sequence
P.S. but why you don't simple manually add setupDatabase() call into getDataSource() method?
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
we are building javafx application which will be presenting information about stocks.
Here is the website:
http://analiza.host-ed.me/
But we've got a huge problem. Every free hosting doesn't allow remote mysql connection. And there is my question. When our site is on the server (which i linked) is this remote connection or local connection?
When we put this javafx app as a site it can't connect like it was on the local machine...
Is there any solution? Thanks for help.
(we need to use free hosting, because it's only a school project..)
You can access MySQL from JavaFX. But JavaFX runs on a client and something like php usually runs on a server. You will need a connection from your java app to MySQL. As your hosting provider won't allow you to directly connect to the database port from your Java Client App, you will need some other way to connect.
You could tunnel through port 80, you could run a servlet (or php server code, etc) to intercept incoming traffic and proxy database calls through a HTTP based REST interface or you could install the DB locally on the client.
I'm going to assume, for a school project, it's ok for each client machine to have it's own database. In which case, instead of using MySQL, use a lightweight Java database like H2, bundle it with your app by including it's jar as a dependent library, package the app plus DB jar as a signed WebStart application using the JavaFX packaging tools and host the files generated by the packaging tools at your hosting provider.
Update
Here is a sample application which uses a local H2 database on the client computer.
import java.sql.*;
import java.util.logging.*;
import javafx.application.Application;
import javafx.collections.*;
import javafx.event.ActionEvent;
import javafx.event.EventHandler;
import javafx.scene.Scene;
import javafx.scene.control.*;
import javafx.scene.layout.*;
import javafx.stage.Stage;
public class H2app extends Application {
private static final Logger logger = Logger.getLogger(H2app.class.getName());
private static final String[] SAMPLE_NAME_DATA = { "John", "Jill", "Jack", "Jerry" };
public static void main(String[] args) { launch(args); }
#Override public void start(Stage stage) {
final ListView<String> nameView = new ListView();
final Button fetchNames = new Button("Fetch names from the database");
fetchNames.setOnAction(new EventHandler<ActionEvent>() {
#Override public void handle(ActionEvent event) {
fetchNamesFromDatabaseToListView(nameView);
}
});
final Button clearNameList = new Button("Clear the name list");
clearNameList.setOnAction(new EventHandler<ActionEvent>() {
#Override public void handle(ActionEvent event) {
nameView.getItems().clear();
}
});
VBox layout = new VBox(10);
layout.setStyle("-fx-background-color: cornsilk; -fx-padding: 15;");
layout.getChildren().setAll(
HBoxBuilder.create().spacing(10).children(
fetchNames,
clearNameList
).build(),
nameView
);
layout.setPrefHeight(200);
stage.setScene(new Scene(layout));
stage.show();
}
private void fetchNamesFromDatabaseToListView(ListView listView) {
try (Connection con = getConnection()) {
if (!schemaExists(con)) {
createSchema(con);
populateDatabase(con);
}
listView.setItems(fetchNames(con));
} catch (SQLException | ClassNotFoundException ex) {
logger.log(Level.SEVERE, null, ex);
}
}
private Connection getConnection() throws ClassNotFoundException, SQLException {
logger.info("Getting a database connection");
Class.forName("org.h2.Driver");
return DriverManager.getConnection("jdbc:h2:~/test", "sa", "");
}
private void createSchema(Connection con) throws SQLException {
logger.info("Creating schema");
Statement st = con.createStatement();
String table = "create table employee(id integer, name varchar(64))";
st.executeUpdate(table);
logger.info("Created schema");
}
private void populateDatabase(Connection con) throws SQLException {
logger.info("Populating database");
Statement st = con.createStatement();
int i = 1;
for (String name: SAMPLE_NAME_DATA) {
st.executeUpdate("insert into employee values(i,'" + name + "')");
i++;
}
logger.info("Populated database");
}
private boolean schemaExists(Connection con) {
logger.info("Checking for Schema existence");
try {
Statement st = con.createStatement();
st.executeQuery("select count(*) from employee");
logger.info("Schema exists");
} catch (SQLException ex) {
logger.info("Existing DB not found will create a new one");
return false;
}
return true;
}
private ObservableList<String> fetchNames(Connection con) throws SQLException {
logger.info("Fetching names from database");
ObservableList<String> names = FXCollections.observableArrayList();
Statement st = con.createStatement();
ResultSet rs = st.executeQuery("select name from employee");
while (rs.next()) {
names.add(rs.getString("name"));
}
logger.info("Found " + names.size() + " names");
return names;
}
}
There is a corresponding NetBeans project for this sample which will generate a deployable application. The project can be tested in webstart and applet mode.
For the sample, the database is stored on the user's computer (not the server from which the application was downloaded) and persists between application runs.
The exact location depends on the jdbc connection initialization string. In the case of my sample the database goes in the user's directory jdbc:h2:~/test, which is OS and User specific. In the case of me for Windows it ends up at C:\Users\john_smith\test.h2.db. Using a jdbc connection string such as jdbc:h2:~/test is preferable to a string such as jdbc:h2:C:\\Baza because a string with C:\\ in it is platform specific and won't work well on non-windows systems. For further information on h2 jdbc connection strings refer to the connections settings in the h2 manual.
The h2 system works such that if the database file already exists, it is reused, otherwise a new database file is created. If you modify the database, shut the application down, then load the application again a week later, it is able to read the data created the week before.
I have 480 "Create table" statements to be inserted into an empty access db. I found the access has no option of multiple query executions...
I have all the create table queries in a text file
Please help me, how can this be achieved.
I am using MS Access 2007. The access db is in local harddrive
Thanks
Ramm
I did this java sample. It worked.. Please let me know if any simple process for this.
import java.io.*;
import java.util.*;
import java.text.*;
import java.sql.*;
import org.apache.poi.ss.usermodel.*;
public class DirReader_fat {
public static void main(String[] args) {
String inputFilePath = "D:\\Sample.xlsx";
String strInputQuery = "";
try{
Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");
String filename = "d:\\Empty_1.mdb";
String database = "jdbc:odbc:Driver={Microsoft Access Driver (*.mdb)};DBQ=";
database+= filename.trim() + ";DriverID=22;READONLY=true}"; // add on to the end
// now we can get the connection from the DriverManager
Connection con = DriverManager.getConnection( database ,"","");
// try and create a java.sql.Statement so we can run queries
Statement s = con.createStatement();
InputStream inputStream = new FileInputStream(new File(inputFilePath));
Workbook wb = WorkbookFactory.create(inputStream);
Sheet sheet = wb.getSheet("Sheet1");
for (Row row : sheet) {
strInputQuery = row.getCell(0).toString();
s.execute(strInputQuery);
}
}catch (Exception e) {
e.printStackTrace();
}
}
}
Thanks
Ramm