I get a connection from c3p0 pool for batch insert and i set the autoCommit=false,after using,i put it back to the poll. If the autoCommit=false still valid for this connection?
No. According the the JDBC spec, new Connections are autocommit=true, and c3p0 implements transparent Connection pooling, meaning an application that functions correctly in a c3p0 application should also function correctly using an unpooled Connection source.
If you wish, you can override this behavior by defining a ConnectionCustomizer. Something like...
package mypkg;
import com.mchange.v2.c3p0.AbstractConnectionCustomizer;
import java.sql.Connection;
public class AutocommitConnectionCustomizer extends AbstractConnectionCustomizer {
#Override
public void onCheckOut(Connection c, String parentDataSourceIdentityToken) throws Exception {
c.setAutoCommit( false );
}
}
Then, in your config...
c3p0.connectionCustomizerClassName=mypkg.AutocommitConnectionCustomizer
I hope this helps!
Related
I'm using Pomelo Entity framework core with MySqlConnector in my asp.net 5 app to connect to my MySql database - using custom DbContext classes. This normally works fine.
However, I have a need to connect to another database than the one in the connection string (for instance 'INFORMATION_SCHEMA').
I can of course change the connection string, replacing the database name, but:
That creates an extra connection pool - one per connection string!
I'm trying to avoid that - having only one connection pool per website.
I was messing around with 'SetDefaultSchema' and other attempts that all fail miserably.
How can I change the database name, the DbContext uses so I only have one connection pool and still each DbContext has its own database to connect to?
The solution is actually quite simple: Use connection interceptor (available from Entity Framework Core 3.0+).
The code below switches the database after the connection has been opened.
Now each DbContext class can use its own database and with only one connection pool in use.
First you create an interceptor class inherited from DbConnectionInterceptor. The constructor takes the database name, you want to switch to, as parameter:
using Microsoft.EntityFrameworkCore.Diagnostics;
using System.Data.Common;
using System.Threading.Tasks;
public class MySqlConnectionInterceptor : DbConnectionInterceptor
{
public MySqlConnectionInterceptor(string databaseName)
{
database = databaseName;
}
readonly string database;
public override void ConnectionOpened(DbConnection connection, ConnectionEndEventData eventData)
{
if (database != null)
{
connection.ChangeDatabase(database); // The 'magic' code
}
base.ConnectionOpened(connection, eventData);
}
public override async Task ConnectionOpenedAsync(DbConnection connection, ConnectionEndEventData eventData, CancellationToken cancellationToken = default)
{
if (database != null)
{
await connection.ChangeDatabaseAsync(database); // The 'magic' code
}
await base.ConnectionOpenedAsync(connection, eventData, cancellationToken);
}
}
Now all you have to is include one line in your DbContext class's OnConfiguring method:
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.AddInterceptors(new MySqlConnectionInterceptor("yourDatabase"));
}
Now the connection will switch to 'yourDatabase' database every time, it's opened.
And it will only use one connection pool (total)! That way the number of 'sleeping' connections are kept at a minimum.
This works because Pomelo Entity Framework Core always resets a connection before reusing it from the pool (unless you specifically sets 'Connectionreset=false' - which is bad anyway). It sets the database back to the one in the connection string, which you of course can override again).
Of course you don't have to hard code the database name. If you for instance use a base DbContext class, that your other DbContexts inherits from, you can create a constructor that takes the database name as parameter, like this:
public class BaseDbContext : DbContext
{
public BaseDbContext (string databaseName)
{
database = databaseName;
}
string database;
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
optionsBuilder.AddInterceptors(new MySqlConnectionInterceptor(database));
}
}
The code has been tested in Asp.Net 5+6 and .Net Windows Forms.
I have a docker-compose setup to start my SpringBoot application and a MySQL database. If the database starts first, then my application can connect successfully. But if my application starts first, no database exists yet, so the application throws the following exception and exits:
app_1 | 2018-05-27 14:15:03.415 INFO 1 --- [ main]
com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
app_1 | 2018-05-27 14:15:06.770 ERROR 1 --- [ main]
com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization
app_1 | com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure
I could edit my docker-compose file to make sure the database is always up before the application starts up, but I want the application to be able to handle this case on its own, and not immediately exit when it cannot reach the database address.
There are ways to configure the datasource in the application.properties file to make the application reconnect to the database, as answered here and here. But that doesn't work for a startup connection to the datasource.
How can I make my SpringBoot application retry the connection at startup to the database at a given interval until it successfully connects to the database?
Set HikariCP's initializationFailTimeout property to 0 (zero), or a negative number. As documented here:
⌚initializationFailTimeout
This property controls whether the pool will "fail fast" if the pool cannot be seeded with an initial connection successfully. Any positive number is taken to be the number of milliseconds to attempt to acquire an initial connection; the application thread will be blocked during this period. If a connection cannot be acquired before this timeout occurs, an exception will be thrown. This timeout is applied after the connectionTimeout period. If the value is zero (0), HikariCP will attempt to obtain and validate a connection. If a connection is obtained, but fails validation, an exception will be thrown and the pool not started. However, if a connection cannot be obtained, the pool will start, but later efforts to obtain a connection may fail. A value less than zero will bypass any initial connection attempt, and the pool will start immediately while trying to obtain connections in the background. Consequently, later efforts to obtain a connection may fail. Default: 1
There is an alternative way to do this, which doesn't rely on a specific Connection Pool library or a specific database. Note that you will need to use spring-retry to achieve the desired behaviour with this approach
First you need to add spring-retry to your dependencies :
<dependency>
<groupId>org.springframework.retry</groupId>
<artifactId>spring-retry</artifactId>
<version>${spring-retry.version}</version>
</dependency>
Then you can create a decorator over DataSource that will extends AbstractDataSource like bellow :
#Slf4j
#RequiredArgsConstructor
public class RetryableDataSource extends AbstractDataSource {
private final DataSource dataSource;
#Override
#Retryable(maxAttempts = 5, backoff = #Backoff(multiplier = 1.3, maxDelay = 10000))
public Connection getConnection() throws SQLException {
log.info("getting connection ...");
return dataSource.getConnection();
}
#Override
#Retryable(maxAttempts = 5, backoff = #Backoff(multiplier = 2.3, maxDelay = 10000))
public Connection getConnection(String username, String password) throws SQLException {
log.info("getting connection by username and password ...");
return dataSource.getConnection(username, password);
}
}
Then you will need to inject this custom DataSource decorator into Spring context by creating a custom BeanPostProcessor :
#Slf4j
#Order(value = Ordered.HIGHEST_PRECEDENCE)
#Component
public class RetryableDatabasePostProcessor implements BeanPostProcessor {
#Override
public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
if(bean instanceof DataSource) {
log.info("-----> configuring a retryable datasource for beanName = {}", beanName);
return new RetryableDataSource((DataSource) bean);
}
return bean;
}
#Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
return bean;
}
}
Last but not least you will need to enable Spring retry by adding #EnableRetry annotation to spring main class, example :
#EnableRetry
#SpringBootApplication
public class RetryableDbConnectionApplication {
public static void main(String[] args) {
SpringApplication.run(RetryableDbConnectionApplication.class, args);
}
}
I am trying to use flyway to create and manage a MySQL database. Here is the code i have got so far.
FlywayMigration.java : Class that applys the migration
public class FlywayMigration
{
public FlywayMigration(DatabaseConfiguration configuration, Flyway flyway)
{
flyway.setDataSource(configuration.getDataSource());
flyway.migrate();
}
public static void main(String[] args)
{
new FlywayMigration(new DatabaseConfiguration("database.properties"), new Flyway());
}
}
DatabaseConfiguration.java : Configuration class, this class will configure the datasource to be applyed to the Flyway.setDataSource method
public class DatabaseConfiguration
{
private final Logger LOGGER = LoggerFactory.getLogger(this.getClass());
private PropertiesUtil prop = null;
public DatabaseConfiguration(String file)
{
prop = new PropertiesUtil(file);
}
public String getDataSourceClass()
{
return prop.getProperty("mysql.data.source.class");
}
public String getURL ()
{
return prop.getProperty("mysql.url");
}
public String getHostName()
{
return prop.getProperty("mysql.host.name");
}
public String getDatabaseName()
{
return prop.getProperty("mysql.database.name");
}
public DataSource getDataSource()
{
MysqlDataSource dataSource = new MysqlDataSource();
dataSource.setURL(getURL());
dataSource.setUser(prop.getProperty("mysql.user.name"));
dataSource.setPassword(null);
return dataSource;
}
}
database.properties is the file where i store the database information, password can be null
mysql.data.source.class=com.mysql.jdbc.Driver
mysql.url=jdbc:mysql://localhost:3306/vmrDB
mysql.host.name=localhost
mysql.database.name=vmrDB
mysql.user.name=root
And i get the folowing error in my trace
Exception in thread "main" org.flywaydb.core.api.FlywayException: Unable to obtain Jdbc connection from DataSource
at org.flywaydb.core.internal.util.jdbc.JdbcUtils.openConnection(JdbcUtils.java:56)
at org.flywaydb.core.Flyway.execute(Flyway.java:1144)
at org.flywaydb.core.Flyway.migrate(Flyway.java:811)
at com.bt.sitb.vmr.migration.FlywayMigration.<init>(FlywayMigration.java:10)
at com.bt.sitb.vmr.migration.FlywayMigration.main(FlywayMigration.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Can someone please tell me why the DataSource from MySQL is not connecting?
It looks like Flyway cannot connect to the database.
One reason for this is that the database in the database URL does not exist.
Question: does your database schema exist?
If your answer is no, then:
connect to jdbc:mysql://localhost:3306/mysql
also specify the schema to use for migration with flyway.setSchemas(configuration.getDatabaseName())
you also need flyway.init() before you can initialize migration of your database.
Ran into this same issue. Apparently, the problem was with my .properties file. The jar was using the one packaged with it and not the external one. So I moved my external properties file out of the resources folder and into the root directory of the jar and problem solved!
Hope this helps someone.
I had this same issue when working on a Java application in Debian 10 using Tomcat Application server.
I defined the connection strings for the database in the context.xml file, however, when I start out the application and try to log into the application, I get the error:
Exception in thread "main" org.flywaydb.core.api.FlywayException: Unable to obtain Jdbc connection from DataSource
at org.flywaydb.core.internal.util.jdbc.JdbcUtils.openConnection(JdbcUtils.java:56)
at org.flywaydb.core.Flyway.execute(Flyway.java:1144)
Here's what I figured out:
I finally realized that the application was using internally defined database connection strings that were packaged with it. The internally defined database connection strings were different from my own database connection strings defined in the context.xml file.
The solution for me was to either modify the internally defined database connection strings that were packaged with the application or use the same internally defined database connection strings that were packaged with application in my context.xml file.
That's all.
I hope this helps.
I have heard alot of success integration story when comes to Apache Camel with Fuse. HEnce. here Im just starting to explore the Fuse IDE, with just a simple task on top of my head, i would like to achieve:
Read a fix length file
Parse the fix length file
persist it to mysql database table
I am only able to get as far as:
Read the fix length file (with Endpoint "file:src/data/Japan?noop=true")
Define a Marshal with Bindy and Define a POJO package model with #FixedLengthRecord annotation
then i am stuck... HOW TO persist the POJO into mysql database table? I can see some JDBC, IBatis and JPA end point, but how to accomplish that in Fuse IDE?
My POJO package:
package com.mbww.model;
import org.apache.camel.dataformat.bindy.annotation.DataField;
import org.apache.camel.dataformat.bindy.annotation.FixedLengthRecord;
#FixedLengthRecord(length=91)
public class Japan {
#DataField(pos=1, length=10)
private String TNR;
#DataField(pos=11, length=10)
private String ATR;
#DataField(pos=21, length=70)
private String STR;
}
Well you can use all of the following components to actually read and write from the database:
JDBC
IBATIS
MyBATIS
SPRING-JDBC
SQL
Custom Processor
I am going to show you how to use the custom processor to insert the rows into a table. The main reason for this is that you will get to work with the messages and exchange and this will give you more of a insight into Camel. All of the other components can be used by following the documentation on the camel site.
So lets review what you have. You are reading the file and converting the body to a bindy object. So for each line in your text file Camel will send a bindy object of class com.mbww.model.JAPAN to the next end point. This next end point needs to talk to the database. There is one problem I can spot immediately you are using a marshal you should be using a unmarshal.
The documentation clearly states: If you receive a message from one of the Camel Components such as File, HTTP or JMS you often want to unmarshal the payload into some bean so that you can process it using some Bean Integration or perform Predicate evaluation and so forth. To do this use the unmarshal word in the DSL in Java or the Xml Configuration.
Your bindy class looks good but it is missing getters and setters modify the class to look like this:
package com.mbww.model;
import org.apache.camel.dataformat.bindy.annotation.DataField;
import org.apache.camel.dataformat.bindy.annotation.FixedLengthRecord;
#FixedLengthRecord(length=91)
public class Japan {
#DataField(pos=1, length=10)
private String TNR;
#DataField(pos=11, length=10)
private String ATR;
#DataField(pos=21, length=70)
private String STR;
public String getTNR() {
return TNR;
}
public void setTNR(String tNR) {
TNR = tNR;
}
public String getATR() {
return ATR;
}
public void setATR(String aTR) {
ATR = aTR;
}
public String getSTR() {
return STR;
}
public void setSTR(String sTR) {
STR = sTR;
}
}
First you need to create a data source to your database in your route. First thing is to add the mysql driver jar to your maven dependencies open your pom.xml file and add the following dependency to it.
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<!-- use this version of the driver or a later version of the driver -->
<version>5.1.25</version>
</dependency>
Right now we need to declare a custom processor to use in the route that will use this driver and insert the received body into a table.
So lets create a new class in Fuse IDE called PersistToDatabase code below:
package com.mbww.JapanData;
import java.sql.DriverManager;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.Map;
import org.apache.camel.Body;
import org.apache.camel.Exchange;
import org.apache.camel.Handler;
import org.apache.camel.Headers;
import com.mbww.model.Japan;
import com.mysql.jdbc.Statement;
public class PersistToDatabase {
#Handler
public void PersistRecord
(
#Body Japan msgBody
, #Headers Map hdr
, Exchange exch
) throws Exception
{
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
System.out.println("Where is your MySQL JDBC Driver?");
e.printStackTrace();
return;
}
System.out.println("MySQL JDBC Driver Registered!");
Connection connection = null;
try {
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/databasename","root", "password");
} catch (SQLException e) {
System.out.println("Connection Failed! Check output console");
e.printStackTrace();
return;
}
if (connection != null) {
System.out.println("You made it, take control your database now!");
} else {
System.out.println("Failed to make connection!");
}
try {
PreparedStatement stmt=connection.prepareStatement("INSERT INTO JapanDate(TNR,ATR,STR) VALUES(?,?,?)");
stmt.setString(1, msgBody.getTNR());
stmt.setString(2, msgBody.getATR());
stmt.setString(1, msgBody.getSTR());
int rows = stmt.executeUpdate();
System.out.println("Number of rows inserted: "+Integer.toString(rows));
}
catch(Exception e){
System.out.println("Error in executing sql statement: "+e.getMessage() );
throw new Exception(e.getMessage());
}
}
}
This class is a POJO nothing fancy except the #Handler annotation on the PersistRecord. This annotation tells camel that the PersistRecord method/procedure will handle the message exchange. You will also notice that the method PersistRecord has a parameter of type Japan. As mentioned earlier when you call the conversion bean in your camel route it translates each line into a Japan object and passes it along the route.
The rest of the code is just how to handle the JDBC connection and calling a insert statement.
We are almost done just one last thing to do. We need to declare this class in our camel route xml. This file will typically be called camel-route.xml or blueprint.xml depending on your arch type. Open the source tab and add the following line <bean id="JapanPersist" class="com.mbww.JapanData.PersistToDatabase"/> before the <camelContext> tag.
This declares a new spring bean called JapanPersist based on the class we just added to the camel route. You can now reference this bean inside your camel route.
Thus the final route xml file should look something like this:
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:camel="http://camel.apache.org/schema/blueprint"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">
<bean id="JapanPersist" class="com.mbww.JapanData.PersistToDatabase"/>
<camelContext trace="false" id="blueprintContext" xmlns="http://camel.apache.org/schema/blueprint">
<route id="JapanDataFromFileToDB">
<from uri="file:src/data/japan"/>
<unmarshal ref="Japan"/>
<bean ref="JapanPersist"/>
</route>
</camelContext>
</blueprint>
Or see screen shot below:
Once you understand this technique you can start scaling the solution by using a splitter, connection pooling and threading to do massive amount of concurrent inserts etc.
Using the technique above you learned how to inject your own beans into a camel route which give you the ability to work with the messages directly in code.
I have not tested the code so there will probably be a bug or two but the idea should be clear.
I have a simple program using JPA entities to write into a Derby DB (the entities were generated from an existing DB tables). I am using Eclipse and there is a working connection between the Derby client and the server via the EclipseLink Data Source Explorer .
Here is the start of my main():
import javax.persistence.*;
import java.sql.Timestamp;
import java.util.*;
import javax.*;
public class start {
/**
* #param args
*/
private static final String PERSISTENCE_UNIT_NAME = "zodiac";
private static EntityManagerFactory factory;
public static void main(String[] args) {
try {
factory = Persistence.createEntityManagerFactory(PERSISTENCE_UNIT_NAME);
EntityManager em = factory.createEntityManager();
System.out.println("after factory gen" );
when I the line with createEntityManager() is executed the following exception is thrown:
[EL Info]: 2012-03-07 22:46:21.892--ServerSession(253038357)--EclipseLink, version: Eclipse Persistence Services - 2.3.2.v20111125-r10461
[EL Severe]: 2012-03-07 22:46:22.064--ServerSession(253038357)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: No suitable driver
Error Code: 0
Any idea what is the problem ? thanks
If you're in Eclipse you need to add the driver to your project classpath. Sounds like you already have a datasource so you must have defined the driver library. All you need to do is "Add Library" to your Java Build Path and choose "Connectivity Driver Definition" and then the Derby driver from the drop down list of available driver definitions.
FYI, there's a checkbox in the New JPA Project wizard where you can select "add driver to classpath" to do this when you create a new project.
Of course you can also add the derbyclient.jar to your classpath directly or define a user library that includes it.
--Shaun