best mysql datasource properties for grails tomcat jndi - mysql

Most examples of JNDI, mysql 5.6 and tomcat 7 have this defined in DataSource.groovy:
dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dialect = 'org.hibernate.dialect.MySQL5InnoDBDialect'
}
environments:
production {
dataSource {
dbCreate = "update"
jndiName = "java:comp/env/myDatasourceName
}
}
and in the conf/context.xml in tomcat:
<Context>
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<Resource name="myDatasourceName" auth="Container" type="javax.sql.DataSource"
maxActive="100" maxIdle="30" maxWait="10000"
username="root" password="password" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/my_db_name"/>
</Context>
The question is, is this the correct setup for a basic low volume production system, which should reconnect if the db goes down then up etc?
In the DataSource.xml is a hugely complex example:
dataSource {
dbCreate = "update"
//url = "jdbc:h2:prodDb;MVCC=TRUE;LOCK_TIMEOUT=10099;DB_CLOSE_ON_EXIT=FALSE"
properties {
// Documentation for Tomcat JDBC Pool
// http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Common_Attributes
// https://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/tomcat/jdbc/pool/PoolConfiguration.html
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
ignoreExceptionOnPreLoad = true
// http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#JDBC_interceptors
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED // safe default
// controls for leaked connections
abandonWhenPercentageFull = 100 // settings are active only when pool is full
removeAbandonedTimeout = 120000
removeAbandoned = true
// use JMX console to change this setting at runtime
logAbandoned = false // causes stacktrace recording overhead, use only for debugging
/*
// JDBC driver properties
// Mysql as example
dbProperties {
// Mysql specific driver properties
// http://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html
// let Tomcat JDBC Pool handle reconnecting
autoReconnect=false
// truncation behaviour
jdbcCompliantTruncation=false
// mysql 0-date conversion
zeroDateTimeBehavior='convertToNull'
// Tomcat JDBC Pool's StatementCache is used instead, so disable mysql driver's cache
cachePrepStmts=false
cacheCallableStmts=false
// Tomcat JDBC Pool's StatementFinalizer keeps track
dontTrackOpenResources=true
// performance optimization: reduce number of SQLExceptions thrown in mysql driver code
holdResultsOpenOverStatementClose=true
// enable MySQL query cache - using server prep stmts will disable query caching
useServerPrepStmts=false
// metadata caching
cacheServerConfiguration=true
cacheResultSetMetadata=true
metadataCacheSize=100
// timeouts for TCP/IP
connectTimeout=15000
socketTimeout=120000
// timer tuning (disable)
maintainTimeStats=false
enableQueryTimeouts=false
// misc tuning
noDatetimeStringSync=true
}
*/
}
}
Now I dont know what 95% of those parameters do, but I guess I need them all?
What I dont understand is this
surely tomcat will pool the connections, so should pooled=true in the grails dataSource?
Surely tomcat will handle reconnecting and validation of connection, so why define validationQuery = "SELECT 1"? Expecially as its saying "autoREconnect=false in the mysql dbPrperties section.
Could someone come up with a general purpose default dataSource definition suitable for mysql for a small number of users? I can tune the min/max active/idle etc, but is this not all done in tomcat, not grails?
"userServerPrepStmts=false" why would anyone want to disable prepared statements? We have been drilled for the last 15 years to only ever use prepared statements otherwise the DB cant cache them.
If I ignore all this complex example, and went with the super smiple one everyone posts, will it work properly? e.g. will it reconnect when the db goes down and up? will it not leak etc.
Thanks!

AFAIK, Tomcat has nothing to do with managing the database connection. Since Grails 2.3.6, these are the recommended default datasource properties:
properties {
//see http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}

Related

Simulate "No operations allowed after connection closed"

I'm getting this error "No operations allowed after connection closed" from grils2x/mysql/dbcp occasionally and I couldn't find a solution.
because I get the error hours later, like next day after a restart, it's difficult to fix it.
I feel like I need to replicate it in a predictable manner and so I can find a definitive fix.
What parameter set can I use for MySQL and Grails side that ends up with that error immediately after it is run?
I ended up setting two timeouts manually , and make a connection afterwards , and see the following in mysql logs , to confirm connection was aborted after 10 seconds.
2022-02-12T22:03:53.690960Z 493 [Note] Aborted connection 493 to db: 'quantanywhere_2' user: 'root' host: '172.17.0.1' (Got timeout reading communication packets)
SET ##GLOBAL.wait_timeout=10;
SET ##SESSION.wait_timeout=10;
And following should be ok with the default wait_timeout of 28800 seconds=8 hours.
validationInterval = 28000
testWhileIdle=true
maxActive = 50
maxIdle = 25
maxWait = 10000
maxAge: 600000
minIdle = 5
validationQuery="select 1"
validationQueryTimeout=3
initialSize = 10
minEvictableIdleTimeMillis = 60000
timeBetweenEvictionRunsMillis = 5000
numTestsPerEvictionRun = 3
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
removeAbandoned = true
removeAbandonedTimeout = 120

How to Set config params in Slick-HikariCP

I am using Slick and Mysql. Getting the Error:
Failed to validate connection com.mysql.jdbc.JDBC4Connection#58e7cfcd (No operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value."
I understand the issue but don't know how to set maxLifetime in slick on config. Here is my config:
mysqldb {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://localhost:3306"
user = root
password = root
connectionTimeout = 10000
queueSize = 50000
minConnections = 20
maxConnections = 40
numThreads = 40
}
Any help would be appreciated.
you can set under db config:
mysqldb {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://localhost:3306"
user = root
password = root
connectionTimeout = 10000
queueSize = 50000
minConnections = 20
maxConnections = 40
numThreads = 40
idleTimeout = 85000 // should be less than maxLifetime
maxLifetime = 90000 //time is in millsecods and should be less than database wait_timeout
}

Configuring the local MySql database connection to grails application

I am a newbie to Grails, I have just started to create my first application just to make CRUD operations by storing the data to my local MySql database and I want to see the entered data in my machine. But I am unable to connect to database itself. Please help me where I am wrong.
My DataSource.groovy code is as shown below:
dataSource {
pooled = true
jmxExport = true
driverClassName = com.mysql.jdbc.Driver
dialect = org.hibernate.dialect.MySQL5InnoDBDialect
username = "test"
password = "test"
}
environments {
development {
dataSource {
dbCreate = "update"
url = "jdbc:mysql://localhost:3306/librarydb"
}
}
test {
dataSource {
dbCreate = "update"
url = "jdbc:mysql://localhost:3306/librarydb"
//url = "jdbc:h2:mem:testDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
}
}
production {
dataSource {
dbCreate = "update"
url = "jdbc:mysql://localhost:3306/librarydb"
//url = "jdbc:h2:prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
properties {
// See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState"
defaultTransactionIsolation =
java.sql.Connection.TRANSACTION_READ_COMMITTED
}
}
}
}
And my BuildConfig.groovy file is configured as shown.
grails.servlet.version = "3.0" // Change depending on target container
compliance (2.5 or 3.0)
grails.project.class.dir = "target/classes"
grails.project.test.class.dir = "target/test-classes"
grails.project.test.reports.dir = "target/test-reports"
grails.project.work.dir = "target/work"
grails.project.target.level = 1.6
grails.project.source.level = 1.6
//grails.project.war.file = "target/${appName}-${appVersion}.war"
grails.project.fork = [
// configure settings for compilation JVM, note that if you alter the Groovy
version forked compilation is required
// compile: [maxMemory: 256, minMemory: 64, debug: false, maxPerm: 256,
daemon:true],
// configure settings for the test-app JVM, uses the daemon by default
test: [maxMemory: 768, minMemory: 64, debug: false, maxPerm: 256,
daemon:true],
// configure settings for the run-app JVM
run: [maxMemory: 768, minMemory: 64, debug: false, maxPerm: 256,
forkReserve:false],
// configure settings for the run-war JVM
war: [maxMemory: 768, minMemory: 64, debug: false, maxPerm: 256,
forkReserve:false],
// configure settings for the Console UI JVM
console: [maxMemory: 768, minMemory: 64, debug: false, maxPerm: 256]
]
grails.project.dependency.resolver = "maven" // or ivy
grails.project.dependency.resolution = {
// inherit Grails' default dependencies
inherits("global") {
// specify dependency exclusions here; for example, uncomment this to
disable ehcache:
// excludes 'ehcache'
}
log "error" // log level of Ivy resolver, either 'error', 'warn', 'info',
'debug' or 'verbose'
checksums true // Whether to verify checksums on resolve
legacyResolve false // whether to do a secondary resolve on plugin
installation, not advised and here for backwards compatibility
repositories {
inherits true // Whether to inherit repository definitions from plugins
grailsPlugins()
grailsHome()
mavenLocal()
grailsCentral()
mavenCentral()
// uncomment these (or add new ones) to enable remote dependency resolution from public Maven repositories
//mavenRepo "http://repository.codehaus.org"
//mavenRepo "http://download.java.net/maven/2/"
//mavenRepo "http://repository.jboss.com/maven2/"
}
dependencies {
// specify dependencies here under either 'build', 'compile', 'runtime', 'test' or 'provided' scopes e.g.
runtime 'mysql:mysql-connector-java:5.1.44'
// runtime 'org.postgresql:postgresql:9.3-1101-jdbc41'
test "org.grails:grails-datastore-test-support:1.0.2-grails-2.4"
}
plugins {
// plugins for the build system only
build ":tomcat:7.0.70" // or ":tomcat:8.0.22"
// plugins for the compile step
compile ":scaffolding:2.1.2"
compile ':cache:1.1.8'
// asset-pipeline 2.0+ requires Java 7, use version 1.9.x with Java 6
compile ":asset-pipeline:2.5.7"
// plugins needed at runtime but not for compilation
runtime ":hibernate4:4.3.10" // or ":hibernate:3.6.10.18"
runtime ":database-migration:1.4.0"
runtime ":jquery:1.11.1"
// Uncomment these to enable additional asset-pipeline capabilities
//compile ":sass-asset-pipeline:1.9.0"
//compile ":less-asset-pipeline:1.10.0"
//compile ":coffee-asset-pipeline:1.8.0"
//compile ":handlebars-asset-pipeline:1.3.0.3"
}
}
But I am getting the below error in my command prompt.
[localhost-startStop-1] ERROR pool.ConnectionPool - Unable to create initial connections of pool.
Message: class com.mysql.jdbc.Driver
Line | Method
->> 266 | run in java.util.concurrent.FutureTask
1149 | runWorker in java.util.concurrent.ThreadPoolExecutor
624 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
748 | run in java.lang.Thread
Caused by ClassNotFoundException: class com.mysql.jdbc.Driver
->> 381 | findClass in java.net.URLClassLoader
424 | loadClass in java.lang.ClassLoader
348 | forName . in java.lang.Class
266 | run in java.util.concurrent.FutureTask
1149 | runWorker in java.util.concurrent.ThreadPoolExecutor
624 | run in java.util.concurrent.ThreadPoolExecutor$Worker
748 | run . . . in java.lang.Thread
Error |
2017-11-21 23:22:20,629 [localhost-startStop-1] ERROR
pool.ConnectionPool - Unable to create initial connections of pool.
Message: class com.mysql.jdbc.Driver
Line | Method
->>266 | run in java.util.concurrent.FutureTask
I am using Grails 2.5.6 version, and I have added the mysql-connector-java-5.1.44-bin.jar file in my lib folder of grails application and JAVA_HOME as well.Please help me to solve this and connect to my database to store the data. Thanks in advance.
Try grails clean
then grails refresh-dependencies ceck and make confirm the jar mysql:mysql-connector-java:5.1.29 is in your build path.
UPDATE :
Remove mysql dependency from BuildConfig.groovy if you have mysql-connector-java:5.x.x jar in the lib or vise-versa.
Multiple Jar file makes trouble.
Related post:
Unable to create initial connections of pool issues in Grails
java.lang.ClassNotFoundException:com.mysql.jdbc.Driver
The database needs to be reated first. Grails will try to connect to existing database, it will not be able to create a database for you.
Run these commands in the terminal, if you are using linux/unix, become a superuser and run the following:
CREATE DATABASE librarydb;
CREATE USER 'test'#'test' IDENTIFIED BY 'test';
GRANT ALL ON librarydb.* TO test#localhost;
if you have done all of that, please check username/password combo.
in terminal run this:
mysql -u test -p test and see if you can connect to the database.
it worked for me when I removed the jar file from the lib folder of my grails application and used the default mysql connection. i.e username="root" and password="root" in my datasource file. Thanks to #devbd for the answer.

Mysql connection lost after a certain interval grails application

I am currently using Grails-2.2.4 on my application, recently I've been facing a strange issue. In my production war, the application is frequently losing mysql connection after some period and the application requires a restart for the connection to work again and it works fine for another couple of days.
I am using tomcat jdbc-pool:7.0.47 and here is my here is my datasource properties
dataSource {
shard = false
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dbCreate = "update" // one of 'create', 'create-drop','update'
properties {
initialSize=5
maxActive=50
minIdle=5
maxIdle=25
maxWait = 10000
maxAge = 10 * 60000
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
validationQuery="SELECT 1"
validationInterval=15000
testWhileIdle=true
testOnBorrow=true
testOnReturn=true
jdbcInterceptors = "ConnectionState"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
}
I have the same configuration in some other application and it has been running without any problems
Try to add autoreconnect argument in your datasource URL:
dataSource {
url = "jdbc:mysql://localhost:3306/mydb?autoReconnect=true"
}
Update
Another reason could be your TCP firewall is dropping the connection after some time being idle. So you need to modify the TCP timeout settings. I'm giving an related doc for how to achieve this on Amazon EC2 http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html. You can do it for your provider, that should be same.

MySQL Connection Timeout Issue - Grails Application on Tomcat using Hibernate and ORM

I have a small grails application running on Tomcat in Ubuntu on a VPS. I use MySql as my datastore and everything works fine unless I leave the application for more than half a day (8 hours?). I did some searching and apparently this is the default wait_timeout in mysql.cnf so after 8 hours the connection will die but Tomcat won't know so when the next user tries to view the site they will see the connection failure error. Refreshing the page will fix this but I want to get rid of the error altogether. For my version of MySql (5.0.75) I have only my.cnf and it doesn't contain such a parameter, In any case changing this parameter doesn't solve the problem.
This Blog Post seems to be reporting a similar error but I still don't fully understand what I need to configure to get this fixed and also I am hoping that there is a simpler solution than another third party library. The machine I'm running on has 256MB ram and I'm trying to keep the number of programs/services running to a minimum.
Is there something I can configure in Grails / Tomcat / MySql to get this to go away?
Thanks in advance,
Gav
From my Catalina.out;
2010-04-29 21:26:25,946 [http-8080-2] ERROR util.JDBCExceptionReporter - The last packet successfully received from the server was 102,906,722 milliseconds$
2010-04-29 21:26:25,994 [http-8080-2] ERROR errors.GrailsExceptionResolver - Broken pipe
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
...
2010-04-29 21:26:26,016 [http-8080-2] ERROR util.JDBCExceptionReporter - Already closed.
2010-04-29 21:26:26,016 [http-8080-2] ERROR util.JDBCExceptionReporter - Already closed.
2010-04-29 21:26:26,017 [http-8080-2] ERROR servlet.GrailsDispatcherServlet - HandlerInterceptor.afterCompletion threw exception
org.hibernate.exception.GenericJDBCException: Cannot release connection
at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.SQLException: Already closed.
at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:84)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
... 1 more
Referring to this article, you have stale connections in your DBCP connections pool that are silently dropped by OS or firewall.
The solution is to define a validation query and do a sanity check of the connection before you actually use it in your application.
In grails this is actually done by modifying the grails-app/conf/spring/Resource.groovy file and add the following:
beans = {
dataSource(BasicDataSource) {
//run the evictor every 30 minutes and evict any connections older than 30 minutes.
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
//test the connection while its idle, before borrow and return it
testOnBorrow=true
testWhileIdle=true
testOnReturn=true
validationQuery="SELECT 1"
}
}
In grails 1.3.X, you can modify the evictor values in the DataSource.groovy file to make sure pooled connections are used during idle. This will make sure the mysql server will not time out the connection.
production {
dataSource {
pooled = true
// Other database parameters..
properties {
maxActive = 50
maxIdle = 25
minIdle = 5
initialSize = 5
minEvictableIdleTimeMillis = 1800000
timeBetweenEvictionRunsMillis = 1800000
maxWait = 10000
}
}
A quick way to verify this works is to modify the MySQL my.cnf configuration file [mysql] element and add wait_time parameter with a low value.
Try increasing the number of open MySQL connections by putting the following in your DataSources.groovy:
dataSource {
driverClassName = "com.mysql.jdbc.Driver"
pooled=true
maxActive=10
initialSize=5
// Remaining connection params
}
If you want to go the whole hog, try implementing a connection pool; here is a useful link on this.
For grails 1.3.X, I had to add the following code to Bootstrap.groovy :
def init = {servletContext ->
def ctx=servletContext.getAttribute(ApplicationAttributes.APPLICATION_CONTEXT)
//implement test on borrow
def dataSource = ctx.dataSource
dataSource.targetDataSource.setMinEvictableIdleTimeMillis(1000 * 60 * 30)
dataSource.targetDataSource.setTimeBetweenEvictionRunsMillis(1000 * 60 * 30)
dataSource.targetDataSource.setNumTestsPerEvictionRun(3)
dataSource.targetDataSource.setTestOnBorrow(true)
dataSource.targetDataSource.setTestWhileIdle(true)
dataSource.targetDataSource.setTestOnReturn(false)
dataSource.targetDataSource.setValidationQuery("SELECT 1")
}
I also had to import org.codehaus.groovy.grails.commons.ApplicationAttributes
Add these parameters to dataSource
testOnBorrow = true
testWhileIdle = true
testOnReturn = true
See this article for more information
http://sacharya.com/grails-dbcp-stale-connections/
Starting from grails 2.3.6 default configuration already has options for preventing closing connection by timeout
These are the new defaults.
properties {
// See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
....
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
What does your JDBC connection string look like? You can set an autoReconneect param in your data source config, e.g.
jdbc:mysql://hostname/mydb?autoReconnect=true