I am currently using Grails-2.2.4 on my application, recently I've been facing a strange issue. In my production war, the application is frequently losing mysql connection after some period and the application requires a restart for the connection to work again and it works fine for another couple of days.
I am using tomcat jdbc-pool:7.0.47 and here is my here is my datasource properties
dataSource {
shard = false
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dbCreate = "update" // one of 'create', 'create-drop','update'
properties {
initialSize=5
maxActive=50
minIdle=5
maxIdle=25
maxWait = 10000
maxAge = 10 * 60000
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
validationQuery="SELECT 1"
validationInterval=15000
testWhileIdle=true
testOnBorrow=true
testOnReturn=true
jdbcInterceptors = "ConnectionState"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
}
I have the same configuration in some other application and it has been running without any problems
Try to add autoreconnect argument in your datasource URL:
dataSource {
url = "jdbc:mysql://localhost:3306/mydb?autoReconnect=true"
}
Update
Another reason could be your TCP firewall is dropping the connection after some time being idle. So you need to modify the TCP timeout settings. I'm giving an related doc for how to achieve this on Amazon EC2 http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html. You can do it for your provider, that should be same.
Related
I want to connect to AWS MySQL database instance. Here is my code:
val hikari = HikariConfig().run {
driverClassName = "com.mysql.jdbc.Driver"
jdbcUrl = "jdbc:mysql://${mainConfig.databaseHost}:${mainConfig.databasePort}" +
"?user=username&password=password"
username = Config.DATABASE_USER
password = Config.DATABASE_PASSWORD
isAutoCommit = false
transactionIsolation = "TRANSACTION_REPEATABLE_READ"
return HikariDataSource(this)
}
Database.connect(hikari(config))
And my mainConfig.databaseHost looks like: mydatabase.xyz.region.rds.amazonaws.com.
So, this connection is working but I can't to exec any SQL statements because:
java.sql.SQLException: No database selected
I've tried to specify database name in my jdbc url but it causes exception.
So how can I connect to specified AWS database?
After research I found that DB identifier and DB name are different, so I specified DB name in jdbc url and now it working!
Most examples of JNDI, mysql 5.6 and tomcat 7 have this defined in DataSource.groovy:
dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dialect = 'org.hibernate.dialect.MySQL5InnoDBDialect'
}
environments:
production {
dataSource {
dbCreate = "update"
jndiName = "java:comp/env/myDatasourceName
}
}
and in the conf/context.xml in tomcat:
<Context>
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<Resource name="myDatasourceName" auth="Container" type="javax.sql.DataSource"
maxActive="100" maxIdle="30" maxWait="10000"
username="root" password="password" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/my_db_name"/>
</Context>
The question is, is this the correct setup for a basic low volume production system, which should reconnect if the db goes down then up etc?
In the DataSource.xml is a hugely complex example:
dataSource {
dbCreate = "update"
//url = "jdbc:h2:prodDb;MVCC=TRUE;LOCK_TIMEOUT=10099;DB_CLOSE_ON_EXIT=FALSE"
properties {
// Documentation for Tomcat JDBC Pool
// http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Common_Attributes
// https://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/tomcat/jdbc/pool/PoolConfiguration.html
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
ignoreExceptionOnPreLoad = true
// http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#JDBC_interceptors
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED // safe default
// controls for leaked connections
abandonWhenPercentageFull = 100 // settings are active only when pool is full
removeAbandonedTimeout = 120000
removeAbandoned = true
// use JMX console to change this setting at runtime
logAbandoned = false // causes stacktrace recording overhead, use only for debugging
/*
// JDBC driver properties
// Mysql as example
dbProperties {
// Mysql specific driver properties
// http://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html
// let Tomcat JDBC Pool handle reconnecting
autoReconnect=false
// truncation behaviour
jdbcCompliantTruncation=false
// mysql 0-date conversion
zeroDateTimeBehavior='convertToNull'
// Tomcat JDBC Pool's StatementCache is used instead, so disable mysql driver's cache
cachePrepStmts=false
cacheCallableStmts=false
// Tomcat JDBC Pool's StatementFinalizer keeps track
dontTrackOpenResources=true
// performance optimization: reduce number of SQLExceptions thrown in mysql driver code
holdResultsOpenOverStatementClose=true
// enable MySQL query cache - using server prep stmts will disable query caching
useServerPrepStmts=false
// metadata caching
cacheServerConfiguration=true
cacheResultSetMetadata=true
metadataCacheSize=100
// timeouts for TCP/IP
connectTimeout=15000
socketTimeout=120000
// timer tuning (disable)
maintainTimeStats=false
enableQueryTimeouts=false
// misc tuning
noDatetimeStringSync=true
}
*/
}
}
Now I dont know what 95% of those parameters do, but I guess I need them all?
What I dont understand is this
surely tomcat will pool the connections, so should pooled=true in the grails dataSource?
Surely tomcat will handle reconnecting and validation of connection, so why define validationQuery = "SELECT 1"? Expecially as its saying "autoREconnect=false in the mysql dbPrperties section.
Could someone come up with a general purpose default dataSource definition suitable for mysql for a small number of users? I can tune the min/max active/idle etc, but is this not all done in tomcat, not grails?
"userServerPrepStmts=false" why would anyone want to disable prepared statements? We have been drilled for the last 15 years to only ever use prepared statements otherwise the DB cant cache them.
If I ignore all this complex example, and went with the super smiple one everyone posts, will it work properly? e.g. will it reconnect when the db goes down and up? will it not leak etc.
Thanks!
AFAIK, Tomcat has nothing to do with managing the database connection. Since Grails 2.3.6, these are the recommended default datasource properties:
properties {
//see http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
Single DB connection (resource configuration) from application.ini:
resources.db.adapter = "pdo_mysql"
resources.db.params.host = "mysql1"
resources.db.params.username = "dbuser"
resources.db.params.password = "dbpass"
resources.db.params.dbname = "dbname"
;resources.db.???
Multiple DB connection (resource configuration) from application.ini:
resources.multidb.mysql1.adapter = "pdo_mysql"
resources.multidb.mysql1.host = "mysql1"
resources.multidb.mysql1.username = "dbuser"
resources.multidb.mysql1.password = "dbpass"
resources.multidb.mysql1.dbname = "dbname"
;resources.multidb.mysql1.???
I don't believe connecting to MySQL with SSL is currently supported in Zend. However, Zend Issue Tracker ticket #6140 may be of help to you.
I tried to install proxy on development machine and I got the following error.
/etc/init.d/mysql-proxyd start
Starting mysql-proxy: 2011-02-26 15:51:45: (critical) admin-plugin.c:569: --admin-username needs to be set
2011-02-26 15:51:45: (critical) mainloop.c:267: applying config of plugin admin failed
2011-02-26 15:51:45: (critical) mysql-proxy-cli.c:596: Failure from chassis_mainloop. Shutting down.
[ OK ]
Since this is only a test machine, I do not want the security feature of proxy. How do I avoid the above error?
Either upgrade your version of mysql-proxy to 0.8.2 or greater or explicitly specify that you don't need the admin plugin by using mysql-proxy --plugins=proxy
[mysql-proxy]
daemon = true
user = mysql
proxy-skip-profiling = true
keepalive = true
max-open-files = 2048
event-threads = 50
pid-file = /var/run/mysql-proxy.pid
log-file = /var/log/mysql-proxy.log
log-level = debug
admin-address=:4401
admin-username=1
admin-password=1
admin-lua-script=/usr/local/lib/mysql-proxy/lua/admin.lua
proxy-address = 0.0.0.0:3307
proxy-backend-addresses = 192.168.2.1:3306
proxy-read-only-backend-addresses=192.168.6.2:3306, 192.168.6.1:3306
proxy-lua-script=/usr/lib/mysql-proxy/lua/proxy/balance.lua
I have a small grails application running on Tomcat in Ubuntu on a VPS. I use MySql as my datastore and everything works fine unless I leave the application for more than half a day (8 hours?). I did some searching and apparently this is the default wait_timeout in mysql.cnf so after 8 hours the connection will die but Tomcat won't know so when the next user tries to view the site they will see the connection failure error. Refreshing the page will fix this but I want to get rid of the error altogether. For my version of MySql (5.0.75) I have only my.cnf and it doesn't contain such a parameter, In any case changing this parameter doesn't solve the problem.
This Blog Post seems to be reporting a similar error but I still don't fully understand what I need to configure to get this fixed and also I am hoping that there is a simpler solution than another third party library. The machine I'm running on has 256MB ram and I'm trying to keep the number of programs/services running to a minimum.
Is there something I can configure in Grails / Tomcat / MySql to get this to go away?
Thanks in advance,
Gav
From my Catalina.out;
2010-04-29 21:26:25,946 [http-8080-2] ERROR util.JDBCExceptionReporter - The last packet successfully received from the server was 102,906,722 milliseconds$
2010-04-29 21:26:25,994 [http-8080-2] ERROR errors.GrailsExceptionResolver - Broken pipe
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
...
2010-04-29 21:26:26,016 [http-8080-2] ERROR util.JDBCExceptionReporter - Already closed.
2010-04-29 21:26:26,016 [http-8080-2] ERROR util.JDBCExceptionReporter - Already closed.
2010-04-29 21:26:26,017 [http-8080-2] ERROR servlet.GrailsDispatcherServlet - HandlerInterceptor.afterCompletion threw exception
org.hibernate.exception.GenericJDBCException: Cannot release connection
at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.SQLException: Already closed.
at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:84)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
... 1 more
Referring to this article, you have stale connections in your DBCP connections pool that are silently dropped by OS or firewall.
The solution is to define a validation query and do a sanity check of the connection before you actually use it in your application.
In grails this is actually done by modifying the grails-app/conf/spring/Resource.groovy file and add the following:
beans = {
dataSource(BasicDataSource) {
//run the evictor every 30 minutes and evict any connections older than 30 minutes.
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
//test the connection while its idle, before borrow and return it
testOnBorrow=true
testWhileIdle=true
testOnReturn=true
validationQuery="SELECT 1"
}
}
In grails 1.3.X, you can modify the evictor values in the DataSource.groovy file to make sure pooled connections are used during idle. This will make sure the mysql server will not time out the connection.
production {
dataSource {
pooled = true
// Other database parameters..
properties {
maxActive = 50
maxIdle = 25
minIdle = 5
initialSize = 5
minEvictableIdleTimeMillis = 1800000
timeBetweenEvictionRunsMillis = 1800000
maxWait = 10000
}
}
A quick way to verify this works is to modify the MySQL my.cnf configuration file [mysql] element and add wait_time parameter with a low value.
Try increasing the number of open MySQL connections by putting the following in your DataSources.groovy:
dataSource {
driverClassName = "com.mysql.jdbc.Driver"
pooled=true
maxActive=10
initialSize=5
// Remaining connection params
}
If you want to go the whole hog, try implementing a connection pool; here is a useful link on this.
For grails 1.3.X, I had to add the following code to Bootstrap.groovy :
def init = {servletContext ->
def ctx=servletContext.getAttribute(ApplicationAttributes.APPLICATION_CONTEXT)
//implement test on borrow
def dataSource = ctx.dataSource
dataSource.targetDataSource.setMinEvictableIdleTimeMillis(1000 * 60 * 30)
dataSource.targetDataSource.setTimeBetweenEvictionRunsMillis(1000 * 60 * 30)
dataSource.targetDataSource.setNumTestsPerEvictionRun(3)
dataSource.targetDataSource.setTestOnBorrow(true)
dataSource.targetDataSource.setTestWhileIdle(true)
dataSource.targetDataSource.setTestOnReturn(false)
dataSource.targetDataSource.setValidationQuery("SELECT 1")
}
I also had to import org.codehaus.groovy.grails.commons.ApplicationAttributes
Add these parameters to dataSource
testOnBorrow = true
testWhileIdle = true
testOnReturn = true
See this article for more information
http://sacharya.com/grails-dbcp-stale-connections/
Starting from grails 2.3.6 default configuration already has options for preventing closing connection by timeout
These are the new defaults.
properties {
// See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
....
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
What does your JDBC connection string look like? You can set an autoReconneect param in your data source config, e.g.
jdbc:mysql://hostname/mydb?autoReconnect=true