I am using Slick and Mysql. Getting the Error:
Failed to validate connection com.mysql.jdbc.JDBC4Connection#58e7cfcd (No operations allowed after connection closed.). Possibly consider using a shorter maxLifetime value."
I understand the issue but don't know how to set maxLifetime in slick on config. Here is my config:
mysqldb {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://localhost:3306"
user = root
password = root
connectionTimeout = 10000
queueSize = 50000
minConnections = 20
maxConnections = 40
numThreads = 40
}
Any help would be appreciated.
you can set under db config:
mysqldb {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://localhost:3306"
user = root
password = root
connectionTimeout = 10000
queueSize = 50000
minConnections = 20
maxConnections = 40
numThreads = 40
idleTimeout = 85000 // should be less than maxLifetime
maxLifetime = 90000 //time is in millsecods and should be less than database wait_timeout
}
Recently upgraded to MySQL 5.7.12 on a Debian (Debian 3.2.78-1 x86_64 GNU/Linux) and have been running into the server hanging after every few hours. This is getting flooded in the syslog and mysql.log:
2016-06-13T18:05:20.261209Z 0 [ERROR] Error in accept: Bad file descriptor
MySQL info:
mysql Ver 14.14 Distrib 5.7.12-5, for debian-linux-gnu (x86_64) using 6.2
Pieces of my.cnf mysqld section that can guide some assistance on tweaking values:
[mysqld]
max_allowed_packet = 64M
thread_stack = 256K
thread_cache_size = 8
max_connections = 150
max_connect_errors = 10000
connect_timeout = 30
wait_timeout = 86400
table_open_cache = 2048
open_files_limit = 65535
query_cache_limit = 4M
query_cache_size = 128M
query_cache_type = 1
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size = 100M
# * InnoDB
innodb_file_per_table
innodb_buffer_pool_instances=2
innodb_buffer_pool_size=2G
thread_pool_size = 24
We had the same issue on a Ubuntu 16.04 system with mysql 5.7.13 . We increased our max open files parameter in systemd like this:
/etc/systemd/system/mysql.service.d/10-ulimit.conf
[Service]
LimitNOFILE=1000000
So far the issue did not happen again. Maybe mysql needs somehow more file descriptors now.
I found the problem (or possibly one of the problems). Here is an extract from strace on mysqld:
...
socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 20
write(2, "2017-01-29T22:22:45.433033Z 0 [N"..., 72) = 72
setsockopt(20, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(20, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
bind(20, {sa_family=AF_INET6, sin6_port=htons(3306), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
listen(20, 70) = 0
fcntl(20, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(20, F_SETFL, O_RDWR|O_NONBLOCK) = 0
...
accept(20, {sa_family=AF_INET6, sin6_port=htons(58332), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 37
rt_sigaction(SIGCHLD, {SIG_DFL, [CHLD], SA_RESTORER|SA_RESTART, 0x7f3ddeac84b0}, {SIG_DFL, [], 0}, 8) = 0
getpeername(37, {sa_family=AF_INET6, sin6_port=htons(58332), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
getsockname(37, {sa_family=AF_INET6, sin6_port=htons(3306), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
open("/etc/hosts.allow", O_RDONLY) = 38
fstat(38, {st_mode=S_IFREG|0644, st_size=589, ...}) = 0
read(38, "# /etc/hosts.allow: list of host"..., 4096) = 589
read(38, "", 4096) = 0
close(38) = 0
open("/etc/hosts.deny", O_RDONLY) = 38
fstat(38, {st_mode=S_IFREG|0644, st_size=704, ...}) = 0
read(38, "# /etc/hosts.deny: list of hosts"..., 4096) = 704
close(38) = 0
socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 38
connect(38, {sa_family=AF_LOCAL, sun_path="/dev/log"}, 110) = 0
sendto(38, "<36>Jan 29 14:23:08 mysqld[13052"..., 72, MSG_NOSIGNAL, NULL, 0) = 72
shutdown(20, SHUT_RDWR) = 0
close(20) = 0
poll([{fd=20, events=POLLIN}, {fd=22, events=POLLIN}], 2, -1) = 1 ([{fd=20, revents=POLLNVAL}])
accept(-1, 0x7ffe6ebd7160, 0x7ffe6ebd70fc) = -1 EBADF (Bad file descriptor)
write(2, "2017-01-29T22:23:08.109451Z 0 [E"..., 75) = 75
... rinse and repeat *REALLY* fast!
In locking down my system with tcp_wrappers I had inadvertently taken mysqld out of both hosts.allow and hosts.deny. It seems that after checking both hosts.allow and hosts.deny mysqld shuts down and closes the socket as you might expect. However it them immediately starts to poll the (now non-existent) socket for activity.
I just did another test where my tcp_wrappers was correctly configured. When I connect from an authorized host all is fine; however when I connected from a blocked address the same issue occurs. Based on this I recommend using other tools to secure mysqld and making your tcp_wrappers config more open than your firewall. That being said the bug should still be fixed!
This fix has yet to stand the test of time so, as usual, YMMV. Hope it helps anyway
Nick
Researched a bit and found following;
Present in MariaDB also
https://lists.launchpad.net/maria-discuss/msg03060.html
https://mariadb.atlassian.net/browse/MDEV-8995
Percona Server/Percona XtraDB Cluster
https://groups.google.com/forum/#!topic/percona-discussion/Tu0S2OvYqKA
Old bug from 2010/2012
https://bugs.mysql.com/bug.php?id=48929
http://lists.mysql.com/commits/96472
Some interesting information (should never happen)
https://lists.mysql.com/mysql/97275
[I work for Percona]
I have the same issue after upgrading to Percona Cluster 5.7.14-26.17-1.trusty.
The ulimit.conf suggestion doesn't help, and I've made sure that there are sufficient file handles, so far as I can tell, by editing /etc/security/limits.conf and /etc/sysctl.conf.
I can reproduce this easily by telnetting to post 3306 and then disconnecting; the server then goes into a spin logging this error.
A horrible workaround for this, which looks promising in my environment, is to avoid using TCP connections on port 3306, and use unix sockets instead.
You can proxy from port 3306 to the socket by changing the port number in /etc/mysql/my.cnf and then using socat
nohup socat TCP4-LISTEN:3306,fork UNIX-CONNECT:/var/run/mysqld/mysqld.sock&
If I then telnet in on port 3306 and disconnect, I can't provoke the problem. I intend to report back on how well this stands up over time.
FWIW, the code looks as though it expects this to happen sometimes:
for (uint retry= 0; retry < MAX_ACCEPT_RETRY; retry++)
{
socket_len_t length= sizeof(struct sockaddr_storage);
connect_sock= mysql_socket_accept(key_socket_client_connection, listen_sock,
(struct sockaddr *)(&cAddr), &length);
if (mysql_socket_getfd(connect_sock) != INVALID_SOCKET ||
(socket_errno != SOCKET_EINTR && socket_errno != SOCKET_EAGAIN))
break;
}
if (mysql_socket_getfd(connect_sock) == INVALID_SOCKET)
{
/*
accept(2) failed on the listening port, after many retries.
There is not much details to report about the client,
increment the server global status variable.
*/
connection_errors_accept++;
if ((m_error_count++ & 255) == 0) // This can happen often
sql_print_error("Error in accept: %s", strerror(errno));
if (socket_errno == SOCKET_ENFILE || socket_errno == SOCKET_EMFILE)
sleep(1); // Give other threads some time
return NULL;
}
I came here with the same error and none of the solutions worked BUT after some research on our end we found that it was apparmor that was denying our logging directory causing the bad file descriptors error message.
I am currently using Grails-2.2.4 on my application, recently I've been facing a strange issue. In my production war, the application is frequently losing mysql connection after some period and the application requires a restart for the connection to work again and it works fine for another couple of days.
I am using tomcat jdbc-pool:7.0.47 and here is my here is my datasource properties
dataSource {
shard = false
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dbCreate = "update" // one of 'create', 'create-drop','update'
properties {
initialSize=5
maxActive=50
minIdle=5
maxIdle=25
maxWait = 10000
maxAge = 10 * 60000
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
validationQuery="SELECT 1"
validationInterval=15000
testWhileIdle=true
testOnBorrow=true
testOnReturn=true
jdbcInterceptors = "ConnectionState"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
}
I have the same configuration in some other application and it has been running without any problems
Try to add autoreconnect argument in your datasource URL:
dataSource {
url = "jdbc:mysql://localhost:3306/mydb?autoReconnect=true"
}
Update
Another reason could be your TCP firewall is dropping the connection after some time being idle. So you need to modify the TCP timeout settings. I'm giving an related doc for how to achieve this on Amazon EC2 http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-firewall-guidance.html. You can do it for your provider, that should be same.
Most examples of JNDI, mysql 5.6 and tomcat 7 have this defined in DataSource.groovy:
dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dialect = 'org.hibernate.dialect.MySQL5InnoDBDialect'
}
environments:
production {
dataSource {
dbCreate = "update"
jndiName = "java:comp/env/myDatasourceName
}
}
and in the conf/context.xml in tomcat:
<Context>
<WatchedResource>WEB-INF/web.xml</WatchedResource>
<Resource name="myDatasourceName" auth="Container" type="javax.sql.DataSource"
maxActive="100" maxIdle="30" maxWait="10000"
username="root" password="password" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/my_db_name"/>
</Context>
The question is, is this the correct setup for a basic low volume production system, which should reconnect if the db goes down then up etc?
In the DataSource.xml is a hugely complex example:
dataSource {
dbCreate = "update"
//url = "jdbc:h2:prodDb;MVCC=TRUE;LOCK_TIMEOUT=10099;DB_CLOSE_ON_EXIT=FALSE"
properties {
// Documentation for Tomcat JDBC Pool
// http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Common_Attributes
// https://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/tomcat/jdbc/pool/PoolConfiguration.html
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
ignoreExceptionOnPreLoad = true
// http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#JDBC_interceptors
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED // safe default
// controls for leaked connections
abandonWhenPercentageFull = 100 // settings are active only when pool is full
removeAbandonedTimeout = 120000
removeAbandoned = true
// use JMX console to change this setting at runtime
logAbandoned = false // causes stacktrace recording overhead, use only for debugging
/*
// JDBC driver properties
// Mysql as example
dbProperties {
// Mysql specific driver properties
// http://dev.mysql.com/doc/connector-j/en/connector-j-reference-configuration-properties.html
// let Tomcat JDBC Pool handle reconnecting
autoReconnect=false
// truncation behaviour
jdbcCompliantTruncation=false
// mysql 0-date conversion
zeroDateTimeBehavior='convertToNull'
// Tomcat JDBC Pool's StatementCache is used instead, so disable mysql driver's cache
cachePrepStmts=false
cacheCallableStmts=false
// Tomcat JDBC Pool's StatementFinalizer keeps track
dontTrackOpenResources=true
// performance optimization: reduce number of SQLExceptions thrown in mysql driver code
holdResultsOpenOverStatementClose=true
// enable MySQL query cache - using server prep stmts will disable query caching
useServerPrepStmts=false
// metadata caching
cacheServerConfiguration=true
cacheResultSetMetadata=true
metadataCacheSize=100
// timeouts for TCP/IP
connectTimeout=15000
socketTimeout=120000
// timer tuning (disable)
maintainTimeStats=false
enableQueryTimeouts=false
// misc tuning
noDatetimeStringSync=true
}
*/
}
}
Now I dont know what 95% of those parameters do, but I guess I need them all?
What I dont understand is this
surely tomcat will pool the connections, so should pooled=true in the grails dataSource?
Surely tomcat will handle reconnecting and validation of connection, so why define validationQuery = "SELECT 1"? Expecially as its saying "autoREconnect=false in the mysql dbPrperties section.
Could someone come up with a general purpose default dataSource definition suitable for mysql for a small number of users? I can tune the min/max active/idle etc, but is this not all done in tomcat, not grails?
"userServerPrepStmts=false" why would anyone want to disable prepared statements? We have been drilled for the last 15 years to only ever use prepared statements otherwise the DB cant cache them.
If I ignore all this complex example, and went with the super smiple one everyone posts, will it work properly? e.g. will it reconnect when the db goes down and up? will it not leak etc.
Thanks!
AFAIK, Tomcat has nothing to do with managing the database connection. Since Grails 2.3.6, these are the recommended default datasource properties:
properties {
//see http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
jmxEnabled = true
initialSize = 5
maxActive = 50
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
I have a small grails application running on Tomcat in Ubuntu on a VPS. I use MySql as my datastore and everything works fine unless I leave the application for more than half a day (8 hours?). I did some searching and apparently this is the default wait_timeout in mysql.cnf so after 8 hours the connection will die but Tomcat won't know so when the next user tries to view the site they will see the connection failure error. Refreshing the page will fix this but I want to get rid of the error altogether. For my version of MySql (5.0.75) I have only my.cnf and it doesn't contain such a parameter, In any case changing this parameter doesn't solve the problem.
This Blog Post seems to be reporting a similar error but I still don't fully understand what I need to configure to get this fixed and also I am hoping that there is a simpler solution than another third party library. The machine I'm running on has 256MB ram and I'm trying to keep the number of programs/services running to a minimum.
Is there something I can configure in Grails / Tomcat / MySql to get this to go away?
Thanks in advance,
Gav
From my Catalina.out;
2010-04-29 21:26:25,946 [http-8080-2] ERROR util.JDBCExceptionReporter - The last packet successfully received from the server was 102,906,722 milliseconds$
2010-04-29 21:26:25,994 [http-8080-2] ERROR errors.GrailsExceptionResolver - Broken pipe
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
...
2010-04-29 21:26:26,016 [http-8080-2] ERROR util.JDBCExceptionReporter - Already closed.
2010-04-29 21:26:26,016 [http-8080-2] ERROR util.JDBCExceptionReporter - Already closed.
2010-04-29 21:26:26,017 [http-8080-2] ERROR servlet.GrailsDispatcherServlet - HandlerInterceptor.afterCompletion threw exception
org.hibernate.exception.GenericJDBCException: Cannot release connection
at java.lang.Thread.run(Thread.java:619)
Caused by: java.sql.SQLException: Already closed.
at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:84)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
... 1 more
Referring to this article, you have stale connections in your DBCP connections pool that are silently dropped by OS or firewall.
The solution is to define a validation query and do a sanity check of the connection before you actually use it in your application.
In grails this is actually done by modifying the grails-app/conf/spring/Resource.groovy file and add the following:
beans = {
dataSource(BasicDataSource) {
//run the evictor every 30 minutes and evict any connections older than 30 minutes.
minEvictableIdleTimeMillis=1800000
timeBetweenEvictionRunsMillis=1800000
numTestsPerEvictionRun=3
//test the connection while its idle, before borrow and return it
testOnBorrow=true
testWhileIdle=true
testOnReturn=true
validationQuery="SELECT 1"
}
}
In grails 1.3.X, you can modify the evictor values in the DataSource.groovy file to make sure pooled connections are used during idle. This will make sure the mysql server will not time out the connection.
production {
dataSource {
pooled = true
// Other database parameters..
properties {
maxActive = 50
maxIdle = 25
minIdle = 5
initialSize = 5
minEvictableIdleTimeMillis = 1800000
timeBetweenEvictionRunsMillis = 1800000
maxWait = 10000
}
}
A quick way to verify this works is to modify the MySQL my.cnf configuration file [mysql] element and add wait_time parameter with a low value.
Try increasing the number of open MySQL connections by putting the following in your DataSources.groovy:
dataSource {
driverClassName = "com.mysql.jdbc.Driver"
pooled=true
maxActive=10
initialSize=5
// Remaining connection params
}
If you want to go the whole hog, try implementing a connection pool; here is a useful link on this.
For grails 1.3.X, I had to add the following code to Bootstrap.groovy :
def init = {servletContext ->
def ctx=servletContext.getAttribute(ApplicationAttributes.APPLICATION_CONTEXT)
//implement test on borrow
def dataSource = ctx.dataSource
dataSource.targetDataSource.setMinEvictableIdleTimeMillis(1000 * 60 * 30)
dataSource.targetDataSource.setTimeBetweenEvictionRunsMillis(1000 * 60 * 30)
dataSource.targetDataSource.setNumTestsPerEvictionRun(3)
dataSource.targetDataSource.setTestOnBorrow(true)
dataSource.targetDataSource.setTestWhileIdle(true)
dataSource.targetDataSource.setTestOnReturn(false)
dataSource.targetDataSource.setValidationQuery("SELECT 1")
}
I also had to import org.codehaus.groovy.grails.commons.ApplicationAttributes
Add these parameters to dataSource
testOnBorrow = true
testWhileIdle = true
testOnReturn = true
See this article for more information
http://sacharya.com/grails-dbcp-stale-connections/
Starting from grails 2.3.6 default configuration already has options for preventing closing connection by timeout
These are the new defaults.
properties {
// See http://grails.org/doc/latest/guide/conf.html#dataSource for documentation
....
minIdle = 5
maxIdle = 25
maxWait = 10000
maxAge = 10 * 60000
timeBetweenEvictionRunsMillis = 5000
minEvictableIdleTimeMillis = 60000
validationQuery = "SELECT 1"
validationQueryTimeout = 3
validationInterval = 15000
testOnBorrow = true
testWhileIdle = true
testOnReturn = false
jdbcInterceptors = "ConnectionState;StatementCache(max=200)"
defaultTransactionIsolation = java.sql.Connection.TRANSACTION_READ_COMMITTED
}
What does your JDBC connection string look like? You can set an autoReconneect param in your data source config, e.g.
jdbc:mysql://hostname/mydb?autoReconnect=true