I'm wondering how to properly determine what value to use for c3p0 max_statements. I've experienced some caching dead locks which seems to point back to my max_statements configuration based on all the SO Q&A I've read.
I'm using mysql and the deadlock appears to happen when I'm doing some multi threading where I have 4 active threads.
My configuration
<property name="hibernate.connection.provider_class">org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.min_size">10</property>
<property name="hibernate.c3p0.max_statements">50</property>
<property name="hibernate.c3p0.max_size">50</property>
<property name="hibernate.c3p0.idle_test_period">1800</property>
<property name="hibernate.c3p0.timeout">3600</property>
The exception
[WARN] async.ThreadPoolAsynchronousRunner com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#72df1587 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$StatementDestructionManager$1UncheckedStatementCloseTask#e877a61
on thread: C3P0PooledConnectionPoolManager[identityToken->1hge0wd9a1o1iea71i8u346|1a799bb]-HelperThread-#2
com.mchange.v2.c3p0.stmt.GooGooStatementCache$StatementDestructionManager$1UncheckedStatementCloseTask#109b1150
on thread: C3P0PooledConnectionPoolManager[identityToken->1hge0wd9a1o1iea71i8u346|1a799bb]-HelperThread-#0
com.mchange.v2.c3p0.stmt.GooGooStatementCache$StatementDestructionManager$1UncheckedStatementCloseTask#3eb42946
on thread: C3P0PooledConnectionPoolManager[identityToken->1hge0wd9a1o1iea71i8u346|1a799bb]-HelperThread-#1
Pending Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask#52729f95
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->1hge0wd9a1o1iea71i8u346|1a799bb]-HelperThread-#0,5,main]
com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:2765)
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:541)
com.mchange.v1.db.sql.StatementUtils.attemptClose(StatementUtils.java:53)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$StatementDestructionManager$1UncheckedStatementCloseTask.run(GooGooStatementCache.java:938)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
Thread[C3P0PooledConnectionPoolManager[identityToken->1hge0wd9a1o1iea71i8u346|1a799bb]-HelperThread-#1,5,main]
com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:2765)
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:541)
com.mchange.v1.db.sql.StatementUtils.attemptClose(StatementUtils.java:53)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$StatementDestructionManager$1UncheckedStatementCloseTask.run(GooGooStatementCache.java:938)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
Thread[C3P0PooledConnectionPoolManager[identityToken->1hge0wd9a1o1iea71i8u346|1a799bb]-HelperThread-#2,5,main]
com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:2765)
com.mysql.jdbc.StatementImpl.close(StatementImpl.java:541)
com.mchange.v1.db.sql.StatementUtils.attemptClose(StatementUtils.java:53)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$StatementDestructionManager$1UncheckedStatementCloseTask.run(GooGooStatementCache.java:938)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
So my question is how do I determine what these values should be. I'm sure there must be away to do this without guessing.
Articles I've read
should i activate c3p0 statement pooling?
How to trace and prevent the deadlock appeared in c3po which is running in seperate processes?
To resolve deadlocks associated with Statement caching under Oracle / jTDS / mySQL, please make sure you are using a recent c3p0 (0.9.5.1 is the current version), and please see statementCacheNumDeferredCloseThreads and Configuring Statement Pooling.
TL;DR set config param
<property name="hibernate.c3p0.statementCacheNumDeferredCloseThreads">1</property>
The exact value of max_statements is only incidentally associated with this issue. If max_statements is too small, you will churn through statements unnecessarily, and this issue, associated with the fragility of PreparedStatement.close(), in some drivers will appear more frequently.
However, your value for hibernate.c3p0.max_statements is too small for a pool of maxPoolSize 50. Even after you fix the deadlock issue, churning through statements will diminish or kill any performance benefit from the statement cache. To compute a good value for hibernate.c3p0.max_statements (which maps to c3p0.maxStatements), count the number of distinct PreparedStatements that are used frequently in your application and multiply that by maxPoolSize (or in your case hibernate.c3p0.max_size). Or, alternatively, just set hibernate.c3p0.maxStatementsPerConnection to the number of distinct PreparedStatements used frequently by your application.
Please see maxStatements, maxStatementsPerConnection, and Configuring Statement Pooling.
Related
I'm trying to run a few JDBC extracts in parallel, but this fails on: java.lang.OutOfMemoryError: Java heap space.
How does Data Connection memory usage work, and how do I resolve this problem?
The Data Connection Agent's memory usage here actually depends mostly on the value of the fetchSize parameter. Per the Oracle JDBC driver docs, fetchSize:
Gives the JDBC driver a hint as to the number of rows that should be fetched from the database when more rows are needed for this ResultSet object.
So, the agent's memory usage should roughly be:
number of JDBC extracts running in parallel x fetchSize x size of each row
Unfortunately, the default value of fetchSize varies vastly among different JDBC drivers. For instance, certain versions of the Hive JDBC driver have it set to 50, while other, newer versions have a default value of 1000. Oracle JDBC drivers have a default of 10. Postgres by default will try to get the entire ResultSet at once.
Hence, Data Connection allows you to configure the fetchSize value. This is configurable both per-source and per-extract.
OOM errors aside, tuning fetchSize can improve performance significantly in general. There isn't a one-size-fits-all solution, though, and you'll have to experiment to figure out the best parameter value for your extracts. It usually lies somewhere in the 500–5000 range.
I'm trying to use C3P0 library to handle connection pooling.
These are my C3P0 settings:
minPoolSize=3
maxPoolSize=20
acquireIncrement=1
maxIdleTime=240
maxStatements=20
In the log I can see that C3P0 seems to be correctly initialized by reading
INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource - Initializing c3p0 pool...
But when I try so see how many connection there are on my MySQL db with
SHOW STATUS WHERE `variable_name` = 'Threads_connected'
I can see that the result is 48, 46, 49 ecc.
I can't understand if is not correct the way I try to see how many connections there are on the db or I did't understand the way C3P0 works
I also faced such confusion in the MySQL for threads & connections. I will explain what I have learned and grasp while studying the same, if anything that I have misunderstood then or still in confusion then please make me correct.
Some basics in MySQL:
- MySQL server is a single process application.
- It is multithreaded.
- It accepts connections like TCP/IP server.
- Each connection gets a thread.
- These threads are sometimes named processes, and sometimes they're referred to as connections.
Last and second last point makes so much confusion, In our mind we think that there is a 1-1 mapping between connections and active threads. It true also but then, there is a thread pool, which means there can be threads which are not associated with any connection.
Every new connection gets its own thread. Every new connection makes new thread, and a disconnect calls for thread's destruction. So, there is a 1-1 mapping between connections and active threads. After destruction of threads it may go into the thread pool. So, the number of threads is greater than or equal to the number of connections.
Also If you run below query
SELECT t.PROCESSLIST_ID,IF (NAME = 'thread/sql/event_scheduler','event_scheduler',t.PROCESSLIST_USER) PROCESSLIST_USER,t.PROCESSLIST_HOST,t.PROCESSLIST_DB,t.PROCESSLIST_COMMAND,t.PROCESSLIST_TIME,t.PROCESSLIST_STATE,t.THREAD_ID,t.TYPE,t.NAME,t.PARENT_THREAD_ID,t.INSTRUMENTED,t.PROCESSLIST_INFO,a.ATTR_VALUE FROM performance_schema.threads t LEFT OUTER JOIN performance_schema.session_connect_attrs a ON t.processlist_id = a.processlist_id AND (a.attr_name IS NULL OR a.attr_name = 'program_name') WHERE 1=1
Then you will see column TYPE in that values are either FOREGROUND or BACKGROUND so this tells there can be some threads which are connected with DB to do some (background) work (eg. event thread, monitor thread etc.).
generally, c3p0 concerns about connection and not for threads, so you should check for SHOW FULL PROCESSLIST for connections with DB server.
I hope I have cleared the confusion which you are having with MySQL threads & connections.
I am trying to run a query from our application through springs 2.5 and hibernate 3,But query is neither timed out nor returning results (query hangs out),When I run the same query from the query browser it is working fine.
Even though I increased query timeout still I am not able to fetch the result.
I tried to increase query execution timeout it failed to return records
<property name="javax.persistence.query.timeout" value="3000" />
<tx:advice id="defaultTxAdvice" transaction-manager="transactionManager">
<tx:attributes>
<!-- Keep SequenceService in a isolation transaction -->
<tx:method name="get*" read-only="true" />
<!-- By default, A runtime exception will rollback transaction. -->
<tx:method name="*" timeout="100" rollback-for="ApplicationException" />
</tx:attributes>
</tx:advice>
Any kind of help/suggestion would be greatly appreciated
These are my thoughts about the issue here.Since i don't find any relevant codes here..
Due to execution of multiple queries in single fetch (issues like n+1 issue in hibernate). Hint:Make sure that you set show SQL value as true in developer machine
Due to irregular handling of queries in code, like if you are iterating for n number of time. The Database server will consider its as some threat and it moves to a lock state and stop execution of queries for some time period.
Improper handling of transaction time-out, Like if you are increasing a transaction time-out value then it will probably lock the database for a long time.
There are also other possibilities, but i cant make any suggestions unless i see the code.
We are getting exceptions like this
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector#5b7a7896 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#55bc5e2a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#41ca435f (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask#460d33b7 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
when load testing our application on MSSQL 2008 R2 (jTDS or official MS JDBC doesn't matter). We never get this exception when running the same tests against PostgreSQL or MySQL.
We don't just want to increase the number of helper threads for c3p0 (which solves the problem, but how long?). We want to know what's the problem as it is workings with other DBMS'.
The applications behaves like:
Send X requests
Wait for a while -> DEADLOCK
Send X requests
Wait for a while -> DEADLOCK
Does anyone know or has an idea why we have this behavior with MSSQL?
Thanks, Adrian
(Btw. BoneCP works without any problem too.)
SQL Server has a much more restrictive locking strategy compared to PostgreSQL or InnoDB.
Especially it will block SELECTs on rows (tables?) that are updated from a different connection/transaction (in the default installation).
You should make sure that you are not selecting the same rows in one session that are being updated from another.
If you can't change the sequence of your code, you might get away with using "dirty reads" in SQL Server.
If I remember that correctly, this is accomplished by adding WITH NOLOCK to the SELECT statements (but I'm not entirely sure)
Edit
A different possibility (if you are on SQL Server 2005 or later) would be to use the new "snapshot isolation" to avoid blocking selects.
In our applications we don't use either ADO.NET transaction or SQL Server transactions in procedures and now we are getting the below error in our website when multiple people are using.
Transaction (Process ID 73) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction
Is this error due to the lack of transactions? I thought the consistency will be handled by the DB itself.
And one thing I noticed that SQLCommand.Timeout property has set to 10000. Will this be an issue for the error?
I am trying to solve this issue ASAP. Please help.
EDIT
I saw the Isolationlevel property of ADO.NET transaction, so if I use ADO.NET transaction with proper isolationlevel property like "ReadUncommitted" during reading and "Serializable" during writing?
Every SQL DML (INSERT, UPDATE, DELETE) or DQL (SELECT) statement runs inside a transaction. The default behaviour for SQL Server is for it to open a new transaction (if one doesn't exist), and if the statement completes without errors, to automatically commit the transaction.
The IMPLICIT_TRANSACTIONS behaviour that Sidharth mentions basically gets SQL Server to change it's behaviour somewhat - it leaves the transaction open when the statement completes.
To get better information in the SQL Server error log, you can turn on a trace flag. This will then tell you which connections were involved in the deadlock (not just the one that got killed), and which resources were involved. You may then be able to determine what pattern of behaviour is leading to the deadlocks.
If you're unable to determine the underlying cause, you may have to add some additional code to your application - that catches sql errors due to deadlocks, and retries the command multiple times. This is usually the last resort - it's better to determine which tables/indexes are involved, and work out a strategy that avoids the deadlocks in the first place.
IsolationLevel is your best bet. Default serialization level of transactions is "Serializable" which is the most stringent and if at this level there is a circular reference chances of deadlock are very high. Set it to ReadCommitted while reading and let it be Serializable while writing.
Sql server can use implicit transactions which is what might be happening in your case. Try setting it off:
SET IMPLICIT_TRANSACTIONS OFF;
Read about it here: http://msdn.microsoft.com/en-us/library/ms190230.aspx