mysql .net connector connection pool wasted - mysql

Question

I have asp.net mvc 3 application using mysql database and mysql .net connector. Application is using membership, role, profile providers and subsonic3 as data layer. Application is able to waste very big number of connections and eventually it crashes with timeout exception. I cloned server/application setup and I managed to reproduce issue with 10 connections limit. Info I have for now is below. For the beginning what does "Cleaning up" in innoDb transaction status means? I found this info: http://dev.mysql.com/doc/refman/5.0/en/general-thread-states.html but I dont see how transaction can stay in that status. Off course I desperately need any info that can help.
I actually debugged Subsonic code and I don't see that its doing anything wrong. When I become more desperate I guess I will do it again. Now I am trying to see whats happening in connector. Connections listed below are wasted i.e. not reusable.
Connections:
mysql> SHOW FULL PROCESSLIST;
+----+------+-----------------+------------+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+------+-----------------+------------+---------+------+-------+-----------------------+
| 2 | root | localhost:49167 | NULL | Query | 0 | init | SHOW FULL PROCESSLIST |
| 15 | root | localhost:49360 | somedbname | Sleep | 260 | | NULL |
| 16 | root | localhost:49361 | NULL | Sleep | 260 | | NULL |
| 19 | root | localhost:49437 | somedbname | Sleep | 3969 | | NULL |
| 20 | root | localhost:49439 | somedbname | Sleep | 3702 | | NULL |
| 21 | root | localhost:49440 | somedbname | Sleep | 3396 | | NULL |
| 22 | root | localhost:49457 | somedbname | Sleep | 3102 | | NULL |
| 23 | root | localhost:49460 | somedbname | Sleep | 2802 | | NULL |
| 24 | root | localhost:49478 | somedbname | Sleep | 1929 | | NULL |
| 26 | root | localhost:49497 | somedbname | Sleep | 1629 | | NULL |
| 27 | root | localhost:49498 | somedbname | Sleep | 1329 | | NULL |
+----+------+-----------------+------------+---------+------+-------+-----------------------+
11 rows in set (0.00 sec)
InnoDb status:
=====================================
2013-02-13 07:54:01 790 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 46 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 311 srv_active, 0 srv_shutdown, 14316 srv_idle
srv_master_thread log flush and writes: 14623
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 300
OS WAIT ARRAY INFO: signal count 296
Mutex spin waits 207, rounds 6140, OS waits 111
RW-shared spins 200, rounds 6000, OS waits 172
RW-excl spins 1, rounds 480, OS waits 15
Spin rounds per wait: 29.66 mutex, 30.00 RW-shared, 480.00 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 7160
Purge done for trx's n:o < 7157 undo n:o < 0 state: running but idle
History list length 650
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 7159, not started
MySQL thread id 27, OS thread handle 0xb6c, query id 11259 localhost ::1 root cleaning up
---TRANSACTION 7124, not started
MySQL thread id 26, OS thread handle 0xc88, query id 11080 localhost ::1 root cleaning up
---TRANSACTION 0, not started
MySQL thread id 2, OS thread handle 0x790, query id 11270 localhost ::1 root init
SHOW ENGINE INNODB STATUS
---TRANSACTION 7005, not started
MySQL thread id 24, OS thread handle 0xde0, query id 10510 localhost ::1 root cleaning up
---TRANSACTION 6865, not started
MySQL thread id 23, OS thread handle 0x1d0, query id 9615 localhost ::1 root cleaning up
---TRANSACTION 6697, not started
MySQL thread id 22, OS thread handle 0x874, query id 8824 localhost ::1 root cleaning up
---TRANSACTION 6647, not started
MySQL thread id 21, OS thread handle 0xfa8, query id 8546 localhost ::1 root cleaning up
---TRANSACTION 6531, not started
MySQL thread id 20, OS thread handle 0x910, query id 8019 localhost ::1 root cleaning up
---TRANSACTION 6243, not started
MySQL thread id 19, OS thread handle 0x740, query id 6886 localhost ::1 root cleaning up
---TRANSACTION 0, not started
MySQL thread id 15, OS thread handle 0x75c, query id 11268 localhost 127.0.0.1 root cleaning up
--------
FILE I/O
--------
I/O thread 0 state: wait Windows aio (insert buffer thread)
I/O thread 1 state: wait Windows aio (log thread)
I/O thread 2 state: wait Windows aio (read thread)
I/O thread 3 state: wait Windows aio (read thread)
I/O thread 4 state: wait Windows aio (read thread)
I/O thread 5 state: wait Windows aio (read thread)
I/O thread 6 state: wait Windows aio (write thread)
I/O thread 7 state: wait Windows aio (write thread)
I/O thread 8 state: wait Windows aio (write thread)
I/O thread 9 state: wait Windows aio (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
1017 OS file reads, 3059 OS file writes, 2067 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 17393, node heap has 1 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 2556460
Log flushed up to 2556460
Pages flushed up to 2556460
Last checkpoint at 2556460
0 pending log writes, 0 pending chkp writes
852 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 8585216; in additional pool allocated 0
Dictionary memory allocated 142202
Buffer pool size 512
Free buffers 255
Database pages 256
Old database pages 0
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0 single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 988, created 63, written 1772
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 256, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Main thread id 1240, state: sleeping
Number of rows inserted 49, updated 955, deleted 0, read 5238
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
Web app gets stuck with following exception and isnt usable until app pool is recycled.
System.Configuration.Provider.ProviderException: An exception occurred.
Please check the Event Log. ---> MySql.Data.MySqlClient.MySqlException:
error connecting: Timeout expired.
The timeout period elapsed prior to obtaining a connection from the pool.
This may have occurred because all pooled connections were in use and max
pool size was reached.

Answer 1

OK I will provide some debugging steps that can be reused in similar situations to wash myself a little.
I did upgrade to Version=6.6.5.0. I downloaded source and attached debugger to connector and pool was working all right but I still had same issue. Connections from pool were not reused.
I added watch to private MySqlPool.inUsePool; And saw that all stuck connections are in use there. By adding another watch (in fact 10 of them): inUsePool[0-10].reader.Command.CommandText
helped me identify part in code that wasn't closing reader/connection. All connections that were stuck were indeed occupied by my reader. All had same SQL Command text that is called only once in application.