Is there a way to have an established connection pool on the client side (running as daemon), so it can be used by the mysql client on linux?
mysql ==(named pipe/unix domain socket?)==> mysql connection pool (daemon) ==> mysql server
After reading the response from you i can propose the following solution:
to have a daemon application (resident in memory) which will accept connections from clients (via sockets or http)
the clients will sent a security token (so that they can be authorized) and the query that needs to be executed
the daemon application can have a pool of mysql connections (fixed number) and it will choose a connection (depending on the load) to exec the queries and return the result (if necessary )
This way you will have full control on the number of mysql connections also you will have a single point where the communication with the db layer is made.
Related
we have mysql-server(5.5.47)that hosted on physical server. It listen external internet interface(with restrict user access), mysql server intensively used from different places(we use different libraries to communicate with mysql). But sometimes whole mysql server(or network) stuck and stop accept connection, and a clients failed with etimedout(connect)/timeout(recv), even direct connection from server to mysql with mysql cli not working(stuck without any response — seems to be try to establish connections).
First thought was that it is related to tcp backlog, so mysql backlog was increased — but this not help at all.
Issue not repeatable, so last time when this issue happened we sniff traffic, and what we get:
http://grab.by/STwq — screenshot
*.*.27.65 — it is client
*.*.20.80 — it is mysql server
From session we can assume that tcp connection established, but server retransmit SYN/ACK to client(from dump we see that server receive ACK, why retransmit ?), but in normal case mysql must generate init packet and send to client, after connection was established.
It is only screen from 1 session, but all other sessions mostly same, SYN -> SYN/ACK -> ACK -> and server retransmit SYN/ACK up to retries_count.
After restart mysql all get normal immediately after restart. So not sure it is related to network or mysql.
Any thoughts would be appropriate.
Thank you!
I'm a little confused around connection pools. I'm using Hibernate on top of a MySQL database. Hibernate's connection pool strategy is determined by c3p0.
What's the relationship between Hibernate's connection pool size vs MySQL's?
My code running Hibernate can be scaled to multiple instances on AWS (so n # of instances each with Hibernate connection pool size of m). All instances however talk to a single RDS MySQL instance, which itself has a connection pool size q.
Does this mean if there are n*m active connections and n*m>q, there will be connections that will have to wait in MySQL's queue?
Thanks!
Your question asks about "Hibernate's connection pool size vs MySQL's". The important distinction to keep in mind is:
A connection pool is an application concept, not a database one. The connection pool used by your application (i.e., Hibernate) is simply a way of speeding up the communication to the database. Since establishing each connection is relatively slow, it's more efficient to pre-establish a bunch of connections and then let your application use them as needed.
On the other hand, a database doesn't pool connections. Instead, it allows clients to establish connections upon request up to a max limit (e.g., the max_connections parameter in MySQL). If a client asks for a connection and the database has already hit the max limit, then a connection error will occur.
So to answer your question, if you configure your application connection pools to try to pre-establish more connections than the database will allow, you will get a "Too many connections" error from MySQL. This could be avoided by either raising the MySQL configuration parameter or by adjusting your c3p0 max pool size per instance.
We have over 500 processes in our service server, and each of them will keep a mysql connection to the mysql server.
If we keep over 500 connections to mysql server at the same time, is it going to affect the performance of mysql?
In my opinion, the performance won't be affect, since mysql server will keep a connection pool, and response to service server when there is a sql request.
For our situation, we will have a connection pool over 500, and I don't think 500 will be too much for mysql.
I don't know your configuration, but I know that this can be bad. If all connections are closed propper this won't be a problem, but If not it can happen that your server won't accept more connections until a timeout is reached which would result a big delay for your clients. If possible pool the connection on the client site too, this is always a good idea, so you don't need to wait that the tcp connection is established and the auth and so on.
In order to avoid the overhead of establishing a new connection each time a query needs fired against MySQL, there are two options available:
Persistent connections, whereby a new connection is requested a check is made to see if an 'identical' connection is already open and if so use it.
Connection pooling, whereby the client maintains a pool of connections, so that each thread that needs to use a connection will check out one from the pool and return it back to the pool when done.
So, if I have a multi-threaded server application expected to handle thousands of requests per second, and each thread needs to fire a query against the database, then what is a better option?
From my understanding, With persistent connections, all the threads in my application will try and use the same persistent connection to the database because they all are using identical connections. So it is one connection shared across multiple application threads - as a result the requests will block on the database side soon.
If I use a connection pooling mechanism, I will have all application threads share a pool of connections. So there is less possibility of a blocking request. However, with connection pooling, should an application thread wait to acquire a connection from the pool or should it send a request on the connections in the pool anyway in a round-robin manner, and let the queuing if any, happen on the database?
Having persistent connections does not imply that all threads use the same connection. It just "says" that you keep the connection open (in contradiction to open a connection each time you need one). Opening a connection is an expensive operation, so - in general - you try to avoid opening connections more often than necessary.
This is the reason why multithreaded applications often use connection pools. The pool takes care of opening and closing connections and every thread that needs a connection requests one from the pool. It is important to take care that the thread returns the connection as soon as possible to the pool, so that another thread can use it.
If your application has only a few long running threads that need connections you can also open a connection for each thread and keep this open.
Using just one connection (as you described it) is equal to a connection pool with the maximum size one. This will be sooner or later your bottleneck as all threads will have to wait for the connection. This could be an option to serialize the database operations (perform them in a certain order), although there are better options to ensure serialisation.
Update: The newer X Protocol supports asynchronous connections, and newer drivers like Node's can utilize this.
Regarding your question about should the application server wait for a connection, the answer is yes.
MySQL connections are blocking. When you issue a request from MySQL server over a connection, the connection will wait, idle, until a response is received from the server.
There is no way to send two requests on the same connection and see which returns first. You can only send one request at a time.
So, generally, a single thread in a connection pool consists of one client side connection (in your case, the application server is the client) and one server side connection (database).
Your application should wait for an available connection thread from the pool, allowing the pool to grow when it's needed, and to shrink back to your default number of threads, when it's less busy.
How are connections to mysql handled in Rails 3?
Do multiple connections to the website share the same mysql connection, or does it take a connection from a connection pool and then release it once the request has closed all connections to mysql?
If there are 10 front end servers all hitting a single db server, are there any issues here?
I' using Phusion passenger if that effects anything.
The doc answers by itself:
A connection pool synchronizes thread access to a limited number of
database connections. The basic idea is that each thread checks out a
database connection from the pool, uses that connection, and checks
the connection back in. ConnectionPool is completely thread-safe, and
will ensure that a connection cannot be used by two threads at the
same time, as long as ConnectionPool’s contract is correctly followed.
It will also handle cases in which there are more threads than
connections: if all connections have been checked out, and a thread
tries to checkout a connection anyway, then ConnectionPool will wait
until some other thread has checked in a connection.