I read a book about MySQL. In the section on threads, I saw this statement:
MySQL Server is not process-based but thread-based
Then it explains about foreground and background threads. I want to know what that means exactly.. please check my thinking..
My assumption:
process-based: can run two or more same programs
thread-based: can run two or more threads in one process (program)
MySQL Server can't run two or more - only one. Foreground or background threads can be multiple.
So MySQL Server is thread-based..?
MySQL can run only one process. but multiple threads can be created.
If the user requests a query from MySQL, the client thread is created.
This thread will return to the thread pool when your session ends.
If there is already enough threads in the thread pool, this thread will be lost.
This value follows the value of the thread_cache_size parameter.
Related
I'm currently using AWS lambda to connect to an AWS mysql RDS. I'm creating a pool like such
pool = mysql.createPool({
host : 'host-details',
user : 'username',
password : 'password',
database : 'db'
});
Now when users of the app 'do something' it basically just connects via my code, grabs data out of the db table and then releases the connection. So let's say there are 10 users and they all simultaneously 'do something' Does that mean:
There will be 10 connections inside this pool and as soon as the connections release that connection count goes back to 0? If so does that mean this one pool (even if it had a limit of 50 connections) could support thousands of users as the DB queries last only few milliseconds?
Looking at the RDS monitoring in AWS, there is a metric for "DB connections (count)". Does the above scenario mean it would stay at 1 connection because it's 1 pool, or would it spike to 10 connections?
From my understanding, if my connection limit is 8 for the pool, and the above scenario occurs, the other 2 connections will queue. What stops me from setting the connection limit extremely high, is this just a factor of what the database can handle? ie if I see my DB memory/cpu performance starting to get into trouble, do I scale the pool connection limit back? and vice versa.
To cap this off, I'm trying to understand how this all works so I can setup my database / code properly so things don't break once it starts to scale.
Thank you.
Assuming you created the pool outside the Lambda handler, each instance of the function will create its own pool. Whenever a request comes in for a function, if there's no idle instance of the function, a new instance is created. Each instance creates its separate pool. A single instance can handle several invocations 1 at a time. All invocations of a single instance share the pool.
Answer to Q1:
Yes, that 1 pool can support 1000s of users if all the invocations triggered by those users end up being handled by a single instance of the function (1 at a time).
Answer to Q2:
The DB connections count metric of RDS would spike to 10. It counts the number of connections. It doesn't understand whether those connection originated in a pool.
Answer to Q3:
Set the connection limit based on your database instance size (CPU / memory) & how much traffic you expect to your Lambdas. So if your database can handle 200 connections at a time & you expect 10 instances of 1 Lambda to be running concurrently (& you only have 1 Lambda in total), set the pool size to 200/10 = 20. This is a very simplistic calculation. Many other factors like query duration would affect this.
For a pictorial view of all this, see my blog post.
Too many connection problem in laravel5.8 application
You can see there 54k+ connection in mysql and 32 is in used only
how to remove unused connection so my application work fast.
Neither 54K connections since startup, nor a max of 32 connections simultaneously doing something, is "too many".
What is the real problem? Sluggishness? Find the slowest queries and let's work on speeding them up. Run SHOW FULL PROCESSLIST to see if any queries have been running for more than a few seconds; they are a prime candidate for optimizing. Or use the slowlog.
Connection is just a "count" of attempted connections. It does not relate to active connections nor max_used_connections.
Run the following commands simultaneously:
SHOW VARIABLES LIKE 'max_connections'
SET GLOBAL max_connections = 1000000;
Connection is just a "count" of attempted connections. It does not relate to active connections nor max_used_connections.
See MySQL show status - active or total connections?
If you really are having many current open connections, you should look into what these connections are. You might have a sub-optimal query in your code or a bot is spamming an open endpoint.
You can see process list by running the query
show processlist;
You can then kill connections for the short term solution or take care of whatever problem was causing the connections in the first place.
If you really do need that many connections (doubt it), you should look into scaling your database instance, e.g. by adding read replicas.
I'm trying to use C3P0 library to handle connection pooling.
These are my C3P0 settings:
minPoolSize=3
maxPoolSize=20
acquireIncrement=1
maxIdleTime=240
maxStatements=20
In the log I can see that C3P0 seems to be correctly initialized by reading
INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource - Initializing c3p0 pool...
But when I try so see how many connection there are on my MySQL db with
SHOW STATUS WHERE `variable_name` = 'Threads_connected'
I can see that the result is 48, 46, 49 ecc.
I can't understand if is not correct the way I try to see how many connections there are on the db or I did't understand the way C3P0 works
I also faced such confusion in the MySQL for threads & connections. I will explain what I have learned and grasp while studying the same, if anything that I have misunderstood then or still in confusion then please make me correct.
Some basics in MySQL:
- MySQL server is a single process application.
- It is multithreaded.
- It accepts connections like TCP/IP server.
- Each connection gets a thread.
- These threads are sometimes named processes, and sometimes they're referred to as connections.
Last and second last point makes so much confusion, In our mind we think that there is a 1-1 mapping between connections and active threads. It true also but then, there is a thread pool, which means there can be threads which are not associated with any connection.
Every new connection gets its own thread. Every new connection makes new thread, and a disconnect calls for thread's destruction. So, there is a 1-1 mapping between connections and active threads. After destruction of threads it may go into the thread pool. So, the number of threads is greater than or equal to the number of connections.
Also If you run below query
SELECT t.PROCESSLIST_ID,IF (NAME = 'thread/sql/event_scheduler','event_scheduler',t.PROCESSLIST_USER) PROCESSLIST_USER,t.PROCESSLIST_HOST,t.PROCESSLIST_DB,t.PROCESSLIST_COMMAND,t.PROCESSLIST_TIME,t.PROCESSLIST_STATE,t.THREAD_ID,t.TYPE,t.NAME,t.PARENT_THREAD_ID,t.INSTRUMENTED,t.PROCESSLIST_INFO,a.ATTR_VALUE FROM performance_schema.threads t LEFT OUTER JOIN performance_schema.session_connect_attrs a ON t.processlist_id = a.processlist_id AND (a.attr_name IS NULL OR a.attr_name = 'program_name') WHERE 1=1
Then you will see column TYPE in that values are either FOREGROUND or BACKGROUND so this tells there can be some threads which are connected with DB to do some (background) work (eg. event thread, monitor thread etc.).
generally, c3p0 concerns about connection and not for threads, so you should check for SHOW FULL PROCESSLIST for connections with DB server.
I hope I have cleared the confusion which you are having with MySQL threads & connections.
Mysql Query Results
some times a get max_user_connections exceeds and ispite of this i can make a connection and other thing is how many currently connections being used . how do i know that please help.
The ‘max_used_connections’ variable defines the total number of connections that have been in use simultaneously since the server was started.
This variable is different when compared to the variable ‘max_connections’.
The system variable ‘max_connections’ defines the maximum permitted number of simultaneous connections by a client.
You can find the maximum connections after which the connections will be refused by using the formula (Max_used_connections/max_connections). You can refer various monitoring services like MONyog which defines these terms well.
There is a option named "unbuffered" of mysql client and a very simple line about it, "Flush the buffer after each query.", in mysql manual.
My question is what is its usage?
I try to read mysql source code and it may be the option "flush mysql client log/output buffer after each query", but I'm not sure.
Thanks.
The default behavior for the db is to buffer your query before outputting any info. If you run an unbiffered query, you are asking mysql to start the output as soo as possible. Theoretically this only stores one row in memory so you can stream huge tables without running out of memory.
Downsides are you cannot run 2 unbuffered queries at the same time. Whereas buffered queries will jjst enqueue the second one, unbuffered statements will throw an error.
Another downside is that you don't know how many rows are left until you end iterating through the redult.