I'm facing an load issue on MySql Server database.
In our application we have an requirement of inserting ,fetching and updating data for each call/click.
We are doing the load testing on this application, we wrote an program to call this application do it functionality. we running this test program for more than 400 times (I mean 400 instances) in MySql more than 100 connections are created and test program was throwing "The underlying provider failed on Open. -Too many connections"
I have set the max_connection variable to 400 and did the same testing but this time it was throwing "Thread stack overrun: 137376 bytes used of a 262144 byte stack, and 128000 bytes needed. Use 'mysqld --thread_stack=#' to specify a bigger stack."
Can anyone help me please.
I have resolved this issue by doing small modification in web.config and mysql variable setting
1) <httpRuntime executionTimeout = "number(in seconds)" />
this is used, to remove the request when it take's more time in process the request
2) By increasing the max_connections variable size in mysql
3) there was issue in the load testing application itself , it was sending the request without any time gap, so set the time gap between each request.
Now I can run up to 600 users instance at a same time , but load will be on CUP and MySql.
Thank you Kay Nelson for your quick reply.
Regards,
Sampath Kumar J
Related
I have an application deployed on GAE having endpoints. Each endpoint make a connection with database , get data and close connection and return data. Normally everything works fine but when there is hike in requests it starts taking more than 60 sec and requests get aborted. Due to this it does not close database connection and there mysql got 1000+ connections and then each requests starts aborting and it shows deadline exceeded error. Is there any solution for this ?
You could wrap the "get data" portion with a try... finally... statement and move the "close connection" portion in the finally section. Then start an "about to exceed deadline" timer before "get data" (something like say 45 seconds) and raise an exception if the timer expires, allowing you to close the connection in the finally portion, which should take care of the orphan open connections (but would not prevent errors in those requests).
If your application tolerates it you could also look into using task queues which have a 10 min deadline, which could help reducing/eliminating the errors in the requests as well.
You can also find some general advice for addressing deadline exceeded errors here: https://cloud.google.com/appengine/articles/deadlineexceedederrors, donno if applicable to your app.
EDIT: actually the suggestion in the first paragraph above doesn't work on GAE as the Python sandbox doesn't allow installing a custom signal handler:
signal.signal(signal.SIGALRM, timer_expired)
AttributeError: 'module' object has no attribute 'signal'
After seeing your code a somehow equivalent solution would be to replace your cursor.fetchall() with a loop of cursor.fetchone() or cursor.fetchmany() to split your operation in smaller pieces:
http://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-fetchone.html. You'd get a start timestamp (with time.time() for example) when entering your request handler. Then inside the loop you'd get another timestamp to measure the time elapsed so far since the start timestamp and you'd break out of the loop and close the DB connection when deadline expiration nears. Again, this won't help with actually replying successfully to the requests if it takes so much time to prepare the replies.
You can use this solution to close connections when deadlines are exceeded:
Dealing with DeadlineExceededErrors
This way you won't have any open connections hanging there forever.
Think about the design of your application -
1.Use the deadline exception handling - Design smells
Because there will be situation(s) where db operation takes more than 60 seconds , If its a simple query then its well and good , but reconsider the design of the application . User Expierence is going to be hurt.
2.Lets change the design to use the endpoints-
https://cloud.google.com/appengine/docs/java/endpoints/
The way to go ,future proof.
3.Using Back-end or Task-queues as descibed in this post
Max Time for computation on Google App Engine
You can set the Timeouts
interactive_timeout
and / or
wait_timeout
based on connection Type they use one of them
My simple question:
How can I increase the possible number of connections of my Amazon RDS Database?
I used a parameter group where I set
max_connections = 30000
which seems to work on the first hand, as
SHOW VARIABLES LIKE 'max_connections';
returns the expected.
But when I run a stress test the monitoring metrics always show a maximum number of 1200 connections.
So obviously there have to be other limiting factors, I just don't know.
Any help would be highly appreciated.
My test setup:
1 Load Balancer
8 fat EC2 instances (m4.4xlarge) (which is a bit overdimensioned, but I'm still testing)
1 DB: r3.4xlarge with 140 GB memory, 1 TB storage and 10.000 provisioned IOPS
Test: 30.000 virtual users in 10 minutes making 4 requests each (2 reading the DB, 1 writing it, 1 not using the DB).
Fails after about two minutes because of too many errors (caused by DB timeouts).
Concerning the hardware this setup should be able to handle the test requests, shouldn't it?
So I hope I'm just missing the obvious and there's a parameter which has to be adapted to make everything working.
I would strongly suggest that the first problem is not with the configuration of the server, but with your test methodology and interpretation of what you are seeing.
Hitting max_connections does not initially cause "db timeouts." It causes connection errors, because the server actively rejects excessive connection attempts, with a refusal to negotiate further. This is not the same thing as a timeout.
At what point, during what operation, are the timeouts occurring? Initial connection phase? That's not going to be related to max_connections, at least not directly.
The maximum connections you observe seems like a suspiciously round number and potentially is even derivable from your test parameters... You mentioned 30000 users and 10 minutes and 4 requests... and 30000 × 4 ÷ 10 ÷ 10 = 1200. Yes, I threw in the "10" twice for no particular reason other than 1200 just seems very suspicious. I wonder whether, if you used 15000 users, the number would drop from 1200 to 600. That would be worth investigating.
Importantly, to serve 30000 concurrent users, your application does not need 30000 database connections. If it does, it's written very, very badly. I don't know how you're testing this, but only a naive implementation given the stated parameters would assume 30000 connections should be established.
Equally important, 30000 connections to a single MySQL server regardless of size seems completely detached from reality, except maybe with thread pooling, which isn't available in the version of MySQL used in RDS. If you were to successfully create that many connections, on a cold server or one without a massive thread cache already warmed up, it would likely take several minutes just for the OS to allow MySQL to create that many new threads. You would indeed see timeouts here, because the OS would not let the server keep up with the incoming demand, but it would be unrelated to max_connections.
It would seem like your most likely path at this point would not be to assume that max_connections isn't actually set to the value that it claims, and to scale down your test parameters, see how the behavior changes and go from there in an effort to understand what is actually happening. Your test parameters also need to be meaningful related to the actual workload you're trying to test against.
Thanks to Michael and the hints of a colleague I was finally able to solve this problem:
As Michael already supposed it wasn't caused by the DB.
The answer was hidden in the Apache configuration which I took under examination after DB problems seem to be out of question (finally).
All my eight EC2 instances were limited by MaxRequestWorkers=150 (-> 8*150=1200).
What is obvious for every holiday admin took me day.
At least everything's working now.
I am using WinInet in C / C++ application to connect to a ASP.NET Web Service.
I want to increase my SESSION TIMEOUT time.
Currently somehow SESSION Time out is 20 minutes and I want to increase it to 50 minutes.
Which option do it use for the option INTERNET_OPTION_XXXXX in
InternetSetOption(hInstance, INTERNET_OPTION_XXXXX,(LPVOID) &timeout, sizeof(timeout));
Unlike WinHTTP that has WinHttpSetTimeouts there is no equivalent function available at WinINet.
I realise this is an old question however it seems there is no info on how to do this on SO. So I am posting this in case someone wants to know how to set timeouts with WinINet.
Normally you would use INTERNET_OPTION_CONNECT_TIMEOUT,INTERNET_OPTION_RECEIVE_TIMEOUT, or INTERNET_OPTION_SEND_TIMEOUT with InternetSetOption. See here for details on the option flags: https://learn.microsoft.com/en-us/windows/win32/wininet/option-flags
However, there is a bug which it seems MS has not fixed in about 20 years. The above timeout flags simply don't work.
So the way to work around this is to create second worker thread to watch for the connection request. The second request will kill the main connection request if it doesn't receive response from server in the timeout set. See this MS KB article for details and example:
https://mskb.pkisolutions.com/kb/224318
I am currently building a client without using any libraries just to understand the protocol really, and I am confused by an access denied reply when sending my computed Auth packet to the MySQL Server. The Mysql server is just a local server running on my computer for testing purposes. Here is the information I am sending:
The test password is 'peanut'.
Stage 1 Hash = b14ab480028768cb748fd97de56144a304eb8a1a
Stage 2 Hash = fd62797ed464c2843942a9167cc0521779d68862 - This is correct but without the * in the database.
Salt & Stage 2 Hash = rC8/$a?Vr\W|.jN)~cVcfd62797ed464c2843942a9167cc0521779d68862
SHA1(Salt + Stage 2 Hash) XOR Stage 1 Hash = 4B19199ECEB929469EA89C0E942D8D5B9ACBE237
String Sent to Server For Authentication in hex:
\x3A\x00\x00\x01 - Standard Header (Payload length / Sequence nUmber)
\x02\x04\x80\x00 - Compatibility flags
\x00\x00\x00\x01 - Maximum packet size
\x08 - Charset
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 - 23 byte filler
\x72\x6F\x6F\x74\x00 - username zero terminated
\x14 - length of token (20 bytes)
\x4B\x19\x19\x9E\xCE\xB9\x29\x46\x9E\xA8\x9C\x0E\x94\x2D\x8D\x5B\x9A\xCB\xE2\x37 - token
When I send this string, the server just sends out the #28000 error that is access denied.
Could this be an access rights issue? A remote user trying to gain root access, is there something i need to enable?
I have changed the connection timeout settings wait_timeout / connect_timeout etc. and still no joy, these are set to 60 Secs.
I am not sure if I should be computing the SHA1(Salt + Stage 2 Hash) with an asterisk or not, as in the database it shows an * before the password. I have tried both ways and it still doesn't auth.
I am running out of ideas now, the only other thing I can think to do is to write another program which will process the client token as the Mysql Server would, but I thought I would double check here first.
I have been working on this for a while now and am stumped.
Any help greatly appreciated. I don't normally post on Forum's so its a new experience, sorry if I haven't followed etiquette.
Regards
James
Without looking into the protocol documentation myself it is hard to see any errors here. It could be anything from endianness to wrong padding, lengths and other calculations etc. The server rightfully does not provide more details as to why the login might have failed, because that would be leaking information to a potential attacker.
I suggest you grab the MySQL source code and look at how they do it in their command line application. See http://dev.mysql.com/doc/internals/en/guided-tour.html for some introduction and download links.
Apart from that, there is always the option of logging the network traffic with Wireshark to see what the stock mysql client sends and compare that to what you have. That might help to figure out padding etc.
Note that if you are working on Windows there might be problems to capture traffic on localhost, because IIRC Windows shortcuts the traffic somehow, so it does not get past the network interface for Wireshark to see. In that case you might have to set up the MySQL server in a VM or on a different machine.
One of the more interesting "features" in Coldfusion is how it handles external requests. The basic gist of it is that when a query is made to an external source through <cfquery> or or any other external request like that it passes the external request on to a specific driver and at that point CF itself is unable to suspend it. Even if a timeout is specified on the query or in the cfsetting it is flatly ignored for all external requests.
http://www.coldfusionmuse.com/index.cfm/2009/6/9/killing.threads
So with that in mind the issue we've run into is that somehow the communication between our CF server and our mySQL server sometimes goes awry and leaves behind hung threads. They have the following characteristics.
The hung thread shows up in CF and cannot be killed from FusionReactor.
There is no hung thread visible in mySQL, and no active running query (just the usual sleeps).
The database is responding to other calls and appears to be operating correctly.
Max connections have not been reached for the DB nor the user.
It seems to me the only likely candidate is that somehow CF is making a request, mySQL is responding to that request but with an answer which CF ignores and continues to keep the thread open waiting for a response from mySQL. That would explain why the database seems to show no signs of problems, but CF keeps a thread open waiting for the mysterious answer.
Usually these hung threads appear randomly on otherwise working scripts (such as posting a comment on a news article). Even while one thread is hung for that script, other requests for that script will go through, which would imply that the script isn't neccessarily at fault, but rather the condition faced when the script was executed.
We ran some test to determine that it was not a mysql generated max_connections error... we created a user, gave it 1 max connections, tied that connection with a sleep(1000) query and executed another query. Unfortunately, it correctly errored out without generating a hung thread.
So, I'm left at this point with absolutely no clue what is going wrong. Is there some other connection limit or timeout which could be causing the communication between the servers to go awry?
One of the things you should start to look at is the hardware between the two servers. It is possible that you have a router or bridge or NIC that is dropping occasional packets. This can result in the mySQL box thinking it has completed the task while the CF server sits there and waits for a complete response indefinitely, creating a hung thread.
3com has some details on testing for packet loss here: http://support.3com.com/infodeli/tools/netmgt/tncsunix/product/091500/c11ploss.htm#22128
We had a similar problem with a MS SQL server. There, the root cause was a known issue in which, for some reason, the server thinks it's shutting down, and the thread hangs (even though the server is, obviously, not shutting down).
We weren't able to eliminate the problem, but were able to reduce it by turning off pooled DB connections and fiddling with the connection refresh rate. (I think I got that label right -- no access to administrator at my new employment.) Both are in the connection properties in Administrator.
Just a note: The problem isn't entirely with CF. The problem, apparently, affects all Java apps. Which does not, in any way, reduce how annoyed I get by this.
Long story short, but I believe the caused was due to Coldfusion's CF8 image processing. It was just buggy and now in CF9 I have never seen that problem again.