Under high traffic my mysql 5.0.45 server /Apache2/ CentOS 5 is getting "Error establishing mySQL database connection". I need to find the root cause.
I would very much appreciate any pointer to information about the procedure I should take to find the cause (memory limit, thread limits, CPU load, slow queries etc, large dataset, wrong keys ...) I would assume it involves looking at relevant log files etc....
Thank you.
That particular error message sounds like it's being generated by your application, and not by a system library. MySQL has functionality to report the specific errors that are occurring, so your best bet would be to utilize that in some way.
For instance, if you were using PHP, there is a function called mysql_error() that returns specifics about the last error encountered (too many connections, etc). You would put in some error handling near your connection call, and log the mysql_error() results if it failed.
You didn't mention what language you were using, but the MySQL libraries would provide the same functionality to whichever you are using. I'd suggest modifying your application code to take advantage of it.
I'm willing to bet this is because you're hitting the max user limit allowed by the mysql server but in general, do print the mysql errors, if not to the screen but at least to the log, or email.
Related
I'm developing a website in php and codeignitor with three collegues, we're using mysql database.
I know that insert can throw an exception due to constraint violation, connect the server can make exception too if the server is busy.
Now what are other exceptions that might occur ? I tried looking in the web and I'm surprised I didn't find what I want, My webapp is a link-sharing website with tags, votes, flags,comments, and search(by title and tags, no advanced search yet) .
PS
Obviously we're not going to handle errors(like bad sector) so exceptions is what we want here.
Other common errors are:
The various php-generated catchable fatal errors. See here. http://php.net/manual/en/errorfunc.constants.php
php's out of memory error, which you cannot catch.
php's maximum execution time error, also which you cannot catch.
all sorts of MySQL errors.
Many web application software developers create a last-chance error handler. It logs the error message and any available stack trace to a log file and presents a "sorry, that didn't work" page to the user.
As you might guess, it's best not to use MySQL to log errors, because if it's MySQL failing, it won't work.
This is a community wiki page. That means anybody can edit it.
I have a Yii application with many concurrent console jobs writing to one database. Due to the high concurrency sometimes I get MySQL deadlock errors. Sometimes these can be too many. The console.log file becomes too big, and it translates to more expenses.
I want to prevent logging of specific CDbException instances, or at least suppress them altogether (I am handling the exceptions and can generate more compact log sentences from there).
YII__DEBUG is already commented out.
Can anyone please help me figure out how to do this?
Thanks a lot!!
Regards.
I decided to modify the log statement in yii/framwework/db/CDbCommand.php that was logging the failed SQL. I converted it into a trace statement:
Yii::trace(Yii::t('yii','CDbCommand::{method}() failed: {error}. The SQL statement executed was: {sql}.', array('{method}'=>$method, '{error}'=>$message, '{sql}'=>$this->getText().$par)),CLogger::LEVEL_ERROR,'system.db.CDbCommand');
I am anyway catching the exception and logging a more compact version of the sentence, so it is OK for me to do it.
This was the easiest way I could find. We don't upgrade Yii very often, so if and when we go to the next version I'll probably repeat the change.
A few weeks ago, I post a question about queuing database access request to prevent 'too many connection' error when massive concurrent db requests happen. People told me ConnectionPool is the right way to go which I agreed at that time. However, I finally realized this is not the solution especially when there are a lot of different clients accessing mysql server through network, because connection pool is at client side it can not prevent the sum of connections of all clients from exceeding the max connection number of mysql server.
I think there should be some middleware on the mysql server working as a queue or pool, is anybody familiar with this? Thank you.
I know this question is widely asked, I am also surprised as if there is no total solution for it.
HAProxy should perform TCP-level queueing for you purpose. Though, would it be better to build an application server in the middle, to handle incoming flow at more conscious level than TCP. This could require rewriting of both server and clients, but could give you more control over what's happening.
What you ask is actually a pretty complicated problem.
First of all you need to decide whether mis-alignments in data are acceptable, for example: if you store in the database the number of Likes received, and you ask this number at 12:00:00, and the number in the DB is 500, and someone posts a LIKE at 12:00:01, and you query it again at 12:00:02; is it OK to receive "500" again, even if the correct number should be 501, provided that in a little time the answer "501" does come out?
If this is acceptable (the infamous "301 bug" in YouTube), then you might start caching some SELECT responses.
You might even cache them in middleware, i.e. have a special process running continuously and hogging ONE connection to MySQL, and answering requests in a queue. You might run it internally in the server as a Web server on port 8001 and have an Apache ReverseProxy, HAproxy, pound, or NginX location to proxy it outside.
You can do the same for special UPDATE/DELETE queries even if it's trickier.
It would be best to cache queries running asynchronously through AJAX first, if any, because serializing queries with a proxy is liable to perceptibly slow down the application.
You have a threefold target:
run queries on MySQL as fast as possible (look into indexing and MySQL caching) in order to free the ConnectionPool and keep it as lightly loaded as possible.
refactor the application in order to extract all information from queries (e.g., the number of rows with a certain property AND those rows as data are often retrieved using TWO queries, but with proper management you need only one and a SQLNumRows() call. Also, quite often similar queries with different informations are run, when a single query might have returned all information at one go: typically, one query to check user/password, another to fetch the complete user profile).
divert the most calls possible to something not at all (NginX, middleware) or lightly (queuing process) bound to MySQL; in the latter case, using a known number of connections in order to run predictably.
Unfortunately there's no easy "magic bullet" to solve this problem (except of course increasing the number of connections, maybe replicating the DB on several hosts running as master-slave. While not really a magic bullet, it is easier to design and implement).
I have been working on eCommerce site (using drupal). Few days ago before i am getting this error my site was working fine no issues was there. But now a days no. of times my site goes offline with the error message ('max_user_connection').
I was using some custom code containing mysql_connect and mysql_query now i changed everything into module and no custom queries left as such.The error is still their. On some of the pages data is populated with two different databases and to handle two database at same page i am using drupal function db_set_active().
I had discussed with hosting provider also they have increased a 'connection_limit' but error is still coming, what will be the possible reasons of having this kind of issue and the ways to handle this.
In this case the dbms is not able to serve all incoming connection requests to the database.
You can check with the "show full processlist" (which requires SUPER privilege) for current count of connections.
You now have either two choices: alter you application logic so that overall connections are descreased or you can try to alter the max_connections system variable in order to allow your DBMS to server more connections (also requires SUPER privilege).
But if your provider already told you that they increased 'connection_limit, you should go for the first approach (alter your application logic).
Right now we are dealing with a bit of a conundrum in my corporate environment where we are being blamed for a server crash, but I'm not 100% sure we are the culprit. Here's the server environment: We have a primary Coldfusion and its MSSQL database. We then also have a secondary database (MySQL) hosted on a different cloud which is used for miscellaneous tasks. The main reason the system is structured this way is because the primary server is operated by our Content Management System thus we are not allowed to modify it, add tables, or any operations like that, so we use the alternate database for that. By design there are no mission critical items on it, and pages are built in a way such that if the alternate DB returns no rows, the pages will continue to render properly.
Basically, I am being told that when the alternate MySQL server goes down, or stops accepting connections, that it is taking the entire primary cloud with it, including 5 other sites hosted on it. I do not have access to the primary Coldfusion or database logs, because the CMS provider will not give them to me. Thus I can only judge based on the validity of the explanation they are giving me.
The explanation for this behavior coming from our CMS provider is that when Coldfusion queries a database it creates a thread, and that if the DB doesn't respond the threads continue to stack. Eventually the processor is capped, and the server goes down. Is that an accurate explanation of how Coldfusion operates? If so, is there anyway to prevent it, possibly with shorter DB timeouts and the like? Or is the entire explanation posed by our CMS a red herring and something else is really causing the crashes.
Any guidance would be greatly appreciated.
Question answered - Documents found
http://kb2.adobe.com/cps/180/tn_18061.html
http://www.adobe.com/devnet/server_archive/articles/cf_timeouts_and_unresponsive_requests.html
Setting timeout requests globally does not timeout internal processes waiting on external resources (cfquery/cfhttp etc). The only way to time those out is by manually setting the timeout attribute. Not setting this could result in thread overload and a crashed server as was occurring with us.
http://kb2.adobe.com/cps/180/tn_18061.html
From reading bullet point 3 and depending on your traffic, your CMS guy might be right.
Also from the link above:
If the database is down and unresponsive, how many times will ColdFusion Server try to reconnect to the database? Will it eventually restart the ColdFusion Server?
If the database is down or the network link to the database goes down when a query request occurs, the connection will timeout (you can customize the timeout period with the timeout attribute in the cfquery tag) and return an error to the user. Please note that the ability to set the timeout for the connection depends on which driver you are using. You can trap this error and handle it programmatically with thecftry/cfcatch tags.
The catch here is that the timeout variable on the cfquery tags are not compatable with the MySQL ODBC driver. Could not find what the default timeout is. Let's say 5 minutes. If you get more than one request in those 5 minutes, it does appear that the connections will start to 'pile up'.