The Site is showing the 504 gate way timeout error frequently. Because the CPU utilization was reached maximum. We have used AWS RDS, Hibernate, MySQL, Spring MVC, Apache tomcat 8 in our application. Unable to find the reason for maximum CPU utilization. Please help me out on this issue.
Though the question is very generic in nature and more information would help however for initial debugging you need to figure out following things :
Figure out process which is taking most CPU. You can easily figure out this using top or htop utility in linux or TaskManager utility in Windows.
Go to Application debugging module and logs to see if there are any error regarding you code,MYSQL or any other API Call.
Also figure out if there are any hanging threads or sleeping process and also make sure whether query execution is working fine inside MYSQL(RDS).
Last, see if the resource allocated is enough to serve the application for the current number of users.
Related
So I installed a single host Openshift OKD v3.11 cluster. I installed it on a VM running Centos 7.8.2003.
It seems to have installed ok except that it continually streams verbose logs to /var/log/messages. Around 5 logs per second and all seem to be about throttling requests. Example of a typical log message:
******Jun 13 15:49:13 centos7 journal: I0613 14:49:13.011402 1 request.go:485] Throttling request took 196.341689ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-cert-signer/serviceaccounts/service-serving-cert-signer-sa*****
The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets.
https://access.redhat.com/solutions/3348921
I assume these logs are nothing to worry about and so my main question is what is the "best"/cleanest/simplest/easiest way to ensure the Openshift cluster doesn't continue to fill up /var/log/messages but will still log any important messages there?
I would recommend looking at the root cause for this behavior. These messages indicate that there are a lot of requests coming to your API. Typically this is due to some application performing calls in a tight loop leading to this many messages. In your case check your openshift-service-cert-signer if you can see any warnings or an abnormal amount of log messages.
If you want to get rid of the throttling messages, you can increase the amount of Queries per second (QPS) for the API server: Recommended Practices for OKD Master Hosts (lower part).
The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets. https://access.redhat.com/solutions/3348921
I do not understand why you're saying that, as I can access that document with my free Red Hat account without any subscriptions. Have you tried with a free account as it says on the site?
Simon's answer was helpful but I've finally got to the bottom of this.
The problem was simply that the version of Docker I had installed was old. At the time of writing the latest version of Centos is 7.8.2003 and if you install that and then simply run "yum install docker" hoping that you'll get something at least reasonably new and certainly compatible with the rest of the linux installation, you'll probably be making a mistake.
The right thing to do is to follow the simple steps here:
https://docs.docker.com/engine/install/centos/
The reason I found the problem was because excessive logging of my openshift cluster wasn't the only issue. I started seeing strange behaviour of other containers. A process of trial and error narrowed down the issue to the default Centos version of docker. Once I followed the page above all my problem vanished including the original problem of /var/log/message getting hammered by openshift containers.
The main reason I decided to answer my own question was because surely someone else is going to be as impatient/thick as me and simply install Centos7 then try "yum install docker" without knowing they're about to enter a world of pain.
I'm getting the following error intermittently when making a call from my ASP.Net MVC web application which is using Dapper to query MySQL.
Unable to connect to any of the specified MySQL hosts.
The exception only occurs when my web app is published to Azure. It has worked 100% of the time when I run the code locally. I've deployed the code to a second azure website, and also get the exception there, again intermittently.
The MySQL database is running on an Azure VM (Ubuntu). This server also has some R scripts that access the database, which are being run at a set interval. I've had no connectivity issue with these either. It is just the .Net code that's struggling.
I've scoured the web, but don't feel like I've turned up anything of value. Most of the links have pointed to a connection string problem, but since it works intermittently that doesn't seem to fit my problem. Some links have referenced DNS issues, but I'm getting the same problem when I use the IP Address instead of the machine name for the DNS server.
I'm sure I need to track down more information, but I'm not sure where it would be. This is my first foray into using a MySQL db in this fashion, and I'm not familiar with config options or log files on that side of things. I feel similarly about Azure websites with database interactions too.
What can I try next?
Just to drive home the point about this error being intermittent, here's a screenshot from the Runscope job that's hitting the page (thus triggering the MySQL query) every 5 minutes:
I was able to fix (or perhaps "circumvent") this problem by adding the --skip-host-cache flag to our mysql configuration file. I still don't fully understand what the root of the problem is, but we haven't had any issues with MySQL connectivity from the Azure website since adding that.
I'm facing a problem with a 3-tier application. It uses IIS and SQL Server. And the problem is that sometimes (longer requests cause higher probability) client does not receive response from IIS. It hangs when there is no activity ON SQL Server, and when even when I kill an SQL Process I get no response with an exception. When I examine data after these hangs I find that my SQL requests have succeeded, so the problem appears to be on IIS. Unfotunately I don't have much control over the place where the application is hosted and I cannot reproduce the problem. There can be many answers to this question, but at least I need a hint in what direction I should investigate.
UPDATE: I also have an app running on the same server that does a small job: it inserts around a 100 lines line-by-line into a database. Artificially I made it run for about 10 minutes (by putting xlock on the table). After removing the xlock the process on IIS continued running and successfully inserted those 100 lines. However, control never returned to the client thereafter, it just waited indefinitely. It looks like this is a problem when returning a response to the client after a long running process.
BTW, this is not a website, but a WinForms app that uses Remoting.
If you're sure that SQL Server is returning to IIS, then you might want to check the IIS logs to see what might be happening to the request. The normal location for these logs is %SystemRoot%\system32\Logfiles\<service_name>.
If you're not sure about SQL Server, you might access the SQL logs, run Profiler, or check the Windows system logs for errors, run your site in debug in Visual Studio or add your own logging to your app to figure out which step it's hanging on.
I have finally figured out where the problem is. The application is run on a web farm and there is a balancer server between the client and IIS. There was a too small timeout on the balancer. For some reason it is not quite friendly with .net remoting and it doesn't return any timeout exceptions to the client. The issue has been solved by increasing the timeout.
I have an application running on Apache 2.2, tomcat 6 and it uses struts framework, hibernate framework. We use Mysql at back end. We also interact with third party servers to place some requests that are requested by the user. Due to confidentiality constrains I can not tell you exactly what we do but I can assure you that we have not customized any thing and we use the most general builds availaible for Apache, tomcat etc. We use Linux platform. Lots of visitors visit our site, where they first pay using payment gateways, and then buy a product. To buy the product we again have to hit a third party site. Its a simple e-commerce kind of setup. The problem is that some times the server hangs. As in it does not responds and when we click on a link that (I know) is served by the tomcat container, it does not get loaded. Here is what i need help for:
Since my hosting is on a headless linux platform, please suggest me a good debugging tool.
We have logging in place and we print stack trace of almost all exception(if they happen), we always monitor catalina.out, but when the server hangs, we dont see any activity on catalina.out. may be this can give some one a clue.
We have show_sql disabled for hibernate, we tried to enable it but still that was not sufficient to figure out if the application stucks on a query. We also have slow_sql log enabled but that does not show any significant queries. How can we check if my application is stuck on a query?
If my application is not stuck on a query, how can i know where it is stuck?
How can I get java stack dump?
What are possible ways to resolve such a problem.
ANy suggestions are welcome. I thank you all in advance for reading my question and for the time you will devote writing an answer.
Facing your situation I would do a thread dump when the servlet container "hangs". A thread dump provides you a list of the stack traces for all Java threads in a given JVM.
Doing a thread dump is pretty easy:
Find the Tomcat's process ID, e.g. ps -ef | grep java
Send a SIGQUIT to the process, e.g. kill -3 tomcatProcessId
You will find the thread dump's content in TOMCAT_HOME/logs/catalina.out.
I have an ASP.Net 3.5 Application hosted on Windows Server 2008 machine using IIS 7. There is a seperate application pool for this application.
The problem I have is that as soon as 2-3 users access website simultaneously, it starts throwing exceptions on almost any task performed say e.g. an exception during logging in the user using built in aspnet_membership. The same things work fine when there is only one user using it.
Any suggestions what things I need to test for?
Thanks,
Ali
I had some problem like this for some web-application, but in my case I was using IIS 6.0. After analyzing the application by means of some DevPartner tools for analyzing memory usage we realize that some components were consuming excessive resources, by replacing the toolkit we were using for simple .Net controls the applications stop crashing. Most of the crashes are due to problems in code. Also check the web config in the following points: sessionState, authentication. Disable custom errors (customErrors) to view the error details. See the application event's log to check the error description.
The first thing I do when setting up a website of this nature is to uncouple the session state from InProc to StateServer. Oftentimes I find that an application recycle or even an application level exception can cause the w3wp.exe process to crash and reload which dumps session errors. As with #Arce Brito's recommendation, you should do everything you can to find the root cause of the exception as decoupling the session will really only mask the symptom.