I have an application running on Apache 2.2, tomcat 6 and it uses struts framework, hibernate framework. We use Mysql at back end. We also interact with third party servers to place some requests that are requested by the user. Due to confidentiality constrains I can not tell you exactly what we do but I can assure you that we have not customized any thing and we use the most general builds availaible for Apache, tomcat etc. We use Linux platform. Lots of visitors visit our site, where they first pay using payment gateways, and then buy a product. To buy the product we again have to hit a third party site. Its a simple e-commerce kind of setup. The problem is that some times the server hangs. As in it does not responds and when we click on a link that (I know) is served by the tomcat container, it does not get loaded. Here is what i need help for:
Since my hosting is on a headless linux platform, please suggest me a good debugging tool.
We have logging in place and we print stack trace of almost all exception(if they happen), we always monitor catalina.out, but when the server hangs, we dont see any activity on catalina.out. may be this can give some one a clue.
We have show_sql disabled for hibernate, we tried to enable it but still that was not sufficient to figure out if the application stucks on a query. We also have slow_sql log enabled but that does not show any significant queries. How can we check if my application is stuck on a query?
If my application is not stuck on a query, how can i know where it is stuck?
How can I get java stack dump?
What are possible ways to resolve such a problem.
ANy suggestions are welcome. I thank you all in advance for reading my question and for the time you will devote writing an answer.
Facing your situation I would do a thread dump when the servlet container "hangs". A thread dump provides you a list of the stack traces for all Java threads in a given JVM.
Doing a thread dump is pretty easy:
Find the Tomcat's process ID, e.g. ps -ef | grep java
Send a SIGQUIT to the process, e.g. kill -3 tomcatProcessId
You will find the thread dump's content in TOMCAT_HOME/logs/catalina.out.
Related
So I installed a single host Openshift OKD v3.11 cluster. I installed it on a VM running Centos 7.8.2003.
It seems to have installed ok except that it continually streams verbose logs to /var/log/messages. Around 5 logs per second and all seem to be about throttling requests. Example of a typical log message:
******Jun 13 15:49:13 centos7 journal: I0613 14:49:13.011402 1 request.go:485] Throttling request took 196.341689ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-cert-signer/serviceaccounts/service-serving-cert-signer-sa*****
The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets.
https://access.redhat.com/solutions/3348921
I assume these logs are nothing to worry about and so my main question is what is the "best"/cleanest/simplest/easiest way to ensure the Openshift cluster doesn't continue to fill up /var/log/messages but will still log any important messages there?
I would recommend looking at the root cause for this behavior. These messages indicate that there are a lot of requests coming to your API. Typically this is due to some application performing calls in a tight loop leading to this many messages. In your case check your openshift-service-cert-signer if you can see any warnings or an abnormal amount of log messages.
If you want to get rid of the throttling messages, you can increase the amount of Queries per second (QPS) for the API server: Recommended Practices for OKD Master Hosts (lower part).
The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets. https://access.redhat.com/solutions/3348921
I do not understand why you're saying that, as I can access that document with my free Red Hat account without any subscriptions. Have you tried with a free account as it says on the site?
Simon's answer was helpful but I've finally got to the bottom of this.
The problem was simply that the version of Docker I had installed was old. At the time of writing the latest version of Centos is 7.8.2003 and if you install that and then simply run "yum install docker" hoping that you'll get something at least reasonably new and certainly compatible with the rest of the linux installation, you'll probably be making a mistake.
The right thing to do is to follow the simple steps here:
https://docs.docker.com/engine/install/centos/
The reason I found the problem was because excessive logging of my openshift cluster wasn't the only issue. I started seeing strange behaviour of other containers. A process of trial and error narrowed down the issue to the default Centos version of docker. Once I followed the page above all my problem vanished including the original problem of /var/log/message getting hammered by openshift containers.
The main reason I decided to answer my own question was because surely someone else is going to be as impatient/thick as me and simply install Centos7 then try "yum install docker" without knowing they're about to enter a world of pain.
The Site is showing the 504 gate way timeout error frequently. Because the CPU utilization was reached maximum. We have used AWS RDS, Hibernate, MySQL, Spring MVC, Apache tomcat 8 in our application. Unable to find the reason for maximum CPU utilization. Please help me out on this issue.
Though the question is very generic in nature and more information would help however for initial debugging you need to figure out following things :
Figure out process which is taking most CPU. You can easily figure out this using top or htop utility in linux or TaskManager utility in Windows.
Go to Application debugging module and logs to see if there are any error regarding you code,MYSQL or any other API Call.
Also figure out if there are any hanging threads or sleeping process and also make sure whether query execution is working fine inside MYSQL(RDS).
Last, see if the resource allocated is enough to serve the application for the current number of users.
I'm getting the following error intermittently when making a call from my ASP.Net MVC web application which is using Dapper to query MySQL.
Unable to connect to any of the specified MySQL hosts.
The exception only occurs when my web app is published to Azure. It has worked 100% of the time when I run the code locally. I've deployed the code to a second azure website, and also get the exception there, again intermittently.
The MySQL database is running on an Azure VM (Ubuntu). This server also has some R scripts that access the database, which are being run at a set interval. I've had no connectivity issue with these either. It is just the .Net code that's struggling.
I've scoured the web, but don't feel like I've turned up anything of value. Most of the links have pointed to a connection string problem, but since it works intermittently that doesn't seem to fit my problem. Some links have referenced DNS issues, but I'm getting the same problem when I use the IP Address instead of the machine name for the DNS server.
I'm sure I need to track down more information, but I'm not sure where it would be. This is my first foray into using a MySQL db in this fashion, and I'm not familiar with config options or log files on that side of things. I feel similarly about Azure websites with database interactions too.
What can I try next?
Just to drive home the point about this error being intermittent, here's a screenshot from the Runscope job that's hitting the page (thus triggering the MySQL query) every 5 minutes:
I was able to fix (or perhaps "circumvent") this problem by adding the --skip-host-cache flag to our mysql configuration file. I still don't fully understand what the root of the problem is, but we haven't had any issues with MySQL connectivity from the Azure website since adding that.
I'm running Windows 7 Pro and have a few servers running. One of the servers is a SSH / file server that was made via Cygwin. I already have logging setup internally using syslog-d; however, it does not provide adequate logging. When a user is connected to the server I can see him/her in the Windows 7 Resource Monitor and it shows his/her IP address as well as how much data is being sent/received. When a user is downloading a file from the file server I can also see in the resource monitor what file he/she is downloading by looking at the disk usage.
Herein lies the first question: How can I log users' IP address, the time they connect & disconnect, what files they download, and what their download speed was, to a database in MySQL?
In addition to the aforementioned server, I also use IIS to host a website, and would like to have some sort of networking logging.
If I could find a tool that would work for both of these servers that would be the best solution.
I did some searching and found a program called Snort that looks like it would work for the network side of things, but not for the disk usage. I'm not familiar with this program at all, but at first glance maybe it could accomplish part of what I want to do? Maybe there is an easier/better way?
I'm pretty new to MySQL and know very little about network and disk logging so any and all help and guidance would be much appreciated. Thanks!
Advanced Web Statistics does a pretty good job of making sense of the IIS log files, and though it will give you more information than you need, it will certainly give you the information you want. It is open source, and my hosting provider uses it for the ASP.NET sites I have developed.
As far as logging the information to MySQL:
I am assuming that you already have, or know how to get the information and you simply want to log it to a MySQL DB.
1st, you will need to create the database.
2nd, you need the MySQL connector for your programming language of choice. The MySQL ADO.NET connector is excellent and easy to use. I am also assuming you know at least one programming language and how to connect it to a database. If not, I recommend C# with ADO.NET-- it is super easy and there are plenty of tutorials online.
3rd, write a program to send your information to the database, when you receive it.
I have an ASP.Net 3.5 Application hosted on Windows Server 2008 machine using IIS 7. There is a seperate application pool for this application.
The problem I have is that as soon as 2-3 users access website simultaneously, it starts throwing exceptions on almost any task performed say e.g. an exception during logging in the user using built in aspnet_membership. The same things work fine when there is only one user using it.
Any suggestions what things I need to test for?
Thanks,
Ali
I had some problem like this for some web-application, but in my case I was using IIS 6.0. After analyzing the application by means of some DevPartner tools for analyzing memory usage we realize that some components were consuming excessive resources, by replacing the toolkit we were using for simple .Net controls the applications stop crashing. Most of the crashes are due to problems in code. Also check the web config in the following points: sessionState, authentication. Disable custom errors (customErrors) to view the error details. See the application event's log to check the error description.
The first thing I do when setting up a website of this nature is to uncouple the session state from InProc to StateServer. Oftentimes I find that an application recycle or even an application level exception can cause the w3wp.exe process to crash and reload which dumps session errors. As with #Arce Brito's recommendation, you should do everything you can to find the root cause of the exception as decoupling the session will really only mask the symptom.