I have a nodejs + mysql application running on AWS on windows (EC2, RDP). I often get the above issue. Googling suggests that it may be due to jobs running in the background. here is something I found in "It sounds like you've forked a process which is blocking things, and Ctrl+C is killing that."
Could this be the reason? can it be something else?
If so, what action can I take?
Is there a way to view all jobs that are running in the background?
Without seeing the code it's hard to guess, but if you just need the application to end, you could invoke process.exit() and it will exit.
Related
So I installed a single host Openshift OKD v3.11 cluster. I installed it on a VM running Centos 7.8.2003.
It seems to have installed ok except that it continually streams verbose logs to /var/log/messages. Around 5 logs per second and all seem to be about throttling requests. Example of a typical log message:
******Jun 13 15:49:13 centos7 journal: I0613 14:49:13.011402 1 request.go:485] Throttling request took 196.341689ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-cert-signer/serviceaccounts/service-serving-cert-signer-sa*****
The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets.
https://access.redhat.com/solutions/3348921
I assume these logs are nothing to worry about and so my main question is what is the "best"/cleanest/simplest/easiest way to ensure the Openshift cluster doesn't continue to fill up /var/log/messages but will still log any important messages there?
I would recommend looking at the root cause for this behavior. These messages indicate that there are a lot of requests coming to your API. Typically this is due to some application performing calls in a tight loop leading to this many messages. In your case check your openshift-service-cert-signer if you can see any warnings or an abnormal amount of log messages.
If you want to get rid of the throttling messages, you can increase the amount of Queries per second (QPS) for the API server: Recommended Practices for OKD Master Hosts (lower part).
The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets. https://access.redhat.com/solutions/3348921
I do not understand why you're saying that, as I can access that document with my free Red Hat account without any subscriptions. Have you tried with a free account as it says on the site?
Simon's answer was helpful but I've finally got to the bottom of this.
The problem was simply that the version of Docker I had installed was old. At the time of writing the latest version of Centos is 7.8.2003 and if you install that and then simply run "yum install docker" hoping that you'll get something at least reasonably new and certainly compatible with the rest of the linux installation, you'll probably be making a mistake.
The right thing to do is to follow the simple steps here:
https://docs.docker.com/engine/install/centos/
The reason I found the problem was because excessive logging of my openshift cluster wasn't the only issue. I started seeing strange behaviour of other containers. A process of trial and error narrowed down the issue to the default Centos version of docker. Once I followed the page above all my problem vanished including the original problem of /var/log/message getting hammered by openshift containers.
The main reason I decided to answer my own question was because surely someone else is going to be as impatient/thick as me and simply install Centos7 then try "yum install docker" without knowing they're about to enter a world of pain.
So, I have several jobs on Hudson that take 3-6 hours to run. The slave machines are a variety of different Windows operating systems running on VMs. Sometimes, there will be a minor hiccup that apparently will cause a socket that's been open for 6 hours to become closed. (which doesn't seem crazy even with perfect networking) And so, I end up with a stacktrace pointing to this:
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: socket closed
Is there any plugin or other way I could fix this extremely annoying problem? When you're 3 hours through a 4 hour build and it fails because of this, it's a bit infuriating.
Maybe off topic, but have you considered an alternative CI server, like JetBrains TeamCity? I've used it for 4 years on .NET projects and highly recommend them.
If build is running for 6 hours then it would fail if connection between Master-Slave breaks down.
So solution lies in creating a custom logic to connect to slave and Hudson provides an option for this. Check this link: http://wiki.hudson-ci.org/display/HUDSON/Distributed+builds#Distributedbuilds-WriteyourownscripttolaunchHudsonslaves
A custom script with retry logic should be the way out.
I'm facing a problem with a 3-tier application. It uses IIS and SQL Server. And the problem is that sometimes (longer requests cause higher probability) client does not receive response from IIS. It hangs when there is no activity ON SQL Server, and when even when I kill an SQL Process I get no response with an exception. When I examine data after these hangs I find that my SQL requests have succeeded, so the problem appears to be on IIS. Unfotunately I don't have much control over the place where the application is hosted and I cannot reproduce the problem. There can be many answers to this question, but at least I need a hint in what direction I should investigate.
UPDATE: I also have an app running on the same server that does a small job: it inserts around a 100 lines line-by-line into a database. Artificially I made it run for about 10 minutes (by putting xlock on the table). After removing the xlock the process on IIS continued running and successfully inserted those 100 lines. However, control never returned to the client thereafter, it just waited indefinitely. It looks like this is a problem when returning a response to the client after a long running process.
BTW, this is not a website, but a WinForms app that uses Remoting.
If you're sure that SQL Server is returning to IIS, then you might want to check the IIS logs to see what might be happening to the request. The normal location for these logs is %SystemRoot%\system32\Logfiles\<service_name>.
If you're not sure about SQL Server, you might access the SQL logs, run Profiler, or check the Windows system logs for errors, run your site in debug in Visual Studio or add your own logging to your app to figure out which step it's hanging on.
I have finally figured out where the problem is. The application is run on a web farm and there is a balancer server between the client and IIS. There was a too small timeout on the balancer. For some reason it is not quite friendly with .net remoting and it doesn't return any timeout exceptions to the client. The issue has been solved by increasing the timeout.
I have an application running on Apache 2.2, tomcat 6 and it uses struts framework, hibernate framework. We use Mysql at back end. We also interact with third party servers to place some requests that are requested by the user. Due to confidentiality constrains I can not tell you exactly what we do but I can assure you that we have not customized any thing and we use the most general builds availaible for Apache, tomcat etc. We use Linux platform. Lots of visitors visit our site, where they first pay using payment gateways, and then buy a product. To buy the product we again have to hit a third party site. Its a simple e-commerce kind of setup. The problem is that some times the server hangs. As in it does not responds and when we click on a link that (I know) is served by the tomcat container, it does not get loaded. Here is what i need help for:
Since my hosting is on a headless linux platform, please suggest me a good debugging tool.
We have logging in place and we print stack trace of almost all exception(if they happen), we always monitor catalina.out, but when the server hangs, we dont see any activity on catalina.out. may be this can give some one a clue.
We have show_sql disabled for hibernate, we tried to enable it but still that was not sufficient to figure out if the application stucks on a query. We also have slow_sql log enabled but that does not show any significant queries. How can we check if my application is stuck on a query?
If my application is not stuck on a query, how can i know where it is stuck?
How can I get java stack dump?
What are possible ways to resolve such a problem.
ANy suggestions are welcome. I thank you all in advance for reading my question and for the time you will devote writing an answer.
Facing your situation I would do a thread dump when the servlet container "hangs". A thread dump provides you a list of the stack traces for all Java threads in a given JVM.
Doing a thread dump is pretty easy:
Find the Tomcat's process ID, e.g. ps -ef | grep java
Send a SIGQUIT to the process, e.g. kill -3 tomcatProcessId
You will find the thread dump's content in TOMCAT_HOME/logs/catalina.out.
I just uploaded a Wordpress site from my local machine to a Bluehost shared server. Ran fine locally, but now it is loading very slowly (107 seconds for home page). Bluehost tech support ran GTMetrix site analyzer and came back with "it's the CSS in your theme". They say nothing is wrong with the server.
I definitely need to clean up my CSS, but I didn't think it could have such a large impact on load times. Am I wrong?
Looking at the resource load times with Chrome's developer tools makes it immediately clear that it's your main document that is responsible for the delay; not the CSS, not anything else.
Therefore we have to assume it's something in your own code that causes the delay. Since it worked fine when run locally, the most likely scenario is that your code is trying to connect to some server (perhaps a database?) that it cannot connect to, and the delay is due to the connection finally timing out.
Recommendation: double check the places where you make connections to any external resource, and especially the credentials used when you do so. Is your host authorized to make these connections the same as your local development machine is? If you are connecting by IP, are those IPs accessible from your host?
The problem is loading
http://ad.doubleclick.net/adi/N5192.395082.LOT18.COM/B5529584;sz=300x250;ord=[timestamp]?
which is probably a script?
That takes 59 seconds for me. The rest is fast.
From the Chrome dev tools (Network tab): http://screencast.com/t/8DdtXeEv
The solution: turn off your ads.
You can use quick cache plugin which will speed up you site without compromise.
http://wordpress.org/extend/plugins/quick-cache/