So, I have several jobs on Hudson that take 3-6 hours to run. The slave machines are a variety of different Windows operating systems running on VMs. Sometimes, there will be a minor hiccup that apparently will cause a socket that's been open for 6 hours to become closed. (which doesn't seem crazy even with perfect networking) And so, I end up with a stacktrace pointing to this:
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.net.SocketException: socket closed
Is there any plugin or other way I could fix this extremely annoying problem? When you're 3 hours through a 4 hour build and it fails because of this, it's a bit infuriating.
Maybe off topic, but have you considered an alternative CI server, like JetBrains TeamCity? I've used it for 4 years on .NET projects and highly recommend them.
If build is running for 6 hours then it would fail if connection between Master-Slave breaks down.
So solution lies in creating a custom logic to connect to slave and Hudson provides an option for this. Check this link: http://wiki.hudson-ci.org/display/HUDSON/Distributed+builds#Distributedbuilds-WriteyourownscripttolaunchHudsonslaves
A custom script with retry logic should be the way out.
Related
So I installed a single host Openshift OKD v3.11 cluster. I installed it on a VM running Centos 7.8.2003.
It seems to have installed ok except that it continually streams verbose logs to /var/log/messages. Around 5 logs per second and all seem to be about throttling requests. Example of a typical log message:
******Jun 13 15:49:13 centos7 journal: I0613 14:49:13.011402 1 request.go:485] Throttling request took 196.341689ms, request: GET:https://172.30.0.1:443/api/v1/namespaces/openshift-service-cert-signer/serviceaccounts/service-serving-cert-signer-sa*****
The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets.
https://access.redhat.com/solutions/3348921
I assume these logs are nothing to worry about and so my main question is what is the "best"/cleanest/simplest/easiest way to ensure the Openshift cluster doesn't continue to fill up /var/log/messages but will still log any important messages there?
I would recommend looking at the root cause for this behavior. These messages indicate that there are a lot of requests coming to your API. Typically this is due to some application performing calls in a tight loop leading to this many messages. In your case check your openshift-service-cert-signer if you can see any warnings or an abnormal amount of log messages.
If you want to get rid of the throttling messages, you can increase the amount of Queries per second (QPS) for the API server: Recommended Practices for OKD Master Hosts (lower part).
The only reference I have managed to find is a question here but the access to the discussion is only available to those with deep pockets. https://access.redhat.com/solutions/3348921
I do not understand why you're saying that, as I can access that document with my free Red Hat account without any subscriptions. Have you tried with a free account as it says on the site?
Simon's answer was helpful but I've finally got to the bottom of this.
The problem was simply that the version of Docker I had installed was old. At the time of writing the latest version of Centos is 7.8.2003 and if you install that and then simply run "yum install docker" hoping that you'll get something at least reasonably new and certainly compatible with the rest of the linux installation, you'll probably be making a mistake.
The right thing to do is to follow the simple steps here:
https://docs.docker.com/engine/install/centos/
The reason I found the problem was because excessive logging of my openshift cluster wasn't the only issue. I started seeing strange behaviour of other containers. A process of trial and error narrowed down the issue to the default Centos version of docker. Once I followed the page above all my problem vanished including the original problem of /var/log/message getting hammered by openshift containers.
The main reason I decided to answer my own question was because surely someone else is going to be as impatient/thick as me and simply install Centos7 then try "yum install docker" without knowing they're about to enter a world of pain.
I have a nodejs + mysql application running on AWS on windows (EC2, RDP). I often get the above issue. Googling suggests that it may be due to jobs running in the background. here is something I found in "It sounds like you've forked a process which is blocking things, and Ctrl+C is killing that."
Could this be the reason? can it be something else?
If so, what action can I take?
Is there a way to view all jobs that are running in the background?
Without seeing the code it's hard to guess, but if you just need the application to end, you could invoke process.exit() and it will exit.
To those of you that are trying to be good little developers and version control their ExpressionEngine sites with git, how do you handle your database?
In my limited experience with multiple developers working on one ExpressionEngine site, we've had to all run off of a single MySQL development database running on a remote web server. For those of you that have tried this, it is PAINFULLY slow. Page loads can easily take 5-10 seconds making development extremely difficult. It would be quicker to work off of a remote development server. I am trying to steer away from working off of a remote MySQL server in order to be able to work from anywhere and not depend on Internet connection speed/quality.
Just wondering how others handle their MySQL databases.
Do all of your developers run off of one central database? Have you dealt with slowness issues like we have?
Do you keep your database under version control? How do you handle export/imports among multiple developers and multiple branches?
With one developer I can import/export/commit the database very easily but as soon as you add another developer to the mix, it gets very VERY muddy. Looking forward to hearing everyone's thoughts on this mammoth topic.
Thanks!
It seems there is a lot of time lost on failing DNS requests, with a remote database.
Start your MySQL server with start mysqld with --skip-name-resolve. (More information on this topic can be found here: http://dev.mysql.com/doc/refman/5.0/en/host-cache.html)
Having a remote database still seems to be the best way for us to work on a project with multiple developers.
I almost always use a central database for development. Depending which host you use, the speed difference may not be huge.
Obviously, if you're not making changes to the database, i.e. only doing template development, keeping the database in sync is not as needed, so you could potentially bring up a local copy of the database. You just have to remember to repeat any database changes, if you do end up making some.
As far as version control, I keep a copy of my base EE install's SQL file in my base repository. Other than that I don't usually keep copies of the database in Git, so I don't do a lot of importing/exporting, etc.
Have you looked at the EE Profiler recently? You'll probably notice in the neighborhood of 20-80 queries on your home page depending on it's complexity.
The problem is that, for each query, MySQL must execute a remote request for data, download the response, and then present ExpressionEngine it's data. The 20-80 round trips to the database is what's causing your delay and I don't think there is much you can do about it. When using a remote (outside our network) database, I get the same delay as you.
When MySQL is running on your machine or the production server, it doesn't have the added network requests causing latency in it's requests for data. This is the difference.
As for fixes, all you can do is move to a database hosted on your internal network. We have a Linux machine that mimics our production environment that we use for staging. Since it's on our network, we can use the local IP address in our database.php file. This is much faster.
The problem that we still have is the issue of channels/fields/entries. When a developer is working on a new section, they'll likely need to create a new channel and fields and/or new entries. When we're ready to push that functionality to production, we have to manually make those changes on the production server as there is no way to reliably export them. I am hopeful of this addon though---we'll see.
In my company (4 developers) we each run our own DB locally. But recently I tested Rackspace Cloud Databases (but there are other cloud db providers) for a heavy DB that could become difficult to run on a little laptop. It's relatively less expensive than running our own db server, and it can be setup or deleted in the minute.
I have an application running on Apache 2.2, tomcat 6 and it uses struts framework, hibernate framework. We use Mysql at back end. We also interact with third party servers to place some requests that are requested by the user. Due to confidentiality constrains I can not tell you exactly what we do but I can assure you that we have not customized any thing and we use the most general builds availaible for Apache, tomcat etc. We use Linux platform. Lots of visitors visit our site, where they first pay using payment gateways, and then buy a product. To buy the product we again have to hit a third party site. Its a simple e-commerce kind of setup. The problem is that some times the server hangs. As in it does not responds and when we click on a link that (I know) is served by the tomcat container, it does not get loaded. Here is what i need help for:
Since my hosting is on a headless linux platform, please suggest me a good debugging tool.
We have logging in place and we print stack trace of almost all exception(if they happen), we always monitor catalina.out, but when the server hangs, we dont see any activity on catalina.out. may be this can give some one a clue.
We have show_sql disabled for hibernate, we tried to enable it but still that was not sufficient to figure out if the application stucks on a query. We also have slow_sql log enabled but that does not show any significant queries. How can we check if my application is stuck on a query?
If my application is not stuck on a query, how can i know where it is stuck?
How can I get java stack dump?
What are possible ways to resolve such a problem.
ANy suggestions are welcome. I thank you all in advance for reading my question and for the time you will devote writing an answer.
Facing your situation I would do a thread dump when the servlet container "hangs". A thread dump provides you a list of the stack traces for all Java threads in a given JVM.
Doing a thread dump is pretty easy:
Find the Tomcat's process ID, e.g. ps -ef | grep java
Send a SIGQUIT to the process, e.g. kill -3 tomcatProcessId
You will find the thread dump's content in TOMCAT_HOME/logs/catalina.out.
I have read about the advantages of using a BitNami stack for LAMP development, now I am wondering if there are any drawbacks to using BitNami vs manually installing PHP, MySQL, and Apache separately. I use Mac OS but I would be interested on how it applies to both Mac and Windows. Any thoughts?
I am one of the developers of BitNami. Whether to use a native stack or a BitNami stack depends on what you are trying to do. Installing the individual items separately should be exactly the same as running our installer, and the whole purpose why we put the installers together is so you would not have to :) In the case of Mac, one of the advantages of BitNami is that you can have more up-to-date components and multiple installations. A disadvantage / difference is that the applications and path will be different than the typical ones so if you are using third-party tutorials or documentation, it may not work right away
There are 3 common drawbacks to Bitnami vs. a native LEMP/LAMP stack:
File paths. Because Bitnami is a container approach to web stacks, it installs everything in Ubuntu (or whatever Linux distro) under the /opt/bitnami directory. So, many developers who are used to customizing their stack using nano or vim editors (via the Bash shell) quickly discover that you first have to figure out where all the different configuration files of your stack modules reside, etc. Even after you figure those out, most of the online tutorials and documentations you might find will not apply to your stack.
Lockdown. This could be seen as either an advantage or a disadvantage, depending on your perspective (and situation). The entire point of using a containerized approach is to have more control of the stack environment, which can improve compatibility, predictability, security, and otherwise. However as #team-life mentioned, this can quickly become frustrating when you are trying to use "standard" Bash shell commands or even the MySQL CLI, e.g. when trying to analyze or replicate your stack, etc. To put it simply, logging into shell on a server where Bitnami is installed is not in fact logging into the actual shell :)
Upgrades. At the end of the day, Bitnami (and other containers, like Docker) are adding another "layer" to your stack, and thus, more bloat. For some users this "bloat" is justifiable, and preferable (for example, very large companies who require across-the-board uniformity). But what many developers discover with Bitnami and containers is upgrading your stack can be rather janky. For all the alleged advantages in terms of environment "stability", it turns out that upgrading your stack can actually introduce quite a bit of instability and unpredictability, often to the extent of canceling out the benefits. As #domi mentioned, all upgrades run through Bitnami (and not Ubuntu mirrors, etc) meaning you are bound to their versions and release schedules; you are also often required to completely re-install the stack again...
Ultimately, containers are a recent trend that have become very popular among so-called "enterprise" and "corporate" in-house teams, but it is one of those things that might not be the best features for smaller agencies or independent developers to embrace.
That is why native LEMP stacks like SlickStack (my project) are gaining momentum.
This Reddit thread has a few other AWS-specific comments as well.
BitNami uses paths that will be very different from the industry standard ones so if you are trying to login to a server to do some task, it will take you a lot of time to understand their custom-made-folder-structure. And that's a big drawback. When you login to a unix server, you know where the files and paths are, maybe you have one or two options, that are standard. BitNami uses a completely different one. Chaos ensues.
I'm a happy bitnami stack user. It's a great stack. I can describe many advantages.
The draw back of using bitnami stack is the update cycle. For example on Debian/Ubuntu based system, you can not use the standard apt-get update/upgrade.
That means some security updates might not get to your system as fast as your standard cron (automated periodic) update mechanism.
To upgrade the system you will need to create backup, install a new stack, then import the backup to the new stack. Which might not be an ideal procedure.
Some people categorize that as non-production-environment.
Bitnami - ease of use, validated components - known working good configuration.
Disadvantage - Patches and updates. you cannot update packages for security like you can for native install. Any bulletins must be addressed by the bitnami team, who may/will roll out an update to address issues. The bitnami updates are full stack upgrades, meaning you can't just upgrade a single component (php for example) - you need to upgrade the whole bitnami stack, and the often recommended method is to backup your application database, install a parallel bitnami stack that has the latest updates, then restore or migrate to the new installation.
Some will tell you that you can shoehorn patches into bitnami stacks, but it's not at all recommended, will lead you off the stack and most likely cause you down stream issues.
Bitnami evidently is unable to use certain commands from their mysql command line. I'm finding this very frustrating. Here is some stuff I found out.
It puts you into its own bash shell bash-4.2#
mysql>SHOW MASTER STATUS returns -> (nothing) doesn't seem to work
rcmysql start or stop doesn't work from mysql> you have to shell out of where your at and run the ctlscript.sh which is a pain.
Just to get to command line you have to run ./use_lampstack
I'm guessing that they are giving us a very paired down mysql group of commands because there will be less for them to support and less for people to jack up.
So this came up for me because I was trying setup replication. I was following directions from someone who had a "regular" install. It was difficult to follow because most of the commands he was suggesting didn't work from the bitnami mysql> command line. So while I really like the uniformity of Bitnami and the modular nature of it I have run into a snag trying to setup replication.