I am planning to install Hudson on Amazon EC2 using Ubuntu image. The code I am going to test does not have a big memory overhead - I will be executing mainly python unit tests.
Which EC2 instance should I use? Would micro instance be sufficient (have enough memory) or should I use a bigger instance?
Jenkins itself will happily run on a micro, but there are two problems: 1) you won't have much memory left for building and testing, around 150MB, but the bigger problem is 2) if your CPU usage spikes for more than a few seconds Amazon will simply crush your instance with throttling cutting off 97% or more of available CPU. http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instance-cpu-steal/
The throttling was completely impossible for us, making a build with testing take 12 minutes on EC2 instead of 25 seconds on a quad i7 laptop.
But! There's a fix for the frugal:
Run a Jenkins master on a micro, but start up a small instance when needed to run the actual tests. That gives us plenty of memory and decent CPU, yet it's still incredibly cheap (ten cents per push [or commit]). However, it substantially increases build time because it has to boot the instance and all that.
The setup is rather involved, and requires working around some limitations of the ec2 plugin (which, overall, works extremely well), so we wrote up a blog post if you want to do this: http://wkmacura.tumblr.com/post/5416465911/jenkins-ec2
We're running hudson on EC2 and integration testing Ruby/Rails. We're doing just fine on a micro instance as I think you will to.
Related
I using PM2 to run my nodejs application.
When starting it in cluster mode "pm2 start server -i 0": PM2 will automatically spawn as many workers as you have CPU cores.
What is the ideal number of workers to run and why?
Beware of the context switch
When running multiple processes on your machine, try to make sure each CPU core will be kepy busy by a single application thread at a time. As a general rule, you should look to spawn N-1 application processes, where N is the number of available CPU cores. That way, each process is guaranteed to get a good slice of one core, and there’s one spare for the kernel scheduler to run other server tasks on. Additionally, try to make sure the server will be running little or no work other than your Node.JS application, so processes don’t fight for CPU.
We made a mistake where we deployed two busy node.js applications to our servers, both apps spawning N-1 processes each. The applications’ processes started vehemently competing for CPU, resulting in CPU load and usage increasing dramatically. Even though we were running these on beefy 8-core servers, we were paying a noticeable penalty due to context switching. Context switching is the behaviour whereby the CPU suspends one task in order to work on another. When context switching, the kernel must suspend all state for one process while it loads and executes state for another. After simply reducing the number of processes the applications spawned such that they each shared an equal number of cores, load dropped significantly:
https://engineering.gosquared.com/optimising-nginx-node-js-and-networking-for-heavy-workloads
I am running an EC2 Instance on Ubuntu Server machine. Tomcat and MySQL are installed and deployed java web-application on it since 1 month. It was running good with great performance for almost 1 month but now my application is responding very slow.
Also, point to note is: Earlier when I used to log into my Ubuntu Server through PuTTY, it was quick but now its taking time even when I enter Ubuntu password.
Is there any solution?
I would start with checking with memory/CPU/network availability to check if it is not bottleneck.
Try following commands:
To check memory availability:
free -m
To check CPU usage:
top
To check network usage:
ntop
To check disk usage:
df -h
To check disk io operations:
iotop
Please also check if when you disable your application you are able to quickly log in to that machine. If login is still slow, then you should contact your EC2 support complaining about poor performance and asking for assigning more resources for that machine.
You can use WAIT Tool to diagnose what is wrong with your server or your application. The tool will gather all information about CPU and memoru utilization, running threads etc.
In addition, I would definitely check Tomcat application server with VisualVM or some other profiler. For configuring JMX for Tomcat you can check article here.
For network monitoring - nload tool is worth your attention. You can launch it in screen so you always check network utilization stats when server is slown.
First check is there any application using too much cpu or memory. This can be checked by using top command. I'll tell you two simple shortcut keys that may be helpful while using top command. In top command result page, if you enter M it will sort application based on memory usage, from highest to lowest. If you enter P it will sort application based on cpu usage, from highest to lowest.
If you are unable to find any suspicious application using top you can use iotop it will show disk I/O usage details.
I was facing the same issue, the solution which worked for me was
Restart the ec2 instance
Edit
lately, I figure out this issue is happening due to the fewer resources (memory, CPU) available to the EC2 machine. So check available resources to the EC2 machine.
Simple question can a Java service layer running on Tomcat7 on a host machine connect to persistent data store (mySQL) running inside a virtual box with portforwarding? I want to know if the hibernate or Jdbc connection strings from host machine work if mySQL server is installed inside a VirtualBox.
Also if it does work can I expect behavioral deviations in terms of speed and connection pooling if everything is packaged into one single system and deployed in a real world web server in a single enviroment?
The short answer is yes, it is possible and will work. You will likely have to play with the firewall settings on your virtual box instance. You don't specify OS, so it's hard to tell you what exactly you'll need to tweak.
As far as deploying this in a real-world environment, if you mean production, you probably should NOT do that. This is a great setup to build on, but not something I would run in production.
To be clear, there won't be any issues behaviorally speaking, it will act as MySQL always acts, but it will absolutely be slower than running it on 'bare metal' -- how much slower will vary based on hardware, workload, etc. and it is generally not a great design for a production deployment..
Are gears sometimes migrated (restarted?) between nodes when the nodes need maintainance, restarts or OS upgrades? Do nodes simply not go down, because fans, hard drives, RAM and CPUs are hotswapped?
How often should I expect my gears to be restarted (if at all)? For example, on EC2 where the hardware is inexpensive, the official figure is (used to be?) 3 months.
I know that there is no official promise of uptime, but what has the historical record been?
OpenShift is built on Amazon EC2. We can migrate gears to different nodes if needed.
Is there an easy way to setup an environment on one machine (or a VM) with MySQL replication? I would like to put together a proof of concept of MySQL replication with one Master write instance and two slave instances for reads.
I can see doing it across 2 or 3 VMs running on my computer, but that would really bog down my system. I'd rather have everything running on the same VM. What's the best way to proof out scalability solutions like this in a local dev environment?
Thanks for your help,
Dave
I think to truly test MySQL Replication it is important to do so in realistic constraints.
If you put all the replicate nodes under one operating system then you no longer have the bandwidth constraint, the data transfer speed would be much higher that what you would get if those replicate DBs are on different sites.
Everything under one VM is a shortcut to configurations, for instance it does not make you go through the configuration of the networking.
I suggest you use multiple VMs, even if you have to put them under one physical machine, you can always configure the hypervisor to make the packets go through a router, in which case the I/O will be bound by whatever the network interface has as throughput.
I can see doing it across 2 or 3 VMs
running on my computer, but that would
really bog down my system.
You can try and make a few VMs with JeOS (Just Enough OS) versions of the operating system you want. I know Ubuntu has one and it can boot on 128 RAM, which makes it convenient to deploy lots of cloned VMs under one physical machine without monster RAM.
Next step would be doing the same thing on a cloud (Infrastructure as a Service, IaaS) provider, and try your setup on different geographical sites.
If what you're testing is machine-to-machine replication, then setting up multiple VMs on a virtual private network would be the correct environment to test it. If you use Ubuntu Server, you don't have to install more than you actually need -- just give the VMs enough space for a base install + MySQL + your data. Memory usage can be as little as 256MB per VM. All you have to do is suspend or shutdown the VMs when you're not running a full-up test.
I've had situations where I was running 4 or more VMs simultaneously on my workstation, either for development or testing purposes -- it's not that taxing unless you're trying to do video rendering in each VM.