I have a bare qemu or kvm virtual machine and would like to observe or wait for the event that the virtual machine has booted in a reliable way.
This is a generic question that may not have a generic answer. If it helps you may assume a subset of the following:
The VM is running Debian GNU/Linux.
The actual question is whether the contained ssh server is reachable. It is exported via user networking and a hostfwd.
Remarks:
reliable means that it is suitable for continous integration testing. It should fail in less than 0.1% of cases.
Running ssh -o ConnectionAttempts=30 sometimes produces a failure even though ssh would work afterwards.
Related
Some software we have developed, we have "encapsulated it" into a virtual machine, that we run with Virtualbox, in command line in a non interactive way (no graphical interface). We send some instructions to the virtual machine, and it outputs some resulting files. We have tested this locally in a Linux machine. Now we would like to send this to many people using Linux, but we realize they will have different distributions, system libraries versions, etc, and then our VM might fail. So my question, is, it is possible to have someething like a static binary version of Virtualbox (or any other similar system / VM / Container) that does not need to use the system libraries, so that it can be run like a static binary?
It would be important to know what are the 'special' requirements of your solution regarding system libraries and the kind.
If you are using a standard host configuration, a standard VirtualBox install should be able to run the VM on any host OS.
Since a VM runs its own kernel, for the most part, is not dependent on host libraries. The the exception to this is when accessing/controlling host resources (disk, net, etc.). This being said, VirtualBox provides ways to access the most common resources (disk, net, etc.) that are transparent for the VM. Meaning that the VM will be configured always in the same way, regardless of whether the host is Win, Linux or Mac, and you can export your VM on Linux and importing it in other platforms without having to tweak it.
A container (eg. dockers) is more complicated, since it shares the kernel of the host, and it depends on how the host kernel is configured.
Again, if your application doesn't depend on 'special' access to host resources, a Docker will run the same way on all host OSs (Linux provides a native kernel, while Win and Mac run a linux virtual machine and then dockers inside it)
If you feel this doesn't answer your question, please share more details about the 'special' needs/configurations of your application, so we can dive deeper into this.
Have you looked at providing portable instances of the VM that can run on different host systems?
This example show how to create one for a windows host, check it out here, but I'm sure it can be done for different host systems as well.
For example, I'm running a Linux guest, and I want to do something like this in my init script just after boot:
savevm-somehow
run-lengthy-benchmark
I know how to use the monitor from the host, but it is hard to stop at the correct point to do the savevm: I could GDB step debug until there and then connect to the monitor, but that would be annoying.
In theory you could tell QEMU to put its monitor on a TCP port, and then also tell QEMU's networking to forward that port to the guest, and then from the guest connect to the forwarded port. I would worry about the possibility of deadlocks in this setup, though...
An approach I've used in the past is to script the QEMU monitor prompt using expect. There's an example here:
https://translatedcode.wordpress.com/2015/07/06/tricks-for-debugging-qemu-savevm-snapshots/
which uses a hardcoded delay time, but you ought to also be able to get expect to look at the serial port output to decide when to send the commands.
I am running an EC2 Instance on Ubuntu Server machine. Tomcat and MySQL are installed and deployed java web-application on it since 1 month. It was running good with great performance for almost 1 month but now my application is responding very slow.
Also, point to note is: Earlier when I used to log into my Ubuntu Server through PuTTY, it was quick but now its taking time even when I enter Ubuntu password.
Is there any solution?
I would start with checking with memory/CPU/network availability to check if it is not bottleneck.
Try following commands:
To check memory availability:
free -m
To check CPU usage:
top
To check network usage:
ntop
To check disk usage:
df -h
To check disk io operations:
iotop
Please also check if when you disable your application you are able to quickly log in to that machine. If login is still slow, then you should contact your EC2 support complaining about poor performance and asking for assigning more resources for that machine.
You can use WAIT Tool to diagnose what is wrong with your server or your application. The tool will gather all information about CPU and memoru utilization, running threads etc.
In addition, I would definitely check Tomcat application server with VisualVM or some other profiler. For configuring JMX for Tomcat you can check article here.
For network monitoring - nload tool is worth your attention. You can launch it in screen so you always check network utilization stats when server is slown.
First check is there any application using too much cpu or memory. This can be checked by using top command. I'll tell you two simple shortcut keys that may be helpful while using top command. In top command result page, if you enter M it will sort application based on memory usage, from highest to lowest. If you enter P it will sort application based on cpu usage, from highest to lowest.
If you are unable to find any suspicious application using top you can use iotop it will show disk I/O usage details.
I was facing the same issue, the solution which worked for me was
Restart the ec2 instance
Edit
lately, I figure out this issue is happening due to the fewer resources (memory, CPU) available to the EC2 machine. So check available resources to the EC2 machine.
We're using OpenNebula to simulate a simple replicated JBoss application.
We've installed all opennebula packages, qemu and kvm and libvirt.
We have created a simple ethernet network ad hoc between my pc (a node) and the one of my friend (which is both node and front-end) by plugging an ethernet cable between me and him (10.0.0.1 and 10.0.0.2).
So we can ping each other correctly, we've set everything to that we can ssh without a password to each other with "oneadmin" user.
We've configured all files such as below:
/etc/libvirt/libvirtd.conf
/etc/default/libvirtd-bin
And so on...
kvm and kvm-intel are both enabled.
The daemon
libvirtd -d -l
seems to start correctly.
In fact, from the gui of opennebula in the front end, we can see both the hosts monitored.
Anyway there's a problem when we try to start the virtual machine on the node which is not the front-end. I mean when we try to do a deploy of a VM on the other node. The error is something like this
cannot stat `/var/lib/one/datastores/1/f5394317d377beaa09fc07697df9ff68
but if, from the front end which has virtual machine n°1 we perform,
cd /var/lib/one/datastores/1
then we can see that file, we've also given all the permissions to it...
Any idea? :(
This may be related with the datastore configuration. If you left the default values, OpenNebula expects a shared filesystem (ie NFS) between the front-end and the virtualization nodes.
More context on the error (which I believe can be found in /var/lib/one/oned.log) would help analysing this problem.
I've not been able to find any documentation stating the existence of an API that can be used to automate things inside of a qemu guest.
For example, I would like to launch a process inside of the guest machine from the host machine. Libvirt does not appear to contain such functionality.
[Note: Automation without using any virtualization API. From my blog post.]
Step 1:
By default, QEMU uses SDL to display the VGA output. So, the first step is make this interaction with QEMU through standard I/O. QEMU provides an option for this.
From QEMU docs:
-nographic Normally, QEMU uses SDL to display the VGA output. With this option, you can totally disable graphical output so that QEMU is
a simple command line application. The emulated serial port is
redirected on the console. Therefore, you can still use QEMU to debug
a Linux kernel with a serial console.
So, all you have to do is invoke QEMU with -nographic.
qemu -nographic -hda guest.disk
Step 2:
Now that you can interact with your guest (or QEMU process) through command line, you have to automate this interaction. The obvious way to do this in python is start the QEMU process (with -nographic) with subprocess module and then communicate with that process. But to my surprise, this just didn’t work out for me. So, I looked for some other way.
Later, I found out that the most awesome tool for this kind of jobs is Expect. It is an automation tool for interactive applications written in TCL.
This guide should help you in getting started with Expect. Here is the script to run a guest with QEMU using Expect.
#!/usr/bin/expect -f
#starts guest vm, run benchmarks, poweroff
set timeout -1
#Assign a variable to the log file
set log [lindex $argv 0]
#Start the guest VM
spawn qemu -nographic -hda guest.disk
#Login process
expect "login: "
#Enter username
send "user\r"
#Enter Password
expect "Password: "
send "user\r"
#Do whatever you want to do with in the guest VM. ( Run a process and write result to log )
#poweroff the Guest VM
expect "# "
send "shutdown -h now\r"
The QEMU Monitor can interact with guest systems to a limited extent using it's own console. This includes reading registers, controlling the mouse/keyboard, and getting screen dumps.
There is a QEMU Monitor Protocol (QMP) that let's you pass JSON commands to and read values from the guest system.
As far as I know, the only way to communicate to the guest is through the network bridge.
I use python with pexpect to interact with spawned VMs using their serial consoles. I generally automate scenarios that have up to 128VMs this way, its reasonably swift. I generally use virt-install to instantiate guests, and use "virsh console (domainname)" using pexpect to get a "handle" to each console, so I can send commands to configure networking, startup tools/utilities/scripts, monitor operation, etc. Pretty sweet in terms of simplicity, and since the scripts are just issuing shell commands, you aren't exposed to APIs that change from version to version, e.g. the serial console will always be there. Sometimes I use qemu directly, (lately I am working with a QEMU that libvirt doesn't support since its too new), in that case I will have the guest console use a telnet port so I can "telnet localhost portnumber" to make a console connection instead of "virsh console (domainname)". Either way, python scripts with the pexpect module for interacting with VMs is great.
PyQemu can theoretically do this. I've used it in the past, although it looks like a stale project now. It provides a python agent (the equivalent of VMWare guest tools) to run on the guest, communicating with the host via serial port. You can get proxies to python modules running in the context of the VM, and any communication with them is marshaled over the serial port. In the following example, AutoIt is being used to automate Notepad:
machine = PyQemu.GetProxy("win2k")
# Wrap the machine object in another proxy representing the 'os'
# module running inside the VM.
os = PyQemu.vm.Module(machine,"os")
# NOTE: This is running on the VM!
os.system("notepad")
# Get an IDispatch object representing the autoit ActiveX control
autoit = PyQemu.vm.Dispatch(machine,"AutoItX3.Control")
# See if a window is active on the VM
state = autoit.WinActive("Untitled -")
Caveat: Due to using the serial port it is far from quick (regardless of serial speed settings), so perhaps best to transfer any bulk data by other means, e.g. Virtual FAT disk image.
You can create a reverse ssh tunnel from guest to host, which will redirect each request to host on specific port to guest. This way will help you to control guest from host.
If you're running Linux in the guest, couldn't you just use ssh/screen to launch remote processes on the guest?
Alternatively, I have seen people write python wrappers that use popen() to grab stdin/stdout and use those to automate some commands (i.e. when you see the login prompt, send the login name to stdin of QEMU.