Randomly can't connect to guest vm in libvirt - qemu

Question

I cannot reliably trigger this, although if I spin up many vms at a time and then attempt to connect to some of them, I run into this condition:
$ ping 192.168.122.135
PING 192.168.122.135 (192.168.122.135) 56(84) bytes of data.
From 192.168.122.1 icmp_seq=1 Destination Host Unreachable
From 192.168.122.1 icmp_seq=2 Destination Host Unreachable
From 192.168.122.1 icmp_seq=3 Destination Host Unreachable
Note that this does not happen for all VMs that I create and start, only a handful of them (randomly).
The vm that has obtained the ip 192.168.122.135 has the following for its network in its domain xml:
<interface type='network'>
<mac address='52:54:00:3d:72:ab'/>
<source network='default'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</interface>
And the default network is defined as (and yes, 22 vms are currently running):
<network connections='22'>
<name>default</name>
<uuid>69674b8b-f067-4513-b594-3e52360f391b</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr0' stp='on' delay='0'/>
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254'/>
</dhcp>
</ip>
</network>
The output from ifconfig for vnet0 (referenced by the VM's network domain xml) and virbr0 (used by the default network as shown above):
$ sudo ifconfig vnet0
vnet0 Link encap:Ethernet HWaddr fe:54:00:3d:72:ab
inet6 addr: fe80::fc54:ff:fe3d:72ab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:425 errors:0 dropped:0 overruns:0 frame:0
TX packets:1304 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:57503 (57.5 KB) TX bytes:67257 (67.2 KB)
and
$ sudo ifconfig virbr0
virbr0 Link encap:Ethernet HWaddr fe:54:00:08:e9:a4
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:882508 errors:0 dropped:0 overruns:0 frame:0
TX packets:2527165 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:93980992 (93.9 MB) TX bytes:3047773583 (3.0 GB)
Below is the partial output from ip route list:
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
The route output above makes me think that it should be working. BUT ITS NOT. and it only fails sometimes, and works most of the time.
Why can't I connect to the guest (192.168.122.135) from the host??
I was originally using filters, but removing the filters from the VM's domain xml has no effect on this condition randomly showing up. If I spin up many VMs at the same time I can get it to happen to a lot of them. Some of the VMs work just fine though and allow me to connect.
Also, I am using ubuntu 14.04.3:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
With kernel 3.19.0-30-generic.
More info - virsh version:
$ virsh --version
1.2.2
libvirtd version:
$ libvirtd --version
libvirtd (libvirt) 1.2.2

Answer 1

I don't have enough reputation to comment... But I have a few suggestions on things you could try to further explore the problem.
Question: Does assigning an IP address in the 192.168.122.X subnet on vnet0 do anything? The route that is configured seems to suggest that your traffic will go to virbr0 since it has the 192.168.122.1 IP address. If you can't ping any other devices in that subnet, then I suspect that's the issue.
If that doesn't get you anywhere...
Packet trace on host / VM
Try doing a packet dump on virbr0 and on the internal VM interface when this occurs. Ping the VM, and see what kind of traffic you see.
sudo tcpdump -n -i virbr0 -v "icmp or arp"
Depending on what you see there, will help narrow down the source of the problem. If you're not even getting your pings on that interface, then it's a routing issue on the host. If pings are going in, but the VM isn't seeing them, then it's a network/routing issue with the libvirt network.
I recommend also doing the above with a working VM, so you have a reference to compare the traffic against.
Check ARP Cache
Check your ARP cache on the host when this occurs. Does the mac address exist in the cache? Maybe it's getting mangled...
To dump the arp cache:
# arp
Check your libvirt logs
If configured, libvirt will log to syslog using the 'libvirtd' tag. Check your configuration to be sure this is enabled. It seems unlikely it's a libvirt issue, but it wouldn't hurt to turn on the logging.
To enable this setting
# vi /etc/libvirt/libvirtd.conf
Add the line
log_outputs_"1:syslog:libvirtd"
Restart libvirt
# service libvirt-bin restart

Answer 2

I had similar issue. I just tried following command to check whether machine is installed properly or not.
lsmod | grep kvm
If it is showing kvm details then machine is installed properly.
After that to restart the services
service libvirtd restart
Also check gateway using the below command
netstat -rn

Answer 3

I have the same network setting, and similar problem in a CentOS 7 host. Eventually, it turned out that the problem was guest VM's firewall setting blocked echo request and other external connection. After changing the firewall setting, the problem is solved.

Answer 4

My case, I've a hardware server where Libvirt is installed.
On this server I create VM in where install libvirt and after that I've get random network interruption and ping response with 192.168.122.1:
From 192.168.122.1 icmp_seq=1 Destination Host Unreachable
I've fixed this be deleting default libvirt network on hardware server like this:
virsh net-destroy default
virsh net-undefine default