Ubuntu Wiki

How to use Ubuntu as a hypervisor?

• Note Recommended 14.10 or later

Ubuntu is able to be used as a hypervisor, as an alternative to PowerKVM. Just install in bare metal (NV) it following the steps in Running Ubuntu non-virtualized on POWER8 After the installation you need:

1. Install the missing virtualization packages:

   # sudo apt-get install qemu-kvm qemu-user qemu-utils cloud-image-utils qemu-system-ppc qemu-slof libvirt-bin numactl

2. Add the users that would be able to run virtual machines into the kvm group:

   # sudo usermod -a -G kvm <user>

3. Disable the SMT:

   # sudo ppc64_cpu --smt=off You can disable it on every boot, changing /etc/rc.local, something as:

   1 #!/bin/sh -e
   2 #
   3 # rc.local
   4 #
   5 # This script is executed at the end of each multiuser runlevel.
   6 # Make sure that the script will "exit 0" on success or any other
   7 # value on error.
   8 #
   9 # In order to enable or disable this script just change the execution
  10 # bits.
  11 #
  12 # By default this script does nothing.
  13 
  14 ppc64_cpu --smt=off
  15 
  16 exit 0

Troubleshooting

1. If you kvm_HV module is not loaded automatically, enable KVM module in the kernel (Add a line "kvm_hv" in /etc/modules in order to automatically load it after a reboot).
2. If you hit "failed to parse default acl file `/etc/qemu/bridge.conf", you need to edit (or create) file /etc/qemu/bridge.conf and add:

   allow br0

3. If you hit "failed to create tun device: Operation not permitted", you need to set the setuid attribute:

   # chmod u+s /usr/lib/qemu/qemu-bridge-helper

PCI hot plug of devices that require 64-bit DMA window

Problem: Guest PCI device, which require 64-bit DMA window, hot plug causes guest to throw kernel fault and gets rebooted.

Solution: In order to perform hot plugging of PCI devices that use 64-bit DMA window to a KVM guest, make sure that the guest is started with a XHCI USB device.

For example, XML clause to add to libvirt guest XML is as follows:

• <controller type='usb' index='0' model='nec-xhci'> </controller>

If you are using the QEMU interface directly, please add below device option while booting the QEMU instance:

• -device nec-usb-xhci,id=u0

Enable 256 memory slots

In Ubuntu 17.04, when trying to increase the number of memory slots from 32 to 256, only 63 slots were available even though 256 slots was specified in the guest xml.

Machine Type = POWER8

Even though 256 slots are supported in QEMU 2.8, distributed with Ubuntu 17.04, there is an implicit limit coming from the vhost backend, limiting the number of max memory slots that are supported.

The work around is to increase the vhost mem slots by reloading the vhost module in the guest:

[root@localhost ~]# modprobe -r vhost [root@localhost ~]# modprobe vhost max_mem_regions=512 [root@localhost ~]# cat /sys/module/vhost/parameters/max_mem_regions 512

p8compat mode

If you are using a old Ubuntu version on a POWER9 hardware guest, you might want to enable p8compat mode, where the guest machine will behave as a POWER8.

In order to do so, you can either configure the the libvirt XML or the qemu.

 <cpu mode='host-model' check='partial'>
    <model fallback='forbid'>power8</model>
    <topology sockets='1' cores='16' threads='1'/>
  </cpu>

If you are starting the qemu manually, you can use -cpu host,compat=power8

CPU hot plug Limitations

Starting on Ubuntu 18.04 and POWER9, there is a limitation on KVM Hot plug CPU being addressed in Launchpad https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1759723.

At this time, it is not recommended to run the CPU with extra CPU to be hotplugged later. This should be fixed soon, but this is a near term limitation.

We basically lose the ability to hotplug/unplug CPUs into virtual machines. Considering that blk-mq enqueue tasks using 'for_each_present_cpu()' (that iterates over CPUs in systems that don't support CPU hotplug - refer to those patches for technical details) we can also have problems if tasks are scheduled to CPUs present but not online.

So I suggest to not run VMs with extra CPUs to be hotplugged later, like:

-smp 8,sockets=1,cores=2,threads=8,maxcpus=16

or

<vcpu placement='static' current='8'>16</vcpu>

If we want a guest with 16 CPUs it's better to define it beforehand, for instance:

-smp 16,sockets=1,cores=2,threads=8

or

<vcpu placement='static'>16</vcpu>

Reference and additional material

For more information about KVM on Power, please check here.