Setting up Nested Virtualization

Peter Boy, Christopher Klooz Version F36-F37 Last review: 2023-00-ß0

This technique uses virtual machines - or more generally virtualization techniques - within an (already) virtualized machine. In Fedora, this works by default "out of the box" and basically requires no further configuration - provided that hardware support is available. This is true for nearly all Intel Core-i processors released in the last decade.

Introduction

With this technique, you can create a (nested) virtual machine running on an already virtualized Fedora Server (referred to as Level 1), which in turn runs on a physical Fedora Server host hypervisor (referred to as Level 0). The level 1 virtualized Fedora Server acts itself as hypervisor for a level 2 virtualized Fedora Server (or comparable virtualization).

Nested virtualization relies on host hardware virtualization extensions to work. For the Intel architecture, this has been a given for the Core-i processor family for more than a decade.

However, that does not mean it should be used as a standard configuration. The technology generates a high processor load and requires very powerful processors to work flawlessly. With older Intel processors, it is more suitable for exceptional cases or for testing and development.

Checking out the current configuration

The (physical) host CPU must support nested virtualization.
```
[…]$ sudo egrep --color 'vmx|svm|ept' /proc/cpuinfo
```
The output must include the vmx and ept flags. This is generally the case on Intel Core-i based CPUs.
Ensure that nested virtualization is enabled in the (physical) hosts kernel:
```
[…]$ sudo cat /sys/module/kvm_intel/parameters/nested
```
If the command returns 1 or Y, the feature is enabled. Continue with Setting up a VM for nested virtualization

If the command returns 0 or N but your system supports nested virtualization, continue here.
1. Execute the following commands to enable the nested virtualization feature
  […]$ sudo modprobe -r kvm_intel […]$ sudo modprobe kvm_intel nested=1
2. Ensure the operation is successfull
  […]$ sudo cat /sys/module/kvm_intel/parameters/nested
3. To make the change permanent and cope with a restart, edit
  […]$ sudo vim /etc/modprobe.d/kvm.conf options kvm_intel nested=1

Setting up a VM for nested virtualization

If not already done, create a Fedora Server Edition VM you want to use with nested virtualization. Follow the steps as described in Creating a virtual machine using Fedora Server Edition disk image.

Log in to the virtual server that is to host additional virtual machines.
Add virtualization support to this virtual Maschine. Follow the steps as described in Adding Virtualization Support.

Check the installation

[…]$ sudo  virt-host-validate qemu
 QEMU: Checking for hardware virtualization                                 : PASS
 QEMU: Checking if device /dev/kvm exists                                   : PASS
 QEMU: Checking if device /dev/kvm is accessible                            : PASS
 QEMU: Checking if device /dev/vhost-net exists                             : PASS
 QEMU: Checking if device /dev/net/tun exists                               : PASS
 ...

There may be 2 warings about IOMMU and secure guest support, which you can savely ignore.

Finally, create a Fedora Server Edition VM following the steps as described in Creating a virtual machine using Fedora Server Edition disk image and configure it as needed.

Optional: Customize the processor configuration

By default, the Prozessor of a virtual machine is a virtualized version of the host processor type. This way, nested virtualization works, but possibly with too large performance losses. A remedy may be a changed kernel configuration, especially a passthrough configuration.

The easiest way is to use the cockpit. Open a Cockpit session to the host system (l0 host), select Virtual machines and open the virtual host you want to optimize for nested virtualization, select "CPU type" edit.

Select host-passthrough in the mode drop down menu and then Apply.

The passthrough mode might resolve a performance issue for nested VMs, but usually comes with performance penalties in other areas. Be aware and review the entire system carefully.

This is also the place where you can configure a specific processor for the VM, if that is required.

Additional documentations

RHEL 8 docs: Chapter 18. Creating nested virtual machines

Want to help? Learn how to contribute to Fedora Docs ›