Running x86/x86-64 applications on Fedora Asahi Remix

There are lots of lots of legacy x86/x86_64 applications that users want to run on arm64 platforms, including Windows applications and games. To support this in Fedora Asahi Remix, we have integrated a stack of existing and bespoke components to make it possible to transparently run x86/x86_64 apps directly on arm64 Linux.

Since Apple platforms use a 16K page size natively and x86/x86_64 processors use a 4K page size, this is especially tricky, as x86/x86_64 applications generally do not work when presented with a host kernel that requires 16K page alignment. To bridge this gap, we are using a microVM to run an entirely separate guest Linux kernel in 4K page size mode. To keep it as seamless as possible, the guest environment is designed to be as close as possible to the host environment, and we use native context GPU passthrough to have high-performance graphics inside the guest.

The stack consists of these components:

  • muvm (package: muvm), our bespoke microVM runner based on libkrun.

  • FEX-emu (package: fex-emu), a fast userspace x86/x86_64 emulator focused on correctness.

  • The Fedora FEX RootFS (package: fex-emu-rootfs-fedora), which provides common x86/x86_64 library dependencies to be used by emulated applications.

  • mesa (packages: mesa-fex-emu-overlay-i386 and mesa-fex-emu-overlay-x86_64), built for the x86/x86_64 architectures and packaged as a FEX RootFS overlay. This provides the OpenGL/OpenCL/Vulkan support for Apple GPUs.

  • sommelier (package: sommelier), a forwarding Wayland compositor for cross-domain window passthrough (will likely be replaced with a native X11 passthrough solution in the future).

  • hidpipe (package: hidpipe), a proxy for HID input devices to allow gamepads to work within the VM.

We also have our own Steam wrapper that automates the process of installing and launching Steam inside the microVM stack. When running Windows games using Steam, these open-source components are used behind the scenes:

  • Proton, a Wine distribution aimed at gaming.

  • dxvk, a translation layer that converts the Windows DirectX 8 - DirectX 11 APIs to Vulkan.

  • vkd3d-proton, a translation layer that converts the Windows DirectX 12 API to Vulkan.

This technology stack is currently primarily aimed at running x86 and x86-64 games, although in the future it should also be useful to run non-game productivity applications. When possible, you should prefer native alternatives over emulation. Please read this FAQ entry for more information.

Usage

Just use dnf install steam to install our Steam wrapper, and then run Steam from your desktop’s launcher (or the steam command) to download and install Steam. This will install all necessary dependencies automatically.

To run applications within a microVM manually, just prefix the command with muvm --. For example, muvm -- sh will give you a shell within the microVM environment. In this environment, the kernel’s binfmt support is already configured to use FEX to run x86/x86_64 applications, so you should be able to just run them. Note that we do not officially support applications outside of Steam right now, as there are a number of issues that can make the experience quite frustrating. This will change in the future.

How it works

muvm creates a virtual machine that shares as much with the host OS as possible. Within the VM, the root filesystem is the same as the host root filesystem, with the following exceptions:

  • /dev, /sys, and /proc are guest-private.

  • /run is also private to the guest

  • The FEX-emu rootfs and overlay images are mounted under /run/fex-emu/, with the combined overlay rootfs available at /run/fex-emu/rootfs.

  • /usr/share/fex-emu and /usr/local/share/fex-emu are overmounted with a tmpfs to inject a FEX Config.json suitable for use within the VM

  • A tmpfs is also mounted on /tmp/.X11-unix, so X11 server sockets are private to the VM

  • The entire host filesystem view is available at /run/muvm-host, including any overlaid mounts. For example, you can access the host’s /run at /run/muvm-host/run. (Note: /run/muvm-host/dev exists but will not do what you might hope it does. Host devices are not available in the guest.)

This means that /usr, /home, /etc, /opt, /var, /tmp, and any other directories in your filesystem root are shared between the guest and the host. The arm64 guest OS does not run its own root filesystem, but rather runs exactly the same binaries as your host OS does.

Additionally, FEX itself uses the filesystem mounted at /run/fex-emu/rootfs as its virtual RootFS. This means that x86/x86_64 applications (and only those) will see the contents of that directory overlaid on top of the root filesystem. This is how we make x86/x86_64 libraries available to those applications, while still sharing most of the filesystem contents.

When muvm starts, it registers FEX as a binfmt provider, so x86/x86_64 applications will be transparently run through it. On startup, FEX will detect that TSO support is available on the Apple Silicon platform (even within the VM), and automatically enable it for faster accurate emulation.

Known issues

Performance isn’t very good

As this project is still in its early stages, we aimed for correctness for the initial release. Performance optimization will happen over time. We’re aware of several changes that should bring significant performance improvements, and we’re actively working on it!

For Windows DX8-DX11 games under Proton in particular, you might want to try WineD3D instead of DXVK. WineD3D uses OpenGL instead of Vulkan as its backend, and it may have better performance thanks to optimizations in our OpenGL driver that are not available on Vulkan. To enable it, change the Steam launch options to PROTON_USE_WINED3D=1 %command%. Note that DXVK tends to have better compatibility, so this is a trade-off. Let us know what games work better using either backend!

Window management is weird (titlebar problems, popup problems, DPI problems, etc.)

These are known issues caused by bugs/limitations in Sommelier. We recommend using our x86/x86_64 stack only to run fullscreen applications (games, Steam in Big Picture mode) at this time. We’re working on a solution to this issue by replacing Sommelier with a more lightweight alternative.

If you need to access the regular Steam client, you can exit from Big Picture using Alt-F4. The menus may not work properly at this time. If you need to access Steam Settings, you can do so by clicking the Library tab, then clicking on the gear icon.

The VM uses a lot of RAM

To allow guest apps to use a large amount of RAM (as some modern games require), by default muvm allows the guest to use up to 80% of the system RAM. This also means that some of that will be taken up by guest page cache. Especially on lower RAM size machines (16GB or lower), we recommend not running any heavy host applications while the VM is in use. We don’t recommend gaming on 8GB machines.

You can configure the guest RAM allocation with the muvm --mem=SIZE parameter.

This will be less of an issue in the future, when we enable virtiofs-DAX in muvm. This directly maps host filesystem cache pages into the guest, and therefore relieves memory pressure.

I can’t access media mounted under /run/media within the VM

This does not work (not even via /run/muvm-host/run/media) due to missing POSIX ACL support in libkrun at this time. You must manually mount any disks that you wish to use within the VM, e.g. under /mnt.

FAQ

Is this like Rosetta on macOS?

This is as close to Rosetta as we can get! The main difference is that Rosetta side-steps the page size issue by instead relying on the XNU kernel’s multiple page size support for user processes, so it doesn’t need a VM. While making Linux support mixed page sizes would not be completely impossible in theory, it would be an enormous project that would likely take years to complete, and it isn’t at all clear whether such a change would be accepted upstream (Linux doesn’t even have boot-time page size selection within a single kernel yet!).

Other than the page size issue, FEX and Rosetta are comparable technologies (both are emulators, despite what Apple marketing might have you believe). Both FEX and Rosetta use the unique Apple Silicon CPU feature that is most important for x86/x86_64 emulation performance: TSO mode. Thanks to this feature, FEX can offer fast and accurate x86/x86_64 emulation on Apple Silicon systems.

Why not just use a 4K host kernel?

While Apple Silicon systems support 4K CPU pages, the rest of the hardware (IOMMUs, GPU) runs with 16K pages only. The Linux kernel does not play nicely in this environment, as it generally assumes that the CPU page size is at least as large or larger than the IOMMU page size. In the past we had some kernel patches to make this partially work, but they were buggy and incomplete, so we abandoned the approach. Even if it did work well, running the whole system using 4K pages has a measurable performance impact, so we would never ship 4K kernels by default. Therefore, running x86/x86_64 would require that users manually change their kernel and reboot, which is quite cumbersome.

Why not box64?

box64 and FEX-Emu have different approaches to emulation, with FEX-Emu aiming for better correctness by default (but requiring a more complex setup) while box64 aims to cover more "out of the box" use cases (like running a subset of applications directly on a 16K kernel without a VM using some tricks). We have chosen FEX-Emu for our stack because we believe it will have higher compatibility with its approach, but both have their uses. box64 is packaged in Fedora, so we encourage users to try it (both natively and within muvm) and let us know how it compares!

Steam says steamwebhelper crashed, what do I do?

Just let it restart, and it should work on the second try. Steam has a timeout for steamwebhelper, and when running under emulation, startup is slow enough that the timeout expires. This usually only happens on a cold startup.

I am unable to download/run certain games in Steam

Ensure Steam Play is enabled for all games. It should be under Menu > Settings > Compatibility > Enable Steam Play for all other titles. Restart Steam once enabled.

Pressing keys makes the touchpad stop responding

This is caused by the "Disable while typing" touchpad feature. You can turn it off in the touchpad/input settings for your desktop environment.

Can I run Windows applications outside of Steam?

At this point, we do not support running Windows apps outside of Steam for two reasons:

  • Non-Proton Wine does not yet work on Fedora + FEX due to FEX issues that we are working on.

  • Sommelier (the cross-vm compositor that we use currently) is quite buggy and not really suitable for non-fullscreen applications. We are working on a new solution that directly proxies the X11 protocol cross-domain to the host XWayland/compositor, which fixes all of these problems and makes the applications act as native X11 apps running on the host OS.

Once these issues are resolved, you will be able to use wine under muvm to run Windows apps.

Can I run x86_64/x86 Linux applications?

Native Linux games should generally work under muvm, as long as they are self-contained and do not depend on complex host system libraries (we ship a large selection of common dependencies, but not everything under the sun).

Non-game productivity apps may work, but window management will probably be quite broken due to Sommelier bugs (see the previous answer).

Is Wayland supported?

Wayland is not supported inside the VM at this time (Sommelier is a Wayland compositor, but it is used in an XWayland-exclusive mode). As most of the legacy x86/x86_64 applications people want to run are X11 applications, we are focusing on X11 support first. This means that you cannot run native Wayland apps inside the VM at this time. Of course, the host desktop is still a Wayland desktop, and X11 support is provided by XWayland.

Can I access hardware from applications running within the microVM?

As the VM does not pass through any host hardware other than the GPU and the virtual filesystem, you will not be able to use applications that require direct hardware access. We use software passthrough for the following interfaces:

  • X11 protocol (display, keyboard, mouse)

  • Gamepads via hid/uinput passthrough

  • Sound I/O via the PulseAudio socket protocol [1]

Is this like a Qemu/libvirt/UTM/Parallels/VMWare/VirtualBox/etc. VM?

No, muvm does not work like a traditional whole-system VM. While it does also use KVM as a backend for efficient virtualization, the concept is very different to traditional VMs running entirely separate guest operating systems. The guest kernel is a special kernel optimized to start up in a fraction of a second, and the VM monitor passes through the host filesystem mostly as-is. There is no low-level hardware passthrough (USB, etc.) and instead we focus on higher-level software protocol passthrough, like X11/Wayland. The VM does not run its own standalone init system, only some minimal startup code. This means that the environment within the VM should "feel" the same as the host OS from the point of view of applications, just with a 4K page size instead of a 16K page size.

Can I sudo inside the VM?

Since the VM monitor runs as your own user identity, it cannot gain root privileges. "root" inside the VM still only has the privileges of your own user, so sudo doesn’t make much sense (and in fact doesn’t work). We recommend installing software that you want to use with muvm+FEX under your home directory. For software that is designed to be installed under /opt or similar, we recommend performing the installation steps manually on the host OS, and then just running the app under muvm.

Can applications within the VM communicate with applications outside the VM?

Communication is mostly limited to the host filesystem. The VM shares your home directory (and in fact most of the filesystem) with the host, so any files you create on one side will be visible on the other.

In the future, we will enable virtiofs-DAX, which will allow for shared memory communication (/dev/shm) between guest apps and host apps, but this isn’t quite ready yet.

It is also possible to share audio between host and guest apps by using the PulseAudio forwarding support. For example, you can record guest audio by using a recording app on the host and recording from the system "Monitor" device. You can also configure virtual sinks/sources in the host using the normal PipeWire mechanisms, and direct guest apps to use those for audio I/O to have custom audio routing and processing. Note that the native PipeWire protocol is not passed through, only the PulseAudio protocol which is more limited (but more commonly used by applications). ALSA applications are supported via the pulse plug-in.

Why do I have fewer CPU cores inside the VM?

By default, muvm passes through as many CPUs as there are performance cores on your host machine, and pins those vCPUs to the physical performance cores. Since the host CPU scheduler has no visibility into the guest CPU scheduler, this ensures that performance is consistent. You can modify this behavior with the muvm --cpu-list=CPU_LIST option.

How do I use an external drive as a Steam library folder?

Follow these steps to get an external drive set up for Steam:

  1. Format your external drive using a Linux-native filesystem such as ext4.

    FAT32, exFAT, and other filesystems without support for Linux permissions will not work.
  2. Mount your drive manually under a directory accessible to muvm, such as /mnt/steam.

    We recommend mounting the drive manually (e.g. sudo mount /dev/sdX1 /mnt/steam, where sdX1 is your drive’s device file). If you configure the drive to mount automatically using /etc/fstab, then your system will not boot if that drive is not connected.
    The default mountpoint for drives mounted via the desktop environment (udisks) will not work at this time.
  3. Ensure that the filesystem root is accessible to your regular user:

    sudo chown ${USER}: /mnt/steam

  4. Create an empty folder named steamapps at the root of the mount:

    mkdir /mnt/steam/steamapps

  5. Start up Steam normally

    Steam will start up in Big Picture mode
  6. Exit Big Picture mode by pressing Alt+F4

  7. Click on the Library tab, then click the gear (settings) icon

    The settings window may pop up with the wrong size. Click and drag the resize handle on the bottom right to snap to the proper size.
  8. Select Storage on the left menu, click on the combo box at the top of the panel, and select Add Drive.

  9. Browse to your drive mountmount (/mnt/steam), such that the only folder visible in the file picker list window is the empty steamapps folder within, then (without making any further selections) click on the Select button.

You should now be able to select the new download location, make it the default, and download games to it.


1. This works with PipeWire running on the host with pipewire-pulse, as installed by default. You do not need to and should not install PulseAudio proper, as that will break your speaker support!