Running Windows with PCI passthrough enabled on Gentoo Linux
- I have a 1080 GTX that I use to power my 2 1440p screens, render high definition video, re-encoding and also to play Dota 2 on Linux.
- I also foresee the card running CUDA specific stuff just out of personal interest.
- I hate rebooting machines (the box is also a ZFS datastore that is accessible over the network, music server, etc.).
- I want to play certain games on Windows (GTA V, Dying Light, etc.)
The requirements for pci passthrough can be found all over the internet, particularly in the Arch wiki, but in short: If you’ve an Intel processor it will need to support VT-d and IOMMU. Most modern processors support that. The QEMU wiki states that most K processors don’t - but the 8700k does support it.
The motherboard needs to support it as well (and again, most modern boards do) but some manufacturers are known to disable it to shave cost. Google around.
The MSI z370-A PRO board supports it. Couldn’t find it in a database somewhere but it does (tested).
I don’t want to have two separate KB and mouse because that beats the point of what I’m trying to achieve. I want a Linux host that can run games at times and sorta ‘alt-tab’ into each other. However, if you care about achieving bare metal latencies - you’d need to get a PCI USB hub that can be passed through.
The z370-A PRO motherboard comes with only 1 xHCI USB hub.
The rest of this post is tailored to hardware I own so
|Intel i7 8700k|
|MSI z370-A PRO|
|G.Skill Ripjaws V 16GB @ 3000MHz CL15|
|Corsair 128GB SATA3 SSD|
|Generic HID Keyboard|
|Generic HID Mouse|
The SSD is from another computer that no longer exists and was scavenged.
It is important that there’s two GPUs available on the system. The i7 8700k comes with its integrated graphics and that works fine.
Enable VT-d in Overclocking (smh) -> CPU features. I have no idea why it’s not enabled by default on this motherboard.
Set the boot GPU to the Intel i915 by navigating to Advanced -> Integrated Graphics Configuration -> Set to IGD.
When I boot from the 1080 GTX everything still works, but I can swap X between nvidia and intel gpus exactly two times. Any more and it’ll stall the CPU and I have to REISUB the machine. Details can be found in the email I sent to the vfio-users mailing list here.
I’m on gentoo-sources 4.17.9 at the time of writing. You need to keyword it so you’d need
Enable the following for IOMMU
I’ll also upload my kernel configuration so that it’s easier to reference.
Then add to
/etc/default/grub and reboot
Check if the options work by checking the log
alex@trixie ~ $ dmesg | grep -e IOMMU -e DMAR
Also enable the relevant Kernel flags for the Intel GPU and NVIDA GPU (see the gentoo handbook for this).
The i965 (newer i915 name in mesa) seems to flicker on my screen and cause
artifacts at times. The Arch wiki recommends adding
i915.enable_psr=0 to get
around that. You can find more details about that here.
I changed my display setup slightly to remove DP chaining. The final setup is like so
I removed the DP chaining because if you’re (and you should be) using the nvidia proprietary drivers on Linux, there’s a bug preventing DDC control from working over DP.
DDC control enables us to switch the inputs of the monitors from mDP <-> DP from software (i2c write). It works fine with the intel GPU which works out well for us since that’s the display that’ll be running X when we’re in passthrough.->
There’s a nice little utility called
ddccontrol that enables us to use this.
And IOMMU group is the smallest set of devices the VM can grab at a time.
You can check your IOMMU groups by running this little command that I got from many sources (Archwiki, /vg/s installgentoo page)
for iommu_group in \
IOMMU group 7
My IOMMU group 1 is perfect, which is basically just my nvidia GPU and its HD audio device. The PCI bridge can be ignored but it needs to be removed from the pcieport driver before we can bind the gpu to the vm.
Though if you have things like your network card inside the same group then grabbing it is going to be more difficult. You can try an ACS patch which may be able to break down your IOMMU groups further. I don’t need it with this setup. Yet.
X server stuff
Like I mentioned before the host graphics is equally important and I do more
work on the host. I use a generic
xorg.conf that is generated by
nvidia-xconfig to configure my graphics initially.
I leave that file untouched in
I have a seperate Xorg configuration that I use for the Intel i915
This filed is named
intel.xorg.conf and dumped in the same directory.
I no longer patch
xinit. I started using a login manager: SLiM, which also
starts a consolekit session (makes things easier). I keep the default config as
/etc/slim.conf.orig and symlink either that or
has the server argument
-config xorg.conf.intel to
Here’s how I start and stop my VM and bind to different environments. This is not perfect (neither is anything in this post) and criticism is welcome. Pull requests too.
I don’t use virt-manager or libvirt because
- I don’t want another layer of abstraction
- XML makes life difficult and there are still parameters that can’t be passed in via the configuration file
- Did I mention XML?
My scripts are
sh compatible and do not require any special shell.
This part prepares the drivers and binding to vfio-pci as required. I don’t think that the rebinding of vtconsole is required.
And then for the X server
And for when I’m done with the VM:
I’m using Qemu 3.1.0 at the time of updating this article.