# Why?

• I have a 1080 GTX that I use to power my 2 1440p screens, render high definition video, re-encoding and also to play Dota 2 on Linux.
• I also foresee the card running CUDA specific stuff just out of personal interest.
• I hate rebooting machines (the box is also a ZFS datastore that is accessible over the network, music server, etc.).
• I want to play certain games on Windows (GTA V, Dying Light, etc.)

# Hardware

The requirements for pci passthrough can be found all over the internet, particularly in the Arch wiki, but in short: If you’ve an Intel processor it will need to support VT-d and IOMMU. Most modern processors support that. The QEMU wiki states that most K processors don’t - but the 8700k does support it.

The motherboard needs to support it as well (and again, most modern boards do) but some manufacturers are known to disable it to shave cost. Google around.

The MSI z370-A PRO board supports it. Couldn’t find it in a database somewhere but it does (tested).

I don’t want to have two separate KB and mouse because that beats the point of what I’m trying to achieve. I want a Linux host that can run games at times and sorta ‘alt-tab’ into each other. However, if you care about achieving bare metal latencies - you’d need to get a PCI USB hub that can be passed through.

The z370-A PRO motherboard comes with only 1 xHCI USB hub.

The rest of this post is tailored to hardware I own so

Components
Intel i7 8700k
MSI z370-A PRO
G.Skill Ripjaws V 16GB @ 3000MHz CL15
Corsair 128GB SATA3 SSD
Generic HID Keyboard
Generic HID Mouse

The SSD is from another computer that no longer exists and was scavenged.

It is important that there’s two GPUs available on the system. The i7 8700k comes with its integrated graphics and that works fine.

# BIOS Setup

Enable VT-d in Overclocking (smh) -> CPU features. I have no idea why it’s not enabled by default on this motherboard.

Set the boot GPU to the Intel i915 by navigating to Advanced -> Integrated Graphics Configuration -> Set to IGD.

When I boot from the 1080 GTX everything still works, but I can swap X between nvidia and intel gpus exactly two times. Any more and it’ll stall the CPU and I have to REISUB the machine. Details can be found in the email I sent to the vfio-users mailing list here.

# Kernel setup

I’m on gentoo-sources 4.17.9 at the time of writing. You need to keyword it so you’d need

Enable the following for IOMMU

I’ll also upload my kernel configuration so that it’s easier to reference.

Then add to /etc/default/grub and reboot

Check if the options work by checking the log

Also enable the relevant Kernel flags for the Intel GPU and NVIDA GPU (see the gentoo handbook for this).

The i965 (newer i915 name in mesa) seems to flicker on my screen and cause artifacts at times. The Arch wiki recommends adding i915.enable_psr=0 to get around that. You can find more details about that here.

# Display setup

I changed my display setup slightly to remove DP chaining. The final setup is like so

I removed the DP chaining because if you’re (and you should be) using the nvidia proprietary drivers on Linux, there’s a bug preventing DDC control from working over DP.

DDC control enables us to switch the inputs of the monitors from mDP <-> DP from software (i2c write). It works fine with the intel GPU which works out well for us since that’s the display that’ll be running X when we’re in passthrough.

There’s a nice little utility called ddccontrol that enables us to use this.

# IOMMU Groups

And IOMMU group is the smallest set of devices the VM can grab at a time.

You can check your IOMMU groups by running this little command that I got from many sources (Archwiki, /vg/s installgentoo page)

Mine are

My IOMMU group 1 is perfect, which is basically just my nvidia GPU and its HD audio device. The PCI bridge can be ignored but it needs to be removed from the pcieport driver before we can bind the gpu to the vm.

Though if you have things like your network card inside the same group then grabbing it is going to be more difficult. You can try an ACS patch which may be able to break down your IOMMU groups further. I don’t need it with this setup. Yet.

# X server stuff

Like I mentioned before the host graphics is equally important and I do more work on the host. I use a generic xorg.conf that is generated by nvidia-xconfig to configure my graphics initially.

I leave that file untouched in /etc/X11.

I have a seperate Xorg configuration that I use for the Intel i915

This filed is named intel.xorg.conf and dumped in the same directory.

Update:

I no longer patch xinit. I started using a login manager: SLiM, which also starts a consolekit session (makes things easier). I keep the default config as /etc/slim.conf.orig and symlink either that or /etc/slim.conf.intel which has the server argument -config xorg.conf.intel to /etc/slim.conf.

# Relevant scripts

Here’s how I start and stop my VM and bind to different environments. This is not perfect (neither is anything in this post) and criticism is welcome. Pull requests too.

I don’t use virt-manager or libvirt because

• I don’t want another layer of abstraction
• XML makes life difficult and there are still parameters that can’t be passed in via the configuration file
• XML
• Did I mention XML?

My scripts are sh compatible and do not require any special shell.

This part prepares the drivers and binding to vfio-pci as required. I don’t think that the rebinding of vtconsole is required.

And then for the X server

And for when I’m done with the VM: