I have a 1080 GTX that I use to power my 2 1440p screens, render high
definition video, re-encoding and also to play Dota 2 on Linux.
I also foresee the card running CUDA specific stuff just out of personal
I hate rebooting machines (the box is also a ZFS datastore that is accessible
over the network, music server, etc.).
I want to play certain games on Windows (GTA V, Dying Light, etc.)
The requirements for pci passthrough can be found all over the internet,
particularly in the Arch wiki, but in short: If you’ve an Intel processor it
will need to support VT-d and IOMMU. Most modern processors support that. The
QEMU wiki states that most K processors don’t - but the 8700k does support it.
The motherboard needs to support it as well (and again, most modern boards do)
but some manufacturers are known to disable it to shave cost. Google around.
The MSI z370-A PRO board supports it. Couldn’t find it in a database somewhere
but it does (tested).
I don’t want to have two separate KB and mouse because that beats the point of
what I’m trying to achieve. I want a Linux host that can run games at times and
sorta ‘alt-tab’ into each other. However, if you care about achieving bare
metal latencies - you’d need to get a PCI USB hub that can be passed through.
The z370-A PRO motherboard comes with only 1 xHCI USB hub.
The rest of this post is tailored to hardware I own so
Intel i7 8700k
MSI z370-A PRO
G.Skill Ripjaws V 16GB @ 3000MHz CL15
Corsair 128GB SATA3 SSD
Generic HID Keyboard
Generic HID Mouse
The SSD is from another computer that no longer exists and was scavenged.
It is important that there’s two GPUs available on the system. The i7 8700k
comes with its integrated graphics and that works fine.
Enable VT-d in Overclocking (smh) -> CPU features. I have no idea why it’s not
enabled by default on this motherboard.
Set the boot GPU to the Intel i915 by navigating to Advanced -> Integrated
Graphics Configuration -> Set to IGD.
When I boot from the 1080 GTX everything still works, but I can swap X between
nvidia and intel gpus exactly two times. Any more and it’ll stall the CPU and
I have to REISUB the machine. Details can be found in the email I sent to the
vfio-users mailing list here.
I’m on gentoo-sources 4.17.9 at the time of writing. You need to keyword it so
Also enable the relevant Kernel flags for the Intel GPU and NVIDA GPU (see the
gentoo handbook for this).
The i965 (newer i915 name in mesa) seems to flicker on my screen and cause
artifacts at times. The Arch wiki recommends adding i915.enable_psr=0 to get
around that. You can find more details about that here.
I changed my display setup slightly to remove DP chaining. The final setup is
I removed the DP chaining because if you’re (and you should be) using the nvidia
proprietary drivers on Linux, there’s a bug preventing DDC control from working
DDC control enables us to switch the inputs of the monitors from mDP <-> DP from
software (i2c write). It works fine with the intel GPU which works out well for
us since that’s the display that’ll be running X when we’re in passthrough.->
There’s a nice little utility called ddccontrol that enables us to use this.
IOMMU group 7 00:1c.0 PCI bridge : Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0) IOMMU group 5 00:16.0 Communication controller : Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba] IOMMU group 3 00:08.0 System peripheral : Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model [8086:1911] IOMMU group 11 03:00.0 Ethernet controller : Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15) IOMMU group 1 00:01.0 PCI bridge : Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07) 01:00.0 VGA compatible controller : NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1) 01:00.1 Audio device : NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1) IOMMU group 8 00:1c.3 PCI bridge : Intel Corporation 200 Series PCH PCI Express Root Port #4 [8086:a293] (rev f0) IOMMU group 6 00:17.0 SATA controller : Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282] IOMMU group 4 00:14.0 USB controller [0c03]: Intel Corporation 200 Series PCH USB 3.0 xHCI Controller [8086:a2af] 00:14.2 Signal processing controller : Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1] IOMMU group 12 04:00.0 Non-Volatile memory controller : Samsung Electronics Co Ltd Device [144d:a808] IOMMU group 2 00:02.0 VGA compatible controller : Intel Corporation Device [8086:3e92] IOMMU group 10 00:1f.0 ISA bridge : Intel Corporation Device [8086:a2c9] 00:1f.2 Memory controller : Intel Corporation 200 Series PCH PMC [8086:a2a1] 00:1f.3 Audio device : Intel Corporation 200 Series PCH HD Audio [8086:a2f0] 00:1f.4 SMBus [0c05]: Intel Corporation 200 Series PCH SMBus Controller [8086:a2a3] IOMMU group 0 00:00.0 Host bridge : Intel Corporation Device [8086:3ec2] (rev 07) IOMMU group 9 00:1d.0 PCI bridge : Intel Corporation 200 Series PCH PCI Express Root Port #9 [8086:a298] (rev f0)
My IOMMU group 1 is perfect, which is basically just my nvidia GPU and its HD
audio device. The PCI bridge can be ignored but it needs to be removed from the
pcieport driver before we can bind the gpu to the vm.
Though if you have things like your network card inside the same group then
grabbing it is going to be more difficult. You can try an ACS patch which may be
able to break down your IOMMU groups further. I don’t need it with this setup.
X server stuff
Like I mentioned before the host graphics is equally important and I do more
work on the host. I use a generic xorg.conf that is generated by
nvidia-xconfig to configure my graphics initially.
I leave that file untouched in /etc/X11.
I have a seperate Xorg configuration that I use for the Intel i915
This filed is named intel.xorg.conf and dumped in the same directory.
I no longer patch xinit. I started using a login manager: SLiM, which also
starts a consolekit session (makes things easier). I keep the default config as
/etc/slim.conf.orig and symlink either that or /etc/slim.conf.intel which
has the server argument -config xorg.conf.intel to /etc/slim.conf.
Here’s how I start and stop my VM and bind to different environments. This is
not perfect (neither is anything in this post) and criticism is welcome. Pull
I don’t use virt-manager or libvirt because
I don’t want another layer of abstraction
XML makes life difficult and there are still parameters that can’t be passed
in via the configuration file
Did I mention XML?
My scripts are sh compatible and do not require any special shell.
This part prepares the drivers and binding to vfio-pci as required. I don’t
think that the rebinding of vtconsole is required.