My Fedora 21 Gaming Rig using VT-D and VFIO without compromise!!
Fedora 21 was released recently, and naturally it was a good excuse to buy a new gaming rig, am I right?
Previously, I was happy with gaming on Linux with my current favourite, Dota 2 but new games were coming out which I just wanted to try, but, winblows...
So here's how I got setup with a Windows 8 VM, with VT-D passthrough, with almost the same level of performance.
- Asrock H97M PRO4 ($105)
- Asrock has better "support" for Linux, where as the popular Asus will full shut you down if you mention Linux.
- EVGA Geforce GTX 750TI ($169)
- Best output display ports, and small form factor.
- Intel i5-4590 ($249)
- Best price per performance other than the G3258, but the i5 is required for Vt-D
- Corsair CS550M 80+ Gold PSU ($119)
- PSU is what you should never skimp on. The power savings of using a 80+ GOLD, will pay for itself after 2-3 years, so it's a no brainer..
- Samsung 830 Series 128GB SSD (reused)
- I reused the SSD from my old rig
- Fractal Design Core 1100 Mini Tower Case ($54)
- Nice small case which looks decent and was cheap, it's internals aren't that bad but cable management is a little annoying.
- Kingston HyperX Fury Blue 1886Mhz Desktop Ram (4GBx2) ($95)
- RAM is RAM now a days, but I definitely need another 8GB as VMs are memory intensive with their overhead. Note, the MB is 1600Mhz but the price for 1886Mhz is only $1 extra..
Total: $791 AUD (from MSY)
Here are just some of my own personal tweaks and package installation. After the Fedora 21 installer has finished (which btw, has improved a lot since 20 -- thank you!!)
sudo -i echo 'vm.swappiness = 10' >> /etc/sysctl.conf yum -y update # Download Google Chrome here sudo yum localinstall --nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm -y # Install packages sudo yum update -y sudo yum install python pyxdg pygobject2 pylast gstreamer-python notify-python dbus-python gstreamer1-plugins-good gstreamer1-plugins-bad-free gstreamer1-plugins-ugly gstreamer-plugins-good gstreamer-plugins-bad gstreamer-plugins-ugly python-setuptools python-distutils-extra git -y sudo yum install nano tmux pithos -y # Enable TRIM cat <<'EOF' > /etc/cron.daily/fstrim.sh #! /bin/sh # By default we assume only / is on an SSD. # You can add more SSD mount points, separated by spaces. # Make sure all mount points are within the quotes. For example: # SSD_MOUNT_POINTS='/ /boot /home /media/my_other_ssd' SSD\_MOUNT\_POINTS='/ /home' for mount_point in $SSD_MOUNT_POINTS do fstrim $mount_point done EOF chmod +x /etc/cron.daily/fstrim.sh # skype sudo yum install lpf-skype lpf-skype sudo yum install alsa-plugins-pulseaudio.i686 # Use virt-preview for the latest goodies cd /etc/yum.repos.d/ wget http://fedorapeople.org/groups/virt/virt-preview/fedora-virt-preview.repo yum update yum install @virtualization
The magic begins here:
# Grab the EFI OVMF image wget https://www.kraxel.org/repos/firmware.repo yum install edk2.git-ovmf-x64 # Give time for guests to shutdown when host is powering off sed -i 's/#ON_SHUTDOWN=.*/ON_SHUTDOWN=shutdown/' /etc/sysconfig/libvirt-guests systemctl enable libvirt-guests systemctl enable libvirtd # Note down the PCI numbers [xxxx:xxxx] of the NVIDIA cards lspci -vvvvv lspci -nn yum -y install nano nano /etc/default/grub
Append to the end of the line:
Note, these values would come from the above lspci commands, they are comma seperated. I took all the NVIDIA ones. What this does, is prevents the host machine (Fedora 21) from claiming these PCI devices, leaving them "hanging" for our VM to later claim them.
YOURS WILL BE DIFFERENT
Now regenerate grub2 config, note EFI, if you didn't install with EFI, then it'll be in a different location (/boot/grub.cfg ?)
grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg reboot
Create a copy of our OVMF efi image.
cp /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd /var/lib/libvirt/images/win8-OVMF.fd restorecon -r /var/lib/libvirt/images/win8-OVMF.fd chmod 755 /var/lib/libvirt/images/win8-OVMF.fd
When you've booted back up, fire up virt-manager (it's gotten better, trust me). Now create a VM and add the PCI cards (all NVIDIA devices), USB keyboard, Mouse etc.
Attach the firmware file as a USB device so selinux labels it correctly (/var/lib/libvirt/images/win8-OVMF.fd). I couldn't find any other way to relabel it correctly as they seem to be assigned per VM.
Attach virtio drivers as CD disk, you can find these here http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/
Now we stopped our VM earlier because we wanted to make a few modifications with can't be done through virt-manager
virsh edit xyz # Append the following sections (between the ...). <domain type='kvm'> <features> ... <kvm> <hidden state='on'/> </kvm> ... </features> ... <os> ... <loader type='pflash'>/var/lib/libvirt/images/win8-OVMF.fd</loader> </os> # Make sure CPU mode is host-passthrough and the topology is the same. <cpu mode='host-passthrough'/> <topology sockets='1' cores='4' threads='1'/> </cpu> # Remove these lines <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> # Remove hypervclock <clock offset='localtime'> ... <timer name='hypervclock' present='yes'/> ... </clock>
These edits load up our EFI OVMF image as our bios loader, allowing us to passthrough our NVIDIA card without gimping our own host PC's performance. More info at this amazing blog site http://vfio.blogspot.com.au/2014/08/primary-graphics-assignment-without-vga.html
Finally, the other changes allow us to bypass the stupid Nvidia's ignorence that blocks the Nvidia drivers from being loaded if it detects that your running it in a VM.
Now fire up the VM through virt manager, attaching your Winblows ISO and watch as it speeds through the install (if you're on an SSD) and get gaming friends!!
Two major issue I noticed, which seems they may be linked is 8GB ram is just not enough. Right now, it's out of my budget to upgrade with another 8GB, but unless your host is running nothing, you may find even if you allocated 4GB to your guest (windows 8), it may end up using close to 6GB~. If the host is running some memory hungry application like Chrome, it'll start hitting swap space, which has negative effects on your VMs performance. I'm not sure if they may be linked, but the other issue is I'm noticing horrible artifacts appearing in my games after about 60-90 minutes of game play. When I'm not hitting swap space, it doesn't seem to happen (so far).
Hope this helps..
If you want to read more, and haven't already gone to playing games. This site has so much interesting read, you'll be stuck there a few hours http://vfio.blogspot.com.au/