My Fedora 21 Gaming Rig using VT-D and VFIO without compromise!!

Fedora 21 was released recently, and naturally it was a good excuse to buy a new gaming rig, am I right?

Previously, I was happy with gaming on Linux with my current favourite, Dota 2 but new games were coming out which I just wanted to try, but, winblows...

So here's how I got setup with a Windows 8 VM, with VT-D passthrough, with almost the same level of performance.

Hardware:

  • Asrock H97M PRO4 ($105)
    • Asrock has better "support" for Linux, where as the popular Asus will full shut you down if you mention Linux.
  • EVGA Geforce GTX 750TI ($169)
    • Best output display ports, and small form factor.
  • Intel i5-4590 ($249)
    • Best price per performance other than the G3258, but the i5 is required for Vt-D
  • Corsair CS550M 80+ Gold PSU ($119)
    • PSU is what you should never skimp on. The power savings of using a 80+ GOLD, will pay for itself after 2-3 years, so it's a no brainer..
  • Samsung 830 Series 128GB SSD (reused)
    • I reused the SSD from my old rig
  • Fractal Design Core 1100 Mini Tower Case ($54)
    • Nice small case which looks decent and was cheap, it's internals aren't that bad but cable management is a little annoying.
  • Kingston HyperX Fury Blue 1886Mhz Desktop Ram (4GBx2) ($95)
    • RAM is RAM now a days, but I definitely need another 8GB as VMs are memory intensive with their overhead. Note, the MB is 1600Mhz but the price for 1886Mhz is only $1 extra..

Total: $791 AUD (from MSY)

Here are just some of my own personal tweaks and package installation. After the Fedora 21 installer has finished (which btw, has improved a lot since 20 -- thank you!!)

sudo -i  
echo 'vm.swappiness = 10' >> /etc/sysctl.conf

yum -y update  
# Download Google Chrome here

sudo yum localinstall --nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm  -y

# Install packages
sudo yum update  -y  
sudo yum install python pyxdg pygobject2 pylast gstreamer-python notify-python dbus-python gstreamer1-plugins-good gstreamer1-plugins-bad-free gstreamer1-plugins-ugly gstreamer-plugins-good gstreamer-plugins-bad gstreamer-plugins-ugly python-setuptools python-distutils-extra git  -y  
sudo yum install nano tmux pithos  -y

# Enable TRIM
cat <<'EOF' > /etc/cron.daily/fstrim.sh  
#! /bin/sh  

# By default we assume only / is on an SSD. 
# You can add more SSD mount points, separated by spaces.
# Make sure all mount points are within the quotes. For example:
# SSD_MOUNT_POINTS='/ /boot /home /media/my_other_ssd'  

SSD\_MOUNT\_POINTS='/ /home'  

for mount_point in $SSD_MOUNT_POINTS  
do  
    fstrim $mount_point  
done  
EOF  
chmod +x /etc/cron.daily/fstrim.sh

# skype
sudo yum install lpf-skype  
lpf-skype  
sudo yum install alsa-plugins-pulseaudio.i686

# Use virt-preview for the latest goodies
cd /etc/yum.repos.d/  
wget http://fedorapeople.org/groups/virt/virt-preview/fedora-virt-preview.repo

yum update  
yum install @virtualization  

The magic begins here:

# Grab the EFI OVMF image
wget https://www.kraxel.org/repos/firmware.repo  
yum install edk2.git-ovmf-x64

# Give time for guests to shutdown when host is powering off
sed -i 's/#ON_SHUTDOWN=.*/ON_SHUTDOWN=shutdown/' /etc/sysconfig/libvirt-guests  
systemctl enable libvirt-guests  
systemctl enable libvirtd

# Note down the PCI numbers [xxxx:xxxx] of the NVIDIA cards
lspci -vvvvv  
lspci -nn

yum -y install nano  
nano /etc/default/grub  

Append to the end of the line: GRUB_CMDLINE_LINUX with:

intel_iommu=on pci-stub.ids=10de:1380,10de:0fbc,8086:0c0c,8086:8ca0  

Note, these values would come from the above lspci commands, they are comma seperated. I took all the NVIDIA ones. What this does, is prevents the host machine (Fedora 21) from claiming these PCI devices, leaving them "hanging" for our VM to later claim them.

YOURS WILL BE DIFFERENT

Now regenerate grub2 config, note EFI, if you didn't install with EFI, then it'll be in a different location (/boot/grub.cfg ?)

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

reboot

Create a copy of our OVMF efi image.

cp /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd /var/lib/libvirt/images/win8-OVMF.fd  
restorecon -r /var/lib/libvirt/images/win8-OVMF.fd  
chmod 755 /var/lib/libvirt/images/win8-OVMF.fd  

When you've booted back up, fire up virt-manager (it's gotten better, trust me). Now create a VM and add the PCI cards (all NVIDIA devices), USB keyboard, Mouse etc.

Attach the firmware file as a USB device so selinux labels it correctly (/var/lib/libvirt/images/win8-OVMF.fd). I couldn't find any other way to relabel it correctly as they seem to be assigned per VM.

Attach virtio drivers as CD disk, you can find these here http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/

Now we stopped our VM earlier because we wanted to make a few modifications with can't be done through virt-manager

virsh edit xyz


# Append the following sections (between the ...).
<domain type='kvm'>  
  <features>
    ...
    <kvm>
      <hidden state='on'/>
    </kvm>
    ...
  </features>
 ...
  <os>
    ...
    <loader type='pflash'>/var/lib/libvirt/images/win8-OVMF.fd</loader>
  </os>

# Make sure CPU mode is host-passthrough and the topology is the same.
<cpu mode='host-passthrough'/>  
   <topology sockets='1' cores='4' threads='1'/>
</cpu>

# Remove these lines
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>

    </hyperv>

# Remove hypervclock

  <clock offset='localtime'>
    ...
    <timer name='hypervclock' present='yes'/>
    ...
  </clock>

These edits load up our EFI OVMF image as our bios loader, allowing us to passthrough our NVIDIA card without gimping our own host PC's performance. More info at this amazing blog site http://vfio.blogspot.com.au/2014/08/primary-graphics-assignment-without-vga.html

Finally, the other changes allow us to bypass the stupid Nvidia's ignorence that blocks the Nvidia drivers from being loaded if it detects that your running it in a VM.

Now fire up the VM through virt manager, attaching your Winblows ISO and watch as it speeds through the install (if you're on an SSD) and get gaming friends!!

Two major issue I noticed, which seems they may be linked is 8GB ram is just not enough. Right now, it's out of my budget to upgrade with another 8GB, but unless your host is running nothing, you may find even if you allocated 4GB to your guest (windows 8), it may end up using close to 6GB~. If the host is running some memory hungry application like Chrome, it'll start hitting swap space, which has negative effects on your VMs performance. I'm not sure if they may be linked, but the other issue is I'm noticing horrible artifacts appearing in my games after about 60-90 minutes of game play. When I'm not hitting swap space, it doesn't seem to happen (so far).

Hope this helps..

If you want to read more, and haven't already gone to playing games. This site has so much interesting read, you'll be stuck there a few hours http://vfio.blogspot.com.au/

comments powered by Disqus