Deploying a semi-HA glusterized oVirt 3.3 Infrastructure

Time has passed since I last played with oVirt, the new ever so "amazing" openstack caught my attention but although it's got the momentem it's just not there yet in terms of easy deployment(maybe in a few months). So after a few weeks of playing with OpenStack we're back to oVirt.

There are a few issues I have with oVirt like the slow cluggy interface where as OpenStack has the lovely simple html5 bootstrap design. (Although there are talks to redesign the oVirt UI).

I guess you can compare the two as the Pet and Farm Animal scenario.

oVirt -> Pets -> Sick Pet (VM) -> Heal It -> Return to Health

OpenStack -> Farm Animals -> Sick Animal (VM) -> Replace It

In theory the two platforms should work hand in hand, and that's what Red Hat's currently doing. Many of the new oVirt features are taking advantage of OpenStack's fast pace development and integrating amazing new features. However, most people don't have the kind of hardware to deploy both OpenStack and oVirt/RHEV side by side (unless you've got the $$$$ to spend).

But to get a proper OpenStack infrastructure that can withstand node failures becomes expensive and tedious to configure and manage. oVirt on the other hand once you get the initial bits working "Just Works", and you can do it with minimal hardware.

I did this all on two CentOS 6.4 hosts.

Controller

eth0 (management/gluster): 172.16.0.11
eth1 (VM Data): n/a
eth2 (Public Network): 192.168.0.11

Compute

eth0 (management/gluster): 172.16.0.12
eth1 (VM Data): n/a

yum -y install wget screen nano  
yum -y update  
yum install http://resources.ovirt.org/releases/ovirt-release-el6-8-1.noarch.rpm -y  
yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm -y


nano /etc/hosts  
172.16.0.11 hv01.lab.example.net hv01  
172.16.0.12 hv02.lab.example.net hv02

# Don't install vdsm-gluster here because it seems to fail the install later on
yum install -y glusterfs glusterfs-fuse glusterfs-server  
mkfs.xfs -f -i size=512 /dev/mapper/vg_gluster-lv_gluster1  
echo "/dev/mapper/vg_gluster-lv_gluster1 /data1  xfs     defaults        1 2" >> /etc/fstab  
mkdir -p /data1/  
mount -a


curl https://raw.github.com/gluster/glusterfs/master/extras/group-virt.example -o /var/lib/glusterd/groups/virt

gluster volume create DATA replica 2 hv01.lab.example.net:/data1/ hv02.lab.example.net:/data1/  
gluster volume start DATA  
gluster volume set DATA auth.allow 172.16.0.*  
gluster volume set DATA group virt  
gluster volume set DATA storage.owner-uid 36  
gluster volume set DATA storage.owner-gid 36  
# Help to avoid split brain
gluster volume set DATA cluster.quorum-type auto  
gluster volume set DATA performance.cache-size 1GB


chown 36:36 /data1

mkdir -p /storage/iso  
gluster volume set STORAGE auth.allow 172.16.0.*  
gluster volume set STORAGE storage.owner-uid 36  
gluster volume set STORAGE storage.owner-gid 36

## MAKE SURE eth1 is set to "onboot=yes"
DEVICE=eth1  
TYPE=Ethernet  
ONBOOT=yes  
NM_CONTROLLED="no"  
BOOTPROTO=none  

The above is very much the same initial configs which I did on my Getting Started with Multi-Node OpenStack RDO Havana + Gluster Backend + Neutron VLAN as they are both very similar.

Now it's time to install the ovirt-engine. I'm looking forward to the hosted-engine solution which is being released soon which'll allow us to put the engine on a VM within the cluster!

# Engine Only
yum install -y ovirt-engine

# Follow the prompts - I found installing the All In One and putting VDSM on the engine had a few issues, so I do it later.

engine-setup

# I run everything in a NAT environment, so externally I need to use a proxy. However I still can't quite get this to work properly..

engine-config -s SpiceProxyDefault=http://proxy:8080  
service ovirt-engine restart  

Here's a quick list of steps I do to get my environment running:

  • Modify your cluster to include the Gluster Service
  • Install new host (hv01 and hv02). Do one at a time, and when you're installing hv01 (the one with the engine) uncheck "configure iptables".
  • If you get an install failed, generally it refers to vdsm-gluster. I have had two cases
    • One time it was installed and the engine complained. So I yum -y remove vdsm-gluster and re-ran the install.
    • The other time it wasn't installed and the engine complained. yum -y install vdsm-gluster (Don't ask me why this happens)
  • Now you should have 2 Hosts installed.
  • Remove ovirtmgmt as a VM network, go to the hosts tab and click setup networks. Edit ovirtmgmt and press resync.
    • While you're here may as well create a few VM networks. I created 10 networks each on their own VLANs. (I used the same VLAN Switch config from my previous post Mikrotik CRS125-24G-1S-RM with OpenStack Neutron)
    • Save your network settings from the ovirt UI (this component is a little buggy as it does network tests so you don't lose connectivity. You may have to try a few times and wait quite a while)

Redundant Gluster Deployment

Now we configure a keepalived redundant gluster volume using Keepalived. This means if one host goes offline, the other will still be able to keep connected to the gluster volume so we keep a semi HA infrastructure (the last part being ovirt-hosted-engine which is still in the works).

yum install -y keepalived

cat /dev/null > /etc/keepalived/keepalived.conf  
nano /etc/keepalived/keepalived.conf

# Node1 (copy this on HV01)
vrrp_instance VI_1 {  
interface ovirtmgmt  
state MASTER  
virtual\_router\_id 10  
priority 100   # master 100  
virtual_ipaddress {  
172.16.0.5  
}
}

# Node2 (copy this on HV02)
vrrp_instance VI_1 {  
interface ovirtmgmt  
state BACKUP  
virtual\_router\_id 10  
priority 99 # master 100  
virtual_ipaddress {  
172.16.0.5  
}
}

service keepalived start  
chkconfig keepalived on

# Work Around until libvirtd fixes the port conflict (http://review.gluster.org/#/c/6147/)
# Using this workaround, remember to include the port 50152 up till however many bricks you'll be using. My iptables gist file above is already updated. The oVirt host-deploy script will not apply the correct rules, so you need to do it manually!

nano /etc/glusterfs/glusterd.vol  
volume management  
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
    option transport-type socket,rdma
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off
    option base-port 50152
end-volume

chkconfig glusterd on  
service glusterd restart  
service glusterfsd restart  
  • Go to the datacenter and create a new DATA domain. I used a POSIX datacenter and mounted the volume as glusterfs. I'm eagerly looking forward to RHEL 6.5 which'll allow us to mount the gluster volumes directly on the VM for a huge performance boost

  • Create an ISO domain again use POSIX and mount it as a gluster volume. You can alternatively choose local storage or NFS too.

    • Upload the an ISO from the command line (there are talks about finally allowing it to be done through the UI).
wget http://mirror.centos.org/centos/6/isos/x86_64/CentOS-6.4-x86_64-minimal.iso  
engine-iso-uploader upload -i ISO CentOS-6.4-x86_64-minimal.iso  

oVirt has too many hidden gems which 5 months later I'm still discovering. Check out all the amazing features like:

  • Snapshots
  • VM Watchdog and HA
  • Vibrant and Active Community
  • Strong integration into OpenStack
  • Option to upgrade to the supported RHEV (which we'll be planning on doing when things take off).

If you don't think oVirt is ready just yet, check out these amazing features which I'm looking forward to:

  • libgfapi in RHEL 6.5 (to be released) which'll allow native gluster access to VM images.
  • ovirt-hosted-engine a true HA environment! The engine gets hosted as a VM within the cluster infrastructure and gets managed and brought up on a different node if it's current one fails.
  • oVirt UI redesign (YES PLEASE! I hate the old cluggy ui)

Tinker, configure, play!

comments powered by Disqus