How to workaround through the maze and reach the goal of the new amazing oVirt Hosted Engine with 3.4.0 Beta

Update I'm still working through a few issues and will update this page shortly so don't expect a perfect solution just yet.

Following from my previous post Deploying a semi-HA glusterized oVirt 3.3 Infrastructure on all the things I wanted for (except a simpler UI) have arrived!! That is:

  • oVirt Hosted Engine (chicken and egg scenario - host the engine within the cluster)
  • semi-libgfapi (native read/write speeds on the VM ontop of a glusterfs storage). This came with the release of RHEL/CentOS 6.5, I say semi because unfortunately there's a few limitations so only a workaround is in place for now.

So now lets take you through the exciting hosted-engine feature. There are two ways to do this:

  • Fresh Install (which is more ideal - I'll take you through the whole process)
  • Migrate Existing Engine (you'll need an extra host to add to your cluster)

My network configurations remain the same as before and we'll do all these on one host first:

oVirt Host

eth0 (Public Network) : 192.168.0.11
eth1 (management/gluster): bond0
eth2 (management/gluster): bond0

bond0.2 (management): 172.16.0.11
bond0.3 (gluster): 172.16.1.11

yum -y install wget screen nano
yum -y update
yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm -y
yum install http://resources.ovirt.org/releases/ovirt-release-el6-10.0.1-2.noarch.rpm -y

We'll use the nightly repo until stable comes a long as there are a few bugs still getting patched

nano /etc/yum.repos.d/el6-ovirt.repo  
    # enable nightly repo

yum -y install ovirt-hosted-engine-setup

nano /etc/hosts  
172.16.0.10 engine engine.lab.example.net  
172.16.0.11 hv01 hv01.lab.example.net  
172.16.0.12 hv02 hv02.lab.example.net

172.16.1.11 gs01 gs01.lab.example.net  
172.16.1.12 gs02 gs02.lab.example.net

# Use your local mirror - we'll be using this later in our hosted-vm
wget http://mirror.optus.net/centos/6/isos/x86_64/CentOS-6.5-x86_64-minimal.iso

yum install -y glusterfs glusterfs-fuse glusterfs-server vdsm-gluster

# This dataallignment is good with most disks but do your research first
pvcreate --dataalignment 2560k

vgcreate vg_gluster /dev/sda3  
lvcreate --extents 100%FREE -n lv_gluster1 vg_gluster

# Here's the secret sauce for ideal glusterfs xfs filesystem
mkfs.xfs -i size=512 -n size=8192 -d su=256k,sw=10 /dev/mapper/vg_gluster-lv_gluster1

# Add to fstab for onboot mounting
echo "/dev/mapper/vg_gluster-lv_gluster1 /data1  xfs     defaults,allocsize=4096,inode64,logbsize=256K,logbufs=8,noatime        1 2" >> /etc/fstab

mkdir -p /data1/  
mount -a  

Modify Network Nics

nano /etc/sysconfig/network-scripts/ifcfg-eth0  
DEVICE=eth0  
TYPE=Ethernet  
ONBOOT=yes  
NM_CONTROLLED=no  
BOOTPROTO=none  
IPADDR=192.168.0.11  
GATEWAY=192.168.0.1  
PREFIX=24


nano /etc/sysconfig/network-scripts/ifcfg-eth1  
DEVICE=eth1  
ONBOOT=yes  
NM_CONTROLLED=no  
BOOTPROTO=none  
MASTER=bond0  
SLAVE=yes

nano /etc/sysconfig/network-scripts/ifcfg-eth2  
DEVICE=eth2  
ONBOOT=yes  
NM_CONTROLLED=no  
BOOTPROTO=none  
MASTER=bond0  
SLAVE=yes

nano /etc/sysconfig/network-scripts/ifcfg-bond0  
DEVICE=bond0  
BOOTPROTO=none  
ONBOOT=yes  
NM_CONTROLLED=no  
BONDING_OPTS="miimon=100 mode=balance-alb"  
IPV6INIT=no  
# Enable Half-Jumbo Frames CRS only supports 4064
MTU=4000

nano /etc/sysconfig/network-scripts/ifcfg-bond0.2  
DEVICE=bond0.2  
ONBOOT=yes  
BOOTPROTO=none  
TYPE=Ethernet  
VLAN=yes  
BRIDGE=ovirtmgmt  
# Enable Half-Jumbo Frames CRS only supports 4064
MTU=4000

nano /etc/sysconfig/network-scripts/ifcfg-bond0.3  
DEVICE=bond0.3  
ONBOOT=yes  
BOOTPROTO=none  
TYPE=Ethernet  
VLAN=yes  
IPADDR=172.16.1.11  
PREFIX=24  
GATEWAY=172.16.1.1  
DEFROUTE=no  
# Enable Half-Jumbo Frames - CRS only supports 4064
MTU=4000

nano /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt  
DEVICE=ovirtmgmt  
NM_CONTROLLED=no  
ONBOOT=yes  
TYPE=Bridge  
BOOTPROTO=none  
IPADDR=172.16.0.11  
PREFIX=24  
GATEWAY=172.16.0.1  
DEFROUTE=yes  
IPV4\_FAILURE\_FATAL=yes  
IPV6INIT=no

# My BZ 1055129 may not have been fixed so just to be safe run we need to run hosted-engine twice to fix some vdsm issues.

hosted-engine --deploy  
# It will error here.
hosted-engine --deploy  
# Let it run first then cancel it when it prompts you for details (Ctrl-D)

# It's important to do this before restarting the network

# Restart Network
service network restart  

My Mikrotik CRS125 24g 1s RM unfortunately only supports a max MTU of 4064, but it's still better than 1500MTU. If you've got one run this quick script to get your MTU updated in bulk:

# Mikrotik Bulk Update MTU for ether7-ether24
/for i from=8 to=23 step=1 do={ /interface ethernet set $i l2mtu=4064 mtu=4064 }

Test New MTU (with another host with same MTU)

ping -s 3964 172.16.0.12

Keepalived Configuration

yum install -y keepalived

cat /dev/null > /etc/keepalived/keepalived.conf  
nano /etc/keepalived/keepalived.conf

# Node1 (copy to HV01)
vrrp_instance VI_1 {  
interface bond0.3  
state MASTER  
virtual\_router\_id 10  
priority 100   # master 100  
virtual_ipaddress {  
172.16.1.5  
}
}

# Node2 (copy to HV02)
vrrp_instance VI_1 {  
interface bond0.3  
state BACKUP  
virtual\_router\_id 10  
priority 99 # master 100  
virtual_ipaddress {  
172.16.1.5  
}
}

service keepalived start  
chkconfig keepalived on

The following workaround is important otherwise live migrations won't work!

#Work Around until libvirtd fixes the port conflict (http://review.gluster.org/#/c/6147/)
nano /etc/glusterfs/glusterd.vol  
    option base-port 50152


chkconfig glusterd on  
service glusterd start

curl https://raw.github.com/gluster/glusterfs/master/extras/group-virt.example -o /var/lib/glusterd/groups/virt

gluster volume create HOSTED-ENGINE gs01.lab.example.net:/data1/hosted-engine  
gluster volume start HOSTED-ENGINE  
gluster volume set HOSTED-ENGINE auth.allow 172.16.*.*  
gluster volume set HOSTED-ENGINE group virt  
gluster volume set HOSTED-ENGINE storage.owner-uid 36  
gluster volume set HOSTED-ENGINE storage.owner-gid 36  
gluster volume set HOSTED-ENGINE nfs.disable off  

There is currently a bug over here https://bugzilla.redhat.com/show_bug.cgi?id=1055153 where vdsm fails at weird points (and trust me I found many along with many workarounds). But all you need now is:

chown vdsm:kvm /var/log/vdsm/vdsm.log

Run It!

screen  
ovirt-hosted-engine-setup  

Follow the install procedure. You can safely leave the defaults for many. When it comes to the storage choose NFS and use the gluster share we setup:

172.16.1.5:/HOSTED-ENGINE

Don't be confused here - hosted-engine only supports NFS storage for the hosted-VM itself. It's a special export domain which is only used for the engine, we'll still be able to use our glusterfs storage for our other VMs.

Then don't forget the ISO file we grabbed earlier:

/root/CentOS-6.5-x86_64-minimal.iso

Once it setups the initial configuration settings the install process will be as follows:

  • The install will give you an IP Address to VNC into, this will get you connected to the screen of your hosted-engine VM which is running on your host.
  • Using the minimal CentOS 6.5 ISO we downloaded earlier we'll follow the normal install procedure or if you're game run it through a kickstart file.
  • When the install finishes let it reboot, and go back to your console session on the host to confirm your VM has installed the OS.
  • It'll give you the VNC session details again for you to go and install the ovirt-engine service in your new VM.

Hosted Engine VM Centos Install

yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm -y  
yum install http://resources.ovirt.org/releases/ovirt-release-el6-10-2.noarch.rpm -y  
yum -y install screen wget nano  
yum -y update

nano /etc/yum.repos.d/el6-ovirt.repo  
    # enable nightly repo

yum -y install ovirt-engine

# Install some useful packages such as ovirt-guest-agent
yum -y install dhclient ntp wget logrotate ntp openssh-clients rng-tools rsync tmpwatch selinux-policy-targeted nano perl ipa-client ipa-admintools screen acpid at curl ovirt-guest-agent

yum -y update

# Run through your setup the old fashion way
# If you're migrating from your engine, read the "cut off" section first http://www.ovirt.org/Migrate_to_Hosted_Engine

engine-setup

# Spice Proxy
yum install squid  
nano /etc/squid/squid.conf  
    # http_access deny CONNECT !SSL_ports
    http_access deny CONNECT !Safe_ports

    acl spice_servers dst 172.16.0.0/24
    http_access allow spice_servers

service squid restart  
chkconfig squid on  
iptables -A INPUT -p tcp --dport 3128 -j ACCEPT

# Setup a system wide spice proxy to save public IP addresses (if you're running behind NAT)
engine-config -s SpiceProxyDefault=http://192.168.0.10:3128/

service ovirt-engine restart

# Do your other post install features here #
  • Go back to your host confirm the engine has been installed. It'll do a quick check that the host can connect to the engine and then it will for you to shutdown the hosted-engine VM so the HA services can manage it.
  • Go ahead and power off the VM and your install is done!
  • Now the HA features of the hosted-engine will take care of keeping the VM alive.

Unfortunately there's still a bug https://bugzilla.redhat.com/show_bug.cgi?id=1055059 in regards to starting the VM. You you'll have to do the following in the mean time:

hosted-engine --vm-start-paused  
virsh start HostedEngine    

Now that we've got our engine setup - Let's quickly get tuned running to optimize our host:

cp -r /etc/tune-profiles/virtual-host /etc/tune-profiles/rhs-virtualization  
tuned-adm profile rhs-virtualization  

And because of our gluster base-port modification, don't forget to grab my custom iptables rules https://gist.github.com/andrewklau/7623169/raw/df69416e0386828d405845692c213b82e3f98e91/ovirt-engine-vdsm-iptables and drop them into /etc/sysconfig/iptables (remove the engine rules).

Finally - we create our final gluster volumes and configure them in the engine the same way we did before.

gluster volume create VM-DATA gs01.lab.example.net:/data1/vm-data  
gluster volume start VM-DATA  
gluster volume set VM-DATA auth.allow 172.16.*.*  
gluster volume set VM-DATA group virt  
gluster volume set VM-DATA storage.owner-uid 36  
gluster volume set VM-DATA storage.owner-gid 36  
# Help to avoid split brain
gluster volume set VM-DATA cluster.quorum-type auto  
gluster volume set VM-DATA performance.cache-size 1GB


chown -R 36:36 /data1

mkdir -p /storage/iso  
gluster volume set ISO auth.allow 172.16.*.*  
gluster volume set ISO storage.owner-uid 36  
gluster volume set ISO storage.owner-gid 36  

When you go to deploy the second host, follow the steps all the way till ovirt-hosted-engine-setup. When it prompts you for the storage connection it'll detect that it is joining a cluster and prompt you with the different steps.

Kudos to the Red Hat team for nice work and help. I hope you enjoy!

Check out this presentation here which is really interesting to how they calculate the score on which host will run the hosted-engine - along with their "jokes" and "comics".
http://www.linux-kvm.org/wiki/images/2/26/Kvm-forum-2013-hosted-engine.pdf

comments powered by Disqus