Deploying a high available OpenShift Origin 3 (now M4) Infrastructure with Ansible and FreeIPA on CentOS 6.5

I've been having quite a few questions about how I put together my OpenShift Origin environment from many jelous friends who were still manually spinning up test VMs.. I mean come on who wants to keep doing that :) So here goes my post on deploying openshift!

If you've been following the PaaS movements, OpenShift Origin M4 recently hit the mirrors and there's a damn fine list of new improvements - my favourite the new console interface and my own commit which allows MariaDB cartridges on RHEL based nodes (previously only available on Fedora).

In this post I'm going to run through the basic concept of installing a high available OpenShift Origin 3 deployment using my ansible script https://github.com/andrewklau/openshift-ansible. I expect you to know what it already does and is - if you don't look around first and come back. I wrote this because currently:

I'm not going to cover the FreeIPA install, as it's very simple and pretty amazing once you understand the concept (I use a 2 host replica behind keepalived).

OpenShift Infrastructure Design

Diagram Source: https://github.com/ansible/ansible-examples/tree/master/openshift

First things first lets do some initial prep and ground work. First you'll need atleast 8 VMs plus a working FreeIPA deployment. These are then broken down into:

  • 2x LVS Servers (active/backup style design we use these for load balancing the load between the broker components and do some redirects)
  • 2x Brokers Servers (these are like your entry points to the to your PaaS environment. They will interact with all the other components on your behalf. Keep in mind, these are standalone, stateless style servers which is why we can run LVS infront of these)
  • 3x Broker Support Nodes (these servers will run your PaaS datastore and amqp routing. MongoDB and ActiveMQ respectively.)
  • 1..N Node Servers (these are where the applications will be hosted and run.)
  • DNS (this is simply our FreeIPA cluster)

Ideal/Minimum Resource Allocation:

  • LVS Servers can run on your lowest spec'd host/VM as all the routing process is done on the kernel level aka SUPER FAST:
    • 1-2 Core CPU
    • 512mb-1GB RAM
    • 15GB HDD (atleast)
    • 2x Gigabit Nics (optional 10 Gigabit Nics)
  • Broker Servers are your main point of entry servers. Since these are fairly stateless they can be scaled out horizontally depending on your load.
    • 2-4 Core CPU
    • 2-4GB RAM
    • 15GB HDD (atleast)
    • 1x Gigabit NIC (optional 10 Gigabit NIC)
  • Broker Support Nodes play a key role in the infrastructure. They can be scaled out horizontally but keep in mind the cluster quorum requirements of ActiveMQ and MongoDB (eg. minimum 3 servers in the cluster).
    • 2-4 Core CPU
    • 2-8GB RAM (4GB Ideally)
    • 25GB HDD (minimum)
    • 1x Gigabit NIC (optional 10 Gigabit NIC)
  • Node Servers are your bread and butter. These need to be spec'd out the most to ensure high performance for the sites and services you'll be hosting. but always remember to think cloud - scale out horizontally is better than vertically. Deploying hundreds of nodes will scale and utilize the OpenShift PaaS infrastructure a lot better than one super power PC. (think redundancy, failover, bottlenecks etc)
    • 4-8 Core CPU
    • 4-16GB RAM
    • 50GB HDD (minimum)
    • 2x Gigabit NIC (optional 10 Gigabit NIC)

Few Things to Note:

  • All data is stored on the Nodes individually, there is no replication of this data. I'd love to see some sort of integration with glusterfs some point in the future though. Do not confuse this with the PaaS datastore. PaaS datastore holds the information such as application routing, names etc.
  • ActiveMQ is used in collaboration with mcollective to share details between the nodes and broker servers.
  • If you're simply building a proof of concept, you can even run this with just one node but you'll need atleast 3 Broker Support Nodes.
  • The whole infrastructure is designed to run in a private network. The only ones you'll need to port forward (if behind NAT or on AWS) are the hosts with dual NICs (eg. LVS and Node Servers). You can modify it all to be public facing but please update the firewall rules!
  • All hosts are already registered as a host within FreeIPA with the hostname formating of:
    • oo-service.lab.example.net
      • eg. oo-lvs1.lab.example.net
      • oo-node1.lab.example.net
  • My ansible script will also install nginx on the broker hosts to do a simple catch-all URL forwarding service. I found the one issue with OpenShift's routing is it relied on CNAMEs which unfortunately cannot be run on the @ of your domain. eg. you would never be able to host example.net on OpenShift. The workaround is to set the A address of your floating LVS IP address and it will redirect any * to www.*
    • Example: example.com -> LVS Floating IP Address (Nginx) -> www.example.com

I deployed all of these VMs with the help of Foreman onto of oVirt. So it made sense to start my ansible install script from the foreman host. However you can run ansible on any host connected to your network of VMs (even your laptop).

Warning! Warning!

Before you continue - I advice you to read the script first and really make sure you can roll back (eg. re-install OS or have snapshots etc.)

Source: https://github.com/andrewklau/openshift-ansible

Ansible is very easy to understand once you grab the concept that each host can be assigned role(s) and each role will have a set of tasks. Those tasks will then be run in sequential order to complete the install.

eg. roles/nodes/tasks/main.yml

Read through all those roles and tasks just to make sure you know what's happening. Usual la-di-dah I'm not responsible if it break something.

Foreman Host

# Make sure you have EPEL installed
yum -y install ansible git  
git clone https://github.com/andrewklau/openshift-ansible.git

cd openshift-ansible  
# Modify the config variables
nano group_vars/all  

I've added some extra notes below to help you modify the group_vars/all file

# Leave As Is
iface: '{{ ansible_default_ipv4.interface }}'

# This is your domain name eg. rhcloud.com
domain_name: cloud.example.net

# Leave Default
dns_port: 53  
rndc_port: 953

# Ignore this as we aren't using dns_keys for BIND updates
dns_key: "+GPQn8ufEpRLk5xgUU+W3CXdD5CKLO5hP4kCy1PRngV26V0eHnBrJF55IWw0HZme6mXJAgn7gkFMQbuQGq7tLQ=="

# Directory to store mongodb data (can leave as is)
mongodb\_datadir\_prefix: /data  
mongod_port:  2700  
# Set a secure mongo admin password
mongo\_admin\_pass: skldflk

# Set secure passwords for these services
mcollective_pass: asdasd  
admin_pass: asdasd  
amquser_pass: asdasd

# This is the LVS floating IP Address - if you aren't using NAT this should be a public address
vip: 10.0.0.4  
vip_netmask: 255.255.255.0

# Kerberos Configuration (make sure it matches your FreeIPA infrastructure)
kerb\_ipa\_server: ipa01.oo.example.net  
kerb_domain: oo.example.net  
kerb_realm: EXAMPLE.NET

### Remove this Uncommented Info Section ###

Node Public Configuration  
You will need to manually create the following A records on your DNS server. This is your PUBLIC ip address. 

The format is:  
{{ ansible_hostname }}-{{ public_domain_suffix }}.{{ public_domain }}

eg. oo-node1-lab.example.net

Notice how this format is slightly different to the above scheme of oo-node1.lab.example.net this is because FreeIPA does not yet support Split DNS. This is only used for the node's dynamic hostname generation.

## End Info Section ###

public_domain: example.net  
public\_domain\_suffix: lab

# FreeIPA DNS Configuration (dirty method)
ipa\_bind\_ip: 10.0.1.10  
ipa_server: ipa01.lab.example.net  
kinit_user: admin  
kinit_password: adminpassword

# Broker Hostname (used for Nginx)
broker_hostname: oo-broker.example.net  

Generate random string and replace contents in /roles/mongodb/files/secret

openssl rand -base64 741  

Generate new salts for authsalt and sessionsecret in /roles/broker/vars/main.yml

openssl rand -base64 64  

eg.

auth_salt: "lk32rsjdpdgSK+BacvsrfdQi6pDW7HMen3uJFykUwOQBoCsvqNZ68po9+N8w=="  
session_secret: "skdlfj3klSDFkjsdO+Yao9VoUx69ikwfiRdph9oXplWDaQ10yWV8y0iFiCf8lTzj40M6b9NIV+wtIuLv/Y/ODjmtJ399g=="  

Regenerate some new shared SSL Certs for inter server communication. Don't use FreeIPA ipa-getcert for this

cd roles/broker/files  
openssl genrsa -out server_priv.pem 2048 openssl rsa -in server_priv.pem -pubout > server_pub.pem  

Modify the hosts file to match your current infrastructure. Make sure all the hostnames resolve!

foreman host

# Make sure we have some fresh ssh keys available
ssh-keygen

# Now we run it!
screen  
ansible-playbook -i hosts site.yml  

The install process can take up to 30 minutes+ depending on your internet speed and host performance.

Once it's done, we still need to do a few post-install clean ups:

Because I have a public (eth1) and private (eth0) network this causes some complications with the install script. You'll simply need to modify /etc/openshift/node.conf

Set The following:

  • external ip address (the public ip address)
  • external hostname (remember those external hostnames we created eg. oo-broker01-melb.example.net)

Configure FreeIPA Dynamic DNS Updates

Now we need to configure the FreeIPA to accept the dynamic DNS updates it'll be receiving from the broker hosts.

Head over to the DNS Section -> Domain Name -> Settings

Append the following to the section for dynamic updates:

grant DNS\[email protected] wildcard * ANY; grant DNS\[email protected] wildcard * ANY;  

This can only be done through the web-ui as through command line it will only replace rather than append.

Finally tick the checkbox "Allow Dynamic Updates" and save changes.

Verification

  • OpenShift Origin comes with some nice tools to check everything is working.
    • On your Nodes run: oo-accept-node -v
      • This will help you debug any issues normally related to mcollective not running or missing packages.
    • On your Brokers run: oo-accept-broker -v
      • As above.
  • Everynow and then don't forget to make sure your ActiveMQ and MongoDB replicated cluster is still chugging along and that disk space isn't filling up too fast.

Closing Notes

Now enjoy your new highly available OpenShift Origin deployment and lookout for the cool pep's on OpenShift docker support and truely highly available hosted applications.

I'm open to comments and feedback and pull requests. I know my script is very dirty and hacky, but was my first attempt at ansible.

Because of the nature of the current openshift design, data is still limited to that one node (Unless they're scaled). I'd love to see some sort of gluster replicated style deployment so each node becomes a gluster-server but until then make sure you are always backing up those data directories ( /var/lib/openshift )

References:

comments powered by Disqus