“The Foreman” – A complete lifecycle management tool

Introduction

This is the start of a blog series through which I would like to introduce “The Foreman” . A complete lifecycle management tool for physical and virtual servers. For the first post I would like to explain what “The Foreman” is made for, how “The Foreman” works and how it can help to achieve a goal like the automatic build of a development environment. Finally i would like to show why the automation of infrastructure management is essential nowadays. In future blog posts I will demonstrate how to install “The Foreman” through puppet and how to use its API to configure “The Foreman” automatically.

Let’s start with the meaning of “complete lifecycle management tool”. A complete lifecycle of a machine consists of the following steps:

• Installation – The initial installation of an operation system.
• Configuration – The installation and configuration of further software packages as well as the configuration of e.g. users and groups or network interfaces.
• Update, Management and Audits – The installation of patches and/or the change of our server configuration and finally the monitoring over the entire lifespan.

In the past, the lifecycle of a server was ‘managed’ for example by filling wiki-pages with installation and configuration scripts or just by mouth-to-mouth recommendation on how to install and configure certain software. However, the wiki-pages were often out-of date and mouth-to-mouth recommendations are error-prone – this lead to installation and configuration processes that took too long and were too error prone. Most of the time the process itself was neither reusable nor fault tolerant and very often the availability of the managed server was far from 100%.

As early as the 90s configuration management (CM) tools like CFEngine started to emerge to tackle this challenge, with tools like Chef and Puppet appearing in the 2000s. For a closer look you can check out the following blog post on Puppet .

These CM tools are concentrating on the installation and configuration steps after the operating system has been installed. This means, we are now able to install and configure our software packages, users and network interfaces automatically. Even more important we are now able to describe our infrastructure as code. Which means, we can manage it similar to source code. To mention just a few advantages: we are now able to use version control, to test and reuse our infrastructure code. This is a first step – but still not the final automatisation of the three steps mentioned above. We are still missing the initial installation of an operation system on bare-metal.

Provisioning Setup

This is where “The Foreman” has its place. “The Foreman” helps us exactly at the point of automatically installing a OS. After that – through a very good integration with puppet – the new system will be configured to our specification. Finally Puppet will send facts about the system to “The Foreman” which helps us to monitor the whole system over its complete life span. With a discovery plugin “The Foreman” can also discover new machines in your network based on their mac address.

How does this work? The first step is the configuration of “The Foreman” and its components – in particular the configuration of:

1. Provisioning Setup – Infrastructure

• Smart-Proxies for different tasks (TFTP, DNS, DHCP, Puppet & Puppet CA)
• A Domain which references to a Smart-Proxy for DNS
• A Subnet which references to Smart-Proxies for DHCP, TFTP, DNS

1. Provisioning Setup – Puppet

• An environment for Puppet classes

1. Provisioning Setup – Host

• OS-Images and a corresponding mirror
• Provisioning templates for unattended installations (preseed, kickstart …)
• Templates for partition tables
• Connections between OS-images and templates for provisioning and partition tables

This configuration can be done either through a web user interface, through an API based on RESTful web services or through a command line interface called Hammer.

The mentioned Smart-Proxies are APIs based on RESTful web services which “The Foreman” builds on or close to e.g. existing DHCP, DNS and TFTP servers to help orchestrating the process to provision systems (see figure 1).

Figure 1: “The Foreman” architecture

Provisioning of a new host

After we have configured “The Foreman” to provision new hosts we need to set up a new host (determine its OS image, the services it will run etc) inside “The Foreman” (see video at the end of the post). After we finished this task, we are now able to boot the new system. What happens now are the following steps (see also figure2):

1. Start of the new system – Preboot Execution Environment (PXE)

• The host boots with the PXE-protocol.
• The host sends a broadcasts to search a DHCP-server that can handle PXE requests.
• The DHCP-server (Smart-Proxy) answers and gives an ip-address to the client.
• The PXE-Server will be contacted, it knows the route to the TFTP-Server.
• The TFTP-Server holds a boot image for the host.

1. OS Installation

• The host starts the installation through the boot image. This installation runs unattended using the the provisioning templates connected to the boot image

1. Configuration

• Puppet will be run and configure the system and services specified for it.
• Reports and facts will be gathered and finally be sent to “The Foreman”.

Figure 2: “The Foreman” boot sequence

The provisioning can take place on bare-metal as well as on other compute resources like OpenStack, EC2 or GCE as shown in figure 1.
As mentioned before, Puppet collects many facts about a host like memory size, timezone, cpu type and many more. These facts can then be used inside “The Foreman” to determine the parameters with which to install software. For example this might help Puppet and “The Foreman” for example to decide whether apache might run on a certain system on port 80 or 8080. Foreman acts as an External Node Classifier (ENC) for Puppet.

Conclusion

I hope you can now imagine how “The Foreman” can be a single source of truth for configuration information and help you create and provision systems faster so that – for example when a new team member joins your team you just press a button and provision his new machine automatically. And there is a lot more where “The Foreman” can help you. Just think of your continuous deployment pipeline and its testing environments. With foreman you could set up a new testing environments with every single deployment run – helping you test different characteristics of features very easily and keeping track of the evolution of the environment together with that of the code.

This article could just scratch the surface and a lot had to be left out, but I hope I could create at least some interest. So if you are keen on learning more about “The Foreman”, with the next blog post I will show how to install “The Foreman” automatically through puppet.

Since a video can show more than thousand words. Check out following official video .