Provisioning IaaS Clouds with Dynamic Ansible Inventories and OpenStack Metadata

My colleague Daniel Schneller gave an introduction to Ansible . A key concept of Ansible is the inventory. It contains all hosts of your site that you want to provision with Ansible. For bare metal hardware, this inventory is a static file enumerating all your nodes. But what do you do when you want to provision Virtual Machines? They tend to come up and go down, multiply and reduce in numbers. Maintaining a static file is cumbersome in this situation. In addition to the mere enumeration, you need to specify the role each virtual node should play. In this article, I present how to build dynamic inventories by querying OpenStack for available Virtual Machines and using metadata to specify the roles for each of them in a dynamic fashion. A fully working example is available on GitHub to get you started with your own dynamic inventories.

Inventories in a Static World

An Ansible inventory consists of host groups enumerating hosts which belong to that group as well as specifying host based variables. These groups are used by Ansible playbooks to associate roles with the respective hosts. A specific host may be part of several groups in order to indicate that it takes several roles. Listing 1 shows a simple inventory where the two nodes node01 and node02 belong to two different host groups and have host based variables associated.

1[openstack_compute_nodes]  
2node01  
3node02  
4 
5[dns_clients]  
6node01 DNS_SERVER=192.168.0.10  
7node02 DNS_SERVER=192.168.0.10

Inventories in a Dynamic World

The inventory in listing 1 suits a static environment where changes occur only rarely. But in an highly dynamic virtual environment where VMs come and go, keeping a static inventories is infeasible. For example, imagine that you run a central DNS server for all virtual machines. So every time a new virtual machine is started, Ansible should provision this DNS server as part of the new VM’s setup. Further, the services the VM should provide need to be provisioned, too. This means we need to first discover the machine, second gather which roles it shall play, and third, provision these roles to the machine.

In the next three paragraphs, we show you how to use OpenStack Metadata, some custom Python, and Ansible’s dynamic inventory feature to solve this challenge in a fully automatic fashion.

Metadata for OpenStack Virtual Machines

In OpenStack, a virtual machine may be annotated with arbitrary metadata. Figure 1 shows a screenshot from OpenStack’s Dashboard presenting metadata of a running instance.

Figure 1

This metadata can be added to an instance via command line while booting the instance. You can find a complete example on GitHub .

1nova boot --flavor m1.medium … --meta \
2 ansible_host_vars=dns_server_for_domains->v.clusterb.centerdevice.local,\
3 infrastructure.v.clusterb.centerdevice.local \
4 --meta ansible_host_groups=admin_v_infrastructure

It is also possible to add metadata to an already running instance with the comannd nova meta set .

In general, you can set the metadata of an instance to arbitrary strings. In this examples, I use a key value pair like convention to specify two pieces of metadata information, i.e., ansible_host_vars and ansible_host_groups which are used by the dynamic inventory to fill the corresponding Ansible inventory values.

Dynamic Inventories for Ansible

An inventory is passed to the command ansible-playbook via the parameter -i. This inventory may be a static file or an executable. In case of the latter, Ansible requires the executable to return a JSON document. The corresponding JSON document for the metadata specified above looks like this:

1➜  ./openstack_inventory.py
2{
3    "admin_v_infrastructure": {
4        "hosts": [
5            "192.168.0.10"
6        ]
7    },
8    "_meta": {
9        "hostvars": {
10            "192.168.0.10": {
11                "dns_server_for_domains": [
12                    "v.clusterb.centerdevice.local",
13                    "infrastructure.v.clusterb.centerdevice.local"
14                ]
15            }
16        }
17    }
18}

You can see the general structure that is required. There are two sub documents admin_v_infrastructure and _meta. The first represents a host group of the same name and the second collects host variables for each host. For details about the JSON structure see the Ansible documentation.

In this way, we can assign a role for a DNS server to the hosts of host group admin_v_infrastructure while the host variable dns_server_for_domains specify for which domains the DNS server is responsible.

Querying OpenStack Metadata with Python

OpenStack is developed using Python, so all required Python modules needed to query OpenStack for instance metadata are already installed. The main function of the script looks like this:

1def main(args):  
2  credentials = getOsCredentialsFromEnvironment()  
3  nt = client.Client(credentials['USERNAME'], 
4    credentials['PASSWORD'], 
5    credentials['TENANT_NAME'], 
6    credentials['AUTH_URL'], 
7    service_type="compute")  
8 
9  inventory = {}  
10  inventory['_meta'] = { 'hostvars': {} }  
11 
12  for server in nt.servers.list():  
13      floatingIp = getFloatingIpFromServerForNetwork(server, 
14        OS_NETWORK_NAME)  
15      if floatingIp:  
16          for group in getAnsibleHostGroupsFromServer(nt, server.id):
17              addServerToHostGroup(group, floatingIp, inventory)  
18          host_vars = getAnsibleHostVarsFromServer(nt, server.id)  
19          if host_vars:
20              addServerHostVarsToHostVars(host_vars, floatingIp, inventory)  
21 
22  dumpInventoryAsJson(inventory)

The script assumes that the environment variables OS_AUTH_URL, OS_USERNAME, OS_PASSWORD, and OS_TENANT_NAME are set in the same way as for all OpenStack command line utilities. In this example, we only provision VMs with a floating IP, because only those are accessible from outside their project. For all instances with a floating IP, the metadata is checked for the variables introduced above and if found added to the dynamic inventory. This inventory is dumped to stdout in JSON format. See the GitHub repository for details.

Using Multiple Inventories for Ansible

Ansible does not restrict you to use either a dynamic or static inventory. It is quite easily possible to combine multiple inventories, static as well as dynamic, by placing the files into a directory and passing the directory via the parameter -i. In this case, Ansible combines all inventories from files and executable results into one inventory.

Please pay attention when using a dictionary based inventory especially when using a dynamic inventory. Since all discovered hosts are added to the all host group, playbooks referring to this group might be unintentionally applied to hosts you have not been thinking of when writing the playbook.

There is more

Ansible is a highly flexible but still simple to use provisioning tool. In our document management cloud CenterDevice Ansible is replacing Puppet as our tool of choice. The ability to mix static and dynamic inventories to manage host groups and automatically provision new hosts by simply checking their metadata prove Ansible’s flexibility.