Testing Crossplane Compositions with kuttl, Part 1: Preparing the TestSuite

Does writing Kubernetes Manifests count as writing code? Should we still bother to test it? Sure! And with the Kubernetes Test Tool (kuttl) there's great tooling available. Let's explore how to use it with Crossplane.

Crossplane – blog series

1. Tame the multi-cloud beast with Crossplane: Let’s start with AWS S3
2. Testing Crossplane Compositions with kuttl, Part 1: Preparing the TestSuite
3. Testing Crossplane Compositions with kuttl, Part 2: Given, When, Assert
4. Create, build & publish Crossplane Configuration Packages with GitHub Actions & Container Registry

Should you test Crossplane Compositions?

If you're new to Crossplane, you might want to learn about all the fancy concepts such as Composite Resource Definitions (XRDs), Compositions and how to use them. And that's great! Crossplane provides a fantastic tooling to drive your Platform Engineering initiative. I wrote about how to get started here. Digging deeper, you learn about more advanced topics such as how to write your own Compositions and how to package them as Configurations. But isn't there something missing?

Lately I gave a Crossplane training at a customer and was surprised to hear that testing infrastructure code is often limited to linting. But regardless of the tooling: If we do Infrastructure-as-Code, we shouldn't forget the principles of modern software engineering, right? And one of these is:

code that is not automatically and constantly executed and tested will eventually rot sooner or later!

This also applies to Kubernetes manifests. Especially since the world around our manifests doesn't stand still. For example, there are constantly new Crossplane releases or Crossplane Provider releases coming. How do you make sure your Crossplane Composition is still compatible with these and provisions the same infrastructure as with the previous releases?

I wrote about the topic of testing infrastructure code with Molecule back in 2018. The meme I used in my post back then just applies today:

Meme source: imgflip.com

So the question is: how can we test our Crossplane Compositions (and Composite Resource Definitions)?

Why Kubernetes Test Tool (kuttl) for Crossplane testing?

You might ask why this post isn't based on uptest, which was donated to the CNCF a while ago. That's because uptest seems to be a work in progress. See this issue comment from an Upbound employee (the company behind Crossplane):

"I think we have to be honest and document somewhere that currently uptest is not really usable without surrounding make targets and the build module :)"

I tried to set up uptest in a Crossplane Configuration repository and was stuck in the middle of nowhere. Documentation is nearly non-existent and I finally resigned – especially when I stumbled upon the mentioned GitHub issue.

But luckily uptest itself is based on the Kubernetes Test Tool (kuttl) and generates a kuttl test case under the hood. You're even encouraged to inspect the generated kuttl tests in the troubleshooting section of the uptest documentation. So why shouldn't we use native kuttl for our Crossplane tests directly?!

Getting started with kuttl

So what is the Kubernetes Test Tool? The kuttl docs have a great explanation for us:

"KUTTL is a declarative integration testing harness for testing operators, KUDO, Helm charts, and any other Kubernetes applications or controllers. Test cases are written as plain Kubernetes resources and can be run against a mocked control plane, locally in kind, or any other Kubernetes cluster."

In other words: kuttl is a test harness for any Kubernetes application. That also makes it a great fit for Crossplane!

Logo sources: Crossplane logo, kuttl logo

So before talking too long, let's dive into the first steps with kuttl. As kuttl uses the Kubernetes in Docker (kind) tooling to run local Kubernetes clusters, we need to have a local Docker installation in place. Also, be sure to have the following command line tools installed: kubectl, helm & kind. As we will be using AWS as our infrastructure resource provider later, there should also be a working installation of the awscli. You can testdrive your awscli configuration with the command aws configure.

Installing kuttl kubectl plugin

Having Docker and the mentioned CLI tools in place, we need to install the kuttl kubectl plugin. On a Mac (or Linux with brew installed) you can simply use the homebrew package:

An alternative way is to leverage the kubectl plugins package manager krew to install the kuttl kubectl plugin. Therefore have a look into the krew installation docs. Installing krew might be a bit more work compared to the homebrew variant, but it also works for other kubectl plugins. If you have krew in place, install kuttl via the command kubectl krew install kuttl:

Add the export PATH... statement to your shell configuration as mentioned in the installation log output.

Now the kubectl kuttl plugin should work as expected:

How to use kuttl with Crossplane

kuttl defines the following building blocks:

• A "test suite" (aka TestSuite) is comprised of many test cases that are run in parallel.
• A "test case" is a collection of test steps that are run serially – if any test step fails then the entire test case is considered failed.
• A "test step" (aka TestStep) defines a set of Kubernetes manifests to apply and a state to assert on (wait for or expect).

In order to use Crossplane with kuttl, we need to install Crossplane and a Crossplane Provider (like AWS) in the kuttl TestSuite. We also need to configure the Crossplane Provider with a ProviderConfig and its matching credentials via a Secret. It depends on the kind of testing we want to do if the Secret needs to hold only fake (Rendering-only tests) or real (Integration Tests) credentials:

Logo sources: Crossplane logo, kuttl logo, kind logo, Docker logo, AWS logo, Azure logo, Google Cloud logo

With that configured in our kuttl TestSuite, we can create a kuttl test case, containing some kuttl TestSteps. These will be shown in the next part of this blog post series.

Now before we can start writing our first kuttl tests, we need to have an example Crossplane Composition in place. As I needed it for a meetup talk and an magazine article, I created a simple Composition to provision a public accessible S3 Bucket. The example code used throughout this blog post is fully available on GitHub.

An example Composition

The example project provides a Crossplane Configuration that will make use of the Upbound Provider for AWS S3 and create a public accessible S3 Bucket in AWS. The Composite Resource Definition defines a Composite Resource Claim called ObjectStorage with the two parameters bucketName and region. It can be found in the file apis/objectstorage/definition.yaml in the example repository:

The Composition is defined in the file apis/objectstorage/composition.yaml. Since it became way more complex to setup a S3 Bucket in AWS that is publicly accessible somewhere in April 2023 , the Composition needs to use a set of Managed Resources. According to this issue and these Terraform docs we need to define the Bucket creation along with a definition of a BucketPublicAccessBlock, BucketOwnershipControls, the BucketACL and a BucketWebsiteConfiguration.

If you have your own Composition in place, you can start from there. Or you just take the example Composition provided here. Anyway, we can now start writing our first Crossplane powered kuttl test suite.

Creating a kuttl TestSuite: The kuttl-test.yaml

For this, we need to create a kuttl-test.yaml defining our TestSuite in the root of our project :

This file is the starting point of the kuttl configuration. Right now it mainly defines two things: The directory where our tests will be stored and if kuttl should fire up a kind cluster for us. Since we defined our testDirs to be tests/e2e/, we now also need to create a new directory tests containing another directory e2e in the root of our project:

We should also add the following lines to our .gitignore to prevent us from checking in temporary kind logs or kubeconfig files to Git:

Installing Crossplane in kuttl TestSuite

To be able to write tests for Crossplane, we need to have it installed in our cluster first. Luckily kuttl has a commands keyword ready for us in the TestSuite and TestStep objects. Starting the command with a binary, we can execute anything we'd like.

Since we need Crossplane installed and ready for all our tests, we will install it in the TestSuite instead of every TestStep. Therefore, inside our kuttl-test.yaml we add command statements to install Crossplane into the kind test cluster:

The installation of Crossplane works "Renovate" enabled via a local Helm Chart. You might recall the intro to this post, where we stated that our Compositions should be tested automatically when new Crossplane versions arrive. Here's how to make sure the update will be triggered by Renovate automatically. For this purpose we define a Chart.yaml in a new directory crossplane/install:

Additionally, we add the following to our .gitignore to prevent us from checking in generated Helm files:

That's it. If we configure Renovate in the repository later, our Composition will be tested for new Crossplane versions automatically :)

Already at this point, we can use the kubectl kuttl test command to verify if our Crossplane installation works as expected:

The --skip-cluster-delete flag comes in handy, since it will preserve our crossplane-test kind cluster for later runs (without the flag it would be deleted everytime). You may double-check if the kind cluster still runs after the kubectl kuttl test command. A docker ps should show the cluster also:

You can even connect to the kind cluster directly setting the KUBECONFIG env variable like this (simply replace your profile and repository path):

With this KUBECONFIG you can access the kuttle created kind cluster in your current shell:

To get your normal kubectx config working again, simply run unset KUBECONFIG.

Since kuttl doesn't remove its kind cluster when we use the --skip-cluster-delete flag, we also need to know how to delete the kind cluster ourselves:

Installing AWS Provider in kuttl TestSuite

Now that we successfully installed Crossplane into our kuttl kind cluster, we also need to install a Provider. As our Composition is based on AWS and provisions a publicly accessible S3 Bucket, we need to use the Provider upbound/provider-aws-s3. To install it we should first create a new directory provider inside our already existant crossplane folder:

Inside of crossplane/provider we create our Provider specification in the file upbound-provider-aws-s3.yaml:

Having the Provider spec in place, we can integrate it into our kuttl TestSuite. Thus inside our kuttl-test.yaml we add additional command statements to install the Provider into the kind test cluster and wait for it to become healthy:

As you can see we also wait until the Provider reached a healthy state, before we proceed. That's crucial for the next steps to work.

Configuring AWS Provider in kuttl for testing Resource rendering (without AWS access)

Often we do not need to really create resources on AWS throughout our tests. It might be enough to just verify if the Managed Resources are rendered correctly. Just as we intended while writing our Composition. Or we can even start with a test-driven approach and start with our kuttl test right away.

To get the Crossplane AWS Provider to render the Managed Resources without actual AWS connectivity, we use the trick described here and create a Secret without actual AWS creds. Let's therefore create a file non-access-secret.yaml in the crossplane/provider/ directory:

Now inside our kuttl-test.yaml we add additional command statements to create the Secret and configure the AWS Provider without actual AWS access:

As you can see, we added another line. In it, we configure the AWS Provider via a ProviderConfig. For this to work, we need to create a file provider-config-aws.yaml inside the crossplane/provider directory:

Now we should be able to successfully run kubectl kuttl test --skip-cluster-delete:

If your output looks similar to this, everything should be prepared for testing resource rendering with Crossplane!

Next up: kuttl TestSteps with Crossplane

In this post, we learned about kuttl and that it can be a great match for Crossplane. We fully set up kuttl to have Crossplane installed and a Crossplane Provider configured. With this in place, we are now able to render our Composition with AWS-based Managed Resources. There's also a full example Composition ready that we will be able to test with kuttl.

So the next part of this blog series will go into the details of how to create test steps with kuttl. The next post explores how to structure the kuttl test steps into a scheme lend from the Behavior-driven Development (BDD). In the end, everybody in the project should be able to read the tests structure and log output. The next post also outlines how we can create Integration Tests with kuttl and provision real infrastructure in them using Crossplane. Finally everything will be packed into a GitHub Actions pipeline.