Testing Crossplane Compositions with kuttl, Part 2: Given, When, Assert

In the first part of this blog series we learned about kuttl and why it's a great idea to write tests for your Crossplane Compositions. Now it's time to set up the kuttl test steps to finally verify our Composition renders correctly.

Crossplane – blog series

1. Tame the multi-cloud beast with Crossplane: Let’s start with AWS S3
2. Testing Crossplane Compositions with kuttl, Part 1: Preparing the TestSuite
3. Testing Crossplane Compositions with kuttl, Part 2: Given, When, Assert
4. Create, build & publish Crossplane Configuration Packages with GitHub Actions & Container Registry

BDD-style Crossplane testing

We already configured a kuttl TestSuite that has a Crossplane installation and Provider configuration ready to use. Also, there was an example Crossplane Composition provided that waits to be tested. Based on that, we can now create a kuttl test case, featuring the kuttl TestSteps. Just as we're used to from writing Unit Tests in any other language, we can leverage the naming scheme lend from the Behavior-driven Development (BDD): Given, When, Assert.

Logo sources: Crossplane logo, kuttl logo, kind logo, Docker logo

I know, in BDD the third step is commonly named "Then". But as this step is always named "Assert" per default in kuttl, we need to arrange ourselves with this slight change :)

Creating a kuttl test case

Before we can craft test steps in kuttl, we need to create a test case. Creating a kuttl test case is fairly simple since it's only defined by the next directory level inside of tests/e2e. So let's create a test case for our objectstorage composition:

Creating a kuttl test step: the "Given" installing XRD & Composition

Now we're where we wanted to be in the first place: writing our first Crossplane-enabled kuttl TestStep. As already stated above, we borrowed the structure of our tests from the BDD-style Given, When, Then syntax. We start with the Given test step, where we install our Composite Resource Definition (XRD) followed by our Composition under test.

To be able to create our Given test step, we need to have a look into how kuttl handles files inside a test case directory:

"In a test case's directory, each file that begins with the same index is considered a part of the same test step. All objects inside of a test step are operated on by the test harness simultaneously, so use separate test steps to order operations."

As kuttl executes every 00-* prefixed test step found in the folder before proceeding to the 01-* one, we can have the 00-given-install-xrd-composition working as our preparation step for the other steps to come. Therefore let's create a new file called 00-given-install-xrd-composition.yaml inside the tests/e2e/objectstorage directory:

As you can see, we install the XRD and Composition first. After that it's a good idea to wait for the XRD to become established before proceeding to the next step.

If the kuttl logs show errors like the path "apis/objectstorage/definition.yaml" does not exist, check the paths in your command statements:

First I also missed this, since in the TestSuite at kuttl-test.yaml everything worked relatively well from the root dir. BUT remember, we're inside tests/e2e/objectstorage now! So we need to go up 3 dirs like ../../../apis/objectstorage/definition.yaml to fetch the correct file.

Creating another kuttl test step: the "When" applying XR or Claim

Now that we have installed our XRD and Composition in the Given step, it's time to apply our XR or Claim (XRC) in a When step.

Therefore let's created a file 01-when-applying-claim.yaml inside the tests/e2e/objectstorage directory:

In this test step's configuration we apply a Composite Resource Claim that should reside in the examples dir. If we don't already have it in place, we should now also create an examples directory in the root of our project.

The examples directory is the Crossplane default to place XRs or Claims in a configuration repository.

Inside the newly created examples folder we also create a objectstorage directory to reflect the folder structure of our Composition and XRD.

Insideexamples/objectstorage we finally create an example Claim in the file claim.yaml. It could look like this, for example:

The third kuttl test step: Validate / Assert Resource rendering (without AWS access)

We finally hit our third kuttl test step: The "Then" or Assert step, where we verify our Crossplane resource is rendered correctly. With kuttl we need to adhere to a specific name scheme here:

It's crucial to use 01-assert as the name here, to get the assertion started after our Claim has been applied!

As kuttl always searches for assert in the file name of our validation test step, we sadly can't use the BDD term "Then" here directly. But that shouldn't prevent us from writing our first kuttl assertion! Let's create a file 01-assert.yaml inside the tests/e2e/objectstorage directory:

This test step will be considered completed once our Managed Resources rendered are matching the state that we have defined.

If the state is not reached by the time the assert's timeout has expired, then the test step and case will be considered failed by kuttl.

Be sure to define the exact metadata like in your Claim! Otherwise kuttl won't find it and will show an error like the following:

Now we're finally able to run our test suite with our already known kubectl kuttl test command:

The --skip-cluster-delete will preserve the kind cluster if our tests failed and thus speed up our development cycle. As already explained, kind and the Crossplane installation/configuration will otherwise take place in every test run. Since kuttl will create a local kubeconfig file, it can reuse the kind cluster automatically in subsequent runs. Therefore, be sure to define export KUBECONFIG="/home/jonashackt/dev/crossplane-kuttl/kubeconfig" once and append --start-kind=false in the following commands (a sole kubectl kuttl test would otherwise give KIND is already running, unable to start errors):

Yay! We just wrote our first complete kuttl TestSuite verifying that our Crossplane Composition renders its Managed Resources exactly as we intended them to! A great advantage of these rendering tests is that they run completely in isolation inside our CI system. And their execution time is relatively fast. Thus just like Unit Tests, they can (and should) be executed very often!

As a bonus with kuttl we're not limited to assert on Crossplane's Managed Resources render correctly. We can even assert on Kubernetes events! Since Crossplane utilizes many Kubernetes events, we can assert on any specific condition in our Crossplane setup. Pretty cool!

Integration Testing: Configuring AWS Provider in kuttl for testing actual infrastructure provisioning (with real AWS access)

Now that we have a full kuttl cycle running and validating our Crossplane resource rendering, there is another scenario we can use this exact tooling: making sure through Integration Testing that our Crossplane Compositions provision real infrastructure correctly:

Logo sources: Crossplane logo, kuttl logo, kind logo, Docker logo, AWS logo, Azure logo, Google Cloud logo

To get this scenario working, we only need to tweak some things a bit we already created.

First we need to create a Secret containing our AWS credentials that Crossplane can later use to access AWS. Therefore we simply create an aws-creds.conf file (remember to have the aws CLI correctly installed and configured) in the root of our project:

ATTENTION: Don't ever check aws-creds.conf into version control. The best is to add it to your .gitignore file right now.

Inside our kuttl-test.yaml we add another command statements to create the Secret and configure the AWS Provider inside our kuttl kind cluster:

You might wonder why we delete the Secret before we even created it. Why is that? Because we want to omit errors like error: failed to create secret secrets "aws-creds" already exists.

See https://stackoverflow.com/a/45881324/4964553 - The best approach using dry-run=client sadly doesn't work with kuttl producing a error: unknown shorthand flag: 'f' in -f error.

You may also have noticed that we configured a higher timeout for resources to become available via the timeout configuration of our TestSuite. This is because provisioning real infrastructure takes its time to be provisioned. And sadly, we can't configure the timeout directly in the 01-when-applying-claim.yaml TestStep. But without the setting we would otherwise run into errors like this one soon:

The final bit is to configure the AWS Provider via a ProviderConfig that actually makes use of the Secret featuring the AWS credentials. Therefore we need to change the provider-config-aws.yaml located in crossplane/provider:

Now our kuttl setup should be able to provision real infrastructure with Crossplane.

Validate / Assert for testing actual infrastructure provisioning (with real AWS access)

But before actually running our kuttl setup, we also need to enhance our 01-assert.yaml located in tests/e2e/objectstorage:

Using an explicit TestAssert definition here we're able to override the TestSuite's timeout again to enable a faster test cycle. Otherwise the assertion would also wait for 300 seconds as defined in the test suite above.

Additionally we use a collector to make sure a cleanup step is also run in case of an error. Without that collector, infrastructure provisioned in our kuttl test steps wouldn't be cleansed and thus create unnecessary costs.

Now run our test suite with a kubectl kuttl test command.

You may even watch your AWS console, where the bucket gets created:

Pretty neat! Our setup with kuttl and Crossplane is now also ready to do Integration Testing with real infrastructure! Just be careful though how many Integration Tests you create and want to maintain. Just as the TDD testing Pyramid states, there should be far more Unit Tests than Integration Tests.

Cleanup after assertion (with real AWS access)

You might already have thought of it: In case of a successful test run, the provisioned infrastructure would also be preserved! Therefore, in case of Integration Tests with real AWS access, we should also clean up all resources after the last assertion ran. Therefore let's create a 02-* step called 02-cleanup.yaml in the tests/e2e/objectstorage dir:

This cleanup step should make sure that our infrastructure gets deleted in the end when everything went fine. If not, we have configured our collector inside the TestAssert config above.

Running Crossplane-featured kuttl tests in GitHub Actions

As mentioned at the beginning of this post, the holy grail is to have your tests being run automatically. Either scheduled or whenever a new Crossplane version or Provider version is released (triggered by Renovate for example). In order to achieve this, we can leverage a CI tooling like GitHub Actions:

Logo sources: Crossplane logo, kuttl logo, kind logo, Docker logo, AWS logo, Azure logo, Google Cloud logo, GitHub Actions logo, Renovate logo

Thus, the following YAML shows a full GitHub Actions workflow executing the Crossplane featured kuttl tests (see the example project workflow in .github/workflows/kuttl-crossplane-aws.yml):

Be sure to have the repository secrets AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in place in case you want to do Integration Testing with kuttl and GitHub Actions! As the example repository on GitHub has Renovate configured, you can see it automerging Crossplane and Crossplane Provider updates after successful kuttl test runs:

Testing Crossplane Compositions with kuttl is a great combination!

In this blog series we saw how to leverage kuttl to test our Crossplane Compositions. We installed and configured Crossplane in kuttl and provided two options for testing: Starting with the Crossplane resource rendering tests, we have a sort of Unit Testing harness available for our Crossplane Configurations! These can be run fast and in isolation in our CI system.

The second option is to use kuttl for Integration Testing Crossplane Compositions. This is also a great option if we really want to make sure our provisioned infrastructure looks exactly the way we intended it to.

The overall structure of our kuttl tests, borrowed from the Behavior-driven Development scheme, will make sure that our tests stay readable and maintainable – which is overall one of the biggest success factors while using tests in our project. Even Test-driven Development of Crossplane components becomes possible with kuttl! And leveraging CI systems such as GitHub Actions and Renovate will automatically make sure our Crossplane Compositions run with future versions of Crossplane and its Providers.

I would be greatly interested to read about your experiences with Crossplane testing in the comments!