Create, build & publish Crossplane Configuration Packages with GitHub Actions & Container Registry

You already created your first Crossplane Compositions? Pretty nice! But how to store them in Git? How to create and build a Configuration Package from it? And finally: how to publish and consume these Configurations in your Crossplane management cluster?

Crossplane – blog series

1. Tame the multi-cloud beast with Crossplane: Let’s start with AWS S3
2. Testing Crossplane Compositions with kuttl, Part 1: Preparing the TestSuite
3. Testing Crossplane Compositions with kuttl, Part 2: Given, When, Assert
4. Create, build & publish Crossplane Configuration Packages with GitHub Actions & Container Registry

Getting started with Configuration Packages

If you started creating Crossplane Composite Resource Definitions (XRDs), Compositions and Functions, you might not think about a suitable repository structure in the first place. Going the initial steps with Crossplane, one often just uses kubectl -f apply and reviews if stuff works as expected. But sooner or later you wonder: where to put these Crossplane manifests? What would be a suitable repository structure? Does this structure make for a good development process? And how to install those Crossplane manifests into the management cluster?

Luckily the Crossplane team answered all these questions already. The answer is simple: Use Configuration Packages! But what are these in Crossplane? Let's quote the docs:

A Configuration package is an OCI container image containing a collection of Compositions, Composite Resource Definitions and any required Providers or Functions. Configuration packages make your Crossplane configuration fully portable.

That's a great design choice I guess! Crossplane simply uses OCI container images for manifest distribution. So in this article we will have a look at how to create Configuration Packages and build, publish, and consume them. As we'll need some sort of Git repository, CI system and OCI registry to see everything working together in action, I opted for GitHub, GitHub Actions and the GitHub Container Registry.

So let's get our hands dirty!

Installing Crossplane CLI

In order to be able to build and publish Crossplane Configuration Packages, we need the crossplane CLI ready on our system. Therefore we can use the install script as stated in the docs:

1curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh" |sh

If that produces an error like Failed to download Crossplane CLI, please make sure version current exists on channel stable., we can try to manually craft the download link:

1curl --output crank "https://releases.crossplane.io/stable/current/bin/linux_amd64/crank"
2chmod +x crank
3sudo mv crank /usr/local/bin/crossplane

Don't get confused by the fact that the binary we will need on our system is named crank! There's also a crossplane binary there, but it is used as the Crossplane pod image. But on our machine, we rename crank to crossplane nontheless.

Our Crossplane CLI should work now:

1crossplane --version
2v1.15.0

Creating a new (GitHub) repository

As a recommendation, each Composition (or multiple nested Compositions) should be developed in their own Git Repository.

With this approach Compositions can be developed and tested in isolation. And they will be distributable and usable via a Crossplane Configuration Package (OCI image) from a Container Registry later.

Therefore let's create a new GitHub repository for our Composition. Head over to your GitHub account and create a new repository. Name it according to your Composition. As we will be creating a simple AWS S3 based Composition here, I named the repository crossplane-objectstorage:

Then clone the repository locally via or start a Codespace:

1git clone git@github.com:yourGitHubUserNameHere/crossplane-objectstorage.git

This post is also accompanied by a GitHub repository with every piece of code used thoughout this article.

A suitable repository structure to place the XRD and Composition

Inside the repository let's create a suitable directory for our XRD and Composition first:

1mkdir -p apis/objectstorage

Since we want to package an XRD and a Composition, we need to have both in place before proceeding. Because we focus on creating Configuration Packages here, I simply took the XRD and Composition already posted in my article about Crossplane Testing with kuttl. The XRD defines a simple Composite Resource with two parameters (see the example definition.yaml on GitHub). It should be placed inside the apis/objectstorage directory as definition.yaml.

The Composition uses multiple Managed Resources from the Upbound Provider for AWS S3 to provision a public accessible S3 Bucket (see the example composition.yaml on GitHub). It resides in the apis/objectstorage directory as composition.yaml. The naming of both definition.yaml and composition.yaml can be seen as a default when writing Crossplane configurations.

Often we want to test-drive our Composition and need to create an example Composite Resource (XR) or Composite Resource Claim ("Claim"). A examples directory is the perfect place for the XR or Claim. Thus create the directory mirroring the apis folder structure:

1mkdir -p examples/objectstorage

The claim.yaml to be placed inside this directory can look like this here on GitHub. Now our repository structure should look like this:

1.
2├── apis
3│   └── objectstorage
4│       ├── composition.yaml
5│       └── definition.yaml
6├── examples
7│   └── objectstorage
8│       └── claim.yaml

Creating the crossplane.yaml

As stated in the docs

A Configuration package requires a crossplane.yaml file and may include Composition and CompositeResourceDefinition files.

Thus we need to create a crossplane.yaml file in the root of our repository. The crossplane.yaml that is shown in the Crossplane docs won't be buildable and will create an error while running the crossplane xpkg build command later. That's because some metadata fields are missing. But here's a fully working example crossplane.yaml:

1apiVersion: meta.pkg.crossplane.io/v1alpha1
2kind: Configuration
3metadata:
4  name: crossplane-objectstorage
5  annotations:
6    # Set the annotations defining the maintainer, source, license, and description of your Configuration
7    meta.crossplane.io/maintainer: Jonas Hecht iam@jonashackt.io
8    meta.crossplane.io/source: github.com/jonashackt/objectstorage
9    # Set the license of your Configuration
10    meta.crossplane.io/license: MIT
11    meta.crossplane.io/description: |
12      Crossplane Configuration delivering CRDs to provision publicly accessible S3 buckets.
13    meta.crossplane.io/readme: |
14      Featuring a Composition with multiple MRs (Bucket + BucketPublicAccessBlock, BucketOwnershipControls, the BucketACL and a BucketWebsiteConfiguration)
15spec:
16  dependsOn:
17    - provider: xpkg.upbound.io/upbound/provider-aws-s3
18      version: ">=v1.4.0"
19  crossplane:
20    version: ">=v1.15.1-0"

As you can see, there are multiple metadate.annotations fields added. In these, useful information about the Crossplane Configuration can be placed. Defining them will also prevent the error crossplane: error: failed to build package: not exactly one package meta type later (see this stackoverflow answer).

Also, we need to define on which providers our Configuration depends on. The dependsOn field is defined as an array and thus supports multiple Crossplane providers. Imagine crafting a Composition to provision an AWS EKS cluster. Then you would need the EC2, EKS and IAM providers for example. Lastly, the minimum Crossplane version our Configurations needs to work is defined.

If you want to generate the crossplane.yaml: there's a template one could use to create it using the Crossplane CLI also. The command crossplane beta xpkg init will do the job:

1crossplane beta xpkg init crossplane-objectstorage configuration-template

The command uses a specific template called configuration-template. It's simply a Git repository and one can provide arbitrary repositories (aka templates) to the command.

The templating command will also create a apis/definition.yaml and apis/composition.yaml. You should delete them before proceeding.

Building the Configuration Package using Crossplane CLI

Now it's time to build the Configuration Package! Since Configuration Packages are fully OCI-compliant, any tool that builds OCI images can build Configuration packages. But it is strongly recommended in the docs to use the Crossplane CLI to have error checking available. For reference the xpgk specification is available on GitHub.

To build the Configuration Package from our XRD and Composition, we need to run the crossplane xpkg build command:

1crossplane xpkg build --package-root=. --examples-root="./examples" --ignore=".github/workflows/*,crossplane/provider/*,kuttl-test.yaml,tests/compositions/objectstorage/*" --verbose

The command should produce a file like crossplane-objectstorage-85ab2ae64465.xpkg. That is our OCI image ready to be published!

But at first, you will be likely exploring errors like crossplane: error: failed to build package: failed to parse package:. The issue is that including YAML files in the build that aren’t Compositions or CompositeResourceDefinitions isn’t supported. This applies to XRs/Claims also, since they are not considered part of a Configuration Package. Therefore we use the --examples-root parameter here to exclude our Claim residing in the examples directory.

Yes, you already guessed it: We really need to ignore every file that is not an XRD or Composition in our repository in order to be able to build our Crossplane Configuration Package OCI image!

So let's also ignore any other files from the build command. This can be done by appending --ignore="file.xyz,directory/*" using a comma-separated list, which will ignore a file file.xyz and all files in directory directory. Sadly, ingoring directories completely isn't supported right now. So as we will create a GitHub Actions workflow later, we need to exclude it. Also, if you use a tool like kuttl to test your Compositions, you need to exclude all its relating files also. Often this is the kuttl-test.yaml, the tests directory and the directory where the Crossplane Provider configurations reside.

Finally appending --verbose makes a lot of sense to see what's going on and to be able to debug in case of an error!

What's more, be sure to enhance your .gitignore file to prevent yourself from checking the *.xpkg files into Git:

1# Don't check in Configuration packages
2*.xpkg

Pushing the Package file .xpkg to GitHub Container Registry

Now that our OCI image has been build as .xpkg file, we can proceed to push it to the GitHub Container Registry.

Therefore the Crossplane CLI also features a crossplane xpkg push command to publish the Configuration package. The following command will create a new GitHub Container Registry package, that matches our repository github.com/yourGitHubUserName/crossplane-objectstorage:

1crossplane xpkg push ghcr.io/yourGitHubUserName/crossplane-objectstorage:v0.0.1

As you see, we can leverage the Container image tag as version number for our Configuration Package here.

If the command gives the following error, we need to set up authentication for our Docker Registry:

1crossplane: error: failed to push package file crossplane-objectstorage-7badc365c06a.xpkg: Post "https://ghcr.io/v2/jonashackt/crossplane-objectstorage/blobs/uploads/": GET https://ghcr.io/token?scope=repository%3Ajonashackt%2Fcrossplane-objectstorage%3Apull&scope=repository%3Ajonashackt%2Fcrossplane-objectstorage%3Apush%2Cpull&service=ghcr.io: DENIED: requested access to the resource is denied

But how can we setup the crossplane xpkg push command to be authenticated against the GitHub Container Registry? Running crossplane xpkg push --help comes to the rescue:

Credentials for the registry are automatically retrieved from xpkg login and dockers configuration as fallback.

So we need to login to GitHub Container Registry first in order to be able to push our OCI image! This can be achieved using the usual docker login command:

1echo YourGitHubPersonalAccessTokenHere | docker login ghcr.io -u YourAccountOrGHOrgaNameHere --password-stdin

Make sure to use a Personal Access Token as described in this post. To create one, head over to your GitHub user Settings/Developer Settings and to Personal Access Tokens. Click on Personal access tokens (classic) and create a classic token with the following scopes (repo, write:packages and delete:packages):

Additionally, we need to add the domain configuration to the crossplane xpkg push command like this: --domain=https://ghcr.io. Otherwise the default domain is upbound.io which will lead to non pushed Configurations (only visible via the verbose flag).

Now our crossplane xpkg push command should finally work as expected:

1$ crossplane xpkg push ghcr.io/jonashackt/crossplane-objectstorage:v0.0.1 --domain=https://ghcr.io --verbose
2
32024-03-21T16:39:48+01:00	DEBUG	Found package in directory	{"path": "crossplane-objectstorage-7badc365c06a.xpkg"}
42024-03-21T16:39:48+01:00	DEBUG	Getting credentials for server	{"serverURL": "ghcr.io"}
52024-03-21T16:39:48+01:00	DEBUG	No profile specified, using default profile
62024-03-21T16:39:49+01:00	DEBUG	Pushed package	{"path": "crossplane-objectstorage-7badc365c06a.xpkg", "ref": "ghcr.io/jonashackt/crossplane-objectstorage:v0.0.1"}

Now head over to your GitHub Organisation's Packages tab and search for the newly created package. Click onto the package and connect the GitHub Repository:

Also – on the right – click on Package settings and scroll down to the Danger Zone. There, click on Change visibility and change it to public. Now your Crossplane Configuration should be available for download without login.

If everything went fine, the package / OCI image should now be visible at your repository.

Building & publishing your Configuration Package automatically with GitHub Actions

Now that we did every step manually, we want to build and publish the Configuration Package every time the XRD and/or Compositions code changed, even when a new Crossplane version or Provider version has been released. The latter can be achieved by configuring Renovate, which will automatically be able to trigger new Configuration Package builds.

Since we want to make sure everything renders successfully, we should be sure to implement Unit and Integration Tests for our Composition and run them right before the Configuration Package build. This post about Testing Crossplane Compositions explains how to do that.

In order to automatically build and publish our Configuration Packages, we need to leverage a CI tooling. So let's finally do all the steps automatically with GitHub Actions. Any Composition code change (git commit/push) should trigger our pipeline. Therefore let's create a new GitHub Actions workflow at .github/workflows/test-composition-and-publish-to-ghcr.yml](.github/workflows/test-composition-and-publish-to-ghcr.yml):

1name: test-composition-and-publish-to-ghcr
2
3on: [push]
4
5env:
6  GHCR_PAT: ${{ secrets.GHCR_PAT }}
7  CONFIGURATION_VERSION: "v0.0.2"
8
9jobs:
10  composition-rendering-test:
11    runs-on: ubuntu-latest
12
13    steps:
14      - name: Run Composition rendering & Integration tests as described in https://www.codecentric.de/wissens-hub/blog/testing-crossplane-compositions-kuttl
15        run: echo "Run tests here!"
16
17  build-configuration-and-publish-to-ghcr:
18    needs: composition-rendering-test
19    runs-on: ubuntu-latest
20
21    steps:
22      - uses: actions/checkout@v4
23
24      - name: Login to GitHub Container Registry
25        uses: docker/login-action@v3
26        with:
27          registry: ghcr.io
28          username: ${{ github.actor }}
29          password: ${{ secrets.GHCR_PAT }}
30
31      - name: Install Crossplane CLI
32        run: |
33          curl -sL "https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh" |sh
34          sudo mv crossplane /usr/local/bin
35
36      - name: Build Crossplane Configuration package & publish it to GitHub Container Registry
37        run: |
38          echo "### Build Configuration .xpkg file"
39          crossplane xpkg build --package-root=. --examples-root="./examples" --ignore=".github/workflows/*" --verbose
40
41          echo "### Publish as OCI image to GHCR"
42          crossplane xpkg push "ghcr.io/jonashackt/crossplane-objectstorage:$CONFIGURATION_VERSION" --domain=https://ghcr.io --verbose

As we added the .github/workflows directory with a workflow yaml file, the crossplane xpkg build command tries to include it. Therefore we need to exclude the workflow file also from the command (as already shown above):

1crossplane xpkg build --package-root=. --examples-root="./examples" --ignore=".github/workflows/*" --verbose

Remember: Using only --ignore=".github/* won't work, since the command doesn't support to exclude directories - only wildcards IN directories.

We also use the Personal Access Token (PAT) we already created above in our GitHub Actions Workflow instead of the default GITHUB_TOKEN in order to have the correct permissions. This should prevent the following error:

1crossplane: error: failed to push package file crossplane-objectstorage-7badc365c06a.xpkg: PUT https://ghcr.io/v2/jonashackt/crossplane-objectstorage/manifests/v0.0.2: DENIED: installation not allowed to Write organization package

Therefore we need to create a new Repository Secret containing our PAT:

With this we're also able to use a ENV var for our Configuration version or even latest.

Now our GitHub Actions pipeline should run as expected and will publish a new Crossplane Configuration Package version every time our XRD or Composition code changed:

Installing the Configuration Package into the management cluster

We finally hit the last question we were asking in the first place: how can we install Crossplane Configuration Packages into our management cluster? That's pretty easy with the help of a Configuration manifest as described in the docs.

Let's assume here you already have a management cluster with Crossplane and the Upbound Provider for AWS S3 running. This can even be a local kind cluster, as described in this post.

It's also important to remind ourselves now: the manifests of our Crossplane management cluster reside in a DIFFERENT Git repository than the one we use to develop, build and publish our Configuration Package!

So inside our Crossplane management cluster's Git repository we for sure already have some Provider-specific directory paths such as upbound/provider-aws, upbound/provider-azure or crossplane-contrib/provider-alibaba. For a full list of Crossplane Providers have a look at the Upbound Marketplace. Now since our created Configuration Package is based on the Upbound Provider for AWS S3, we can create a directory apis inside upbound/provider-aws:

1mkdir -p upbound/provider-aws/apis

Managing Configurations in a Provider-specific directory makes sense, since most Configuration Packages will be dependent on a specific kind of Provider.

Now inside the upbound/provider-aws/apis directory let's create a file called crossplane-objectstorage.yaml, which represents the Configuration Package we want to install:

1apiVersion: pkg.crossplane.io/v1
2kind: Configuration
3metadata:
4  name: crossplane-objectstorage
5spec:
6  package: ghcr.io/jonashackt/crossplane-objectstorage:v0.0.1

Installing our published Configuration Package into our management cluster is now nothing more than running a kubectl apply -f like this:

1kubectl apply -f upbound/provider-aws/apis/crossplane-objectstorage.yaml

With that, we have everything in place to actually use our Composition! Simply create a Claim to use the newly installed Crossplane API and kubectl apply it to your management cluster:

1apiVersion: crossplane.jonashackt.io/v1alpha1
2kind: ObjectStorage
3metadata:
4  namespace: default
5  name: managed-upbound-s3
6spec:
7  parameters:
8    bucketName: crossplane-storage
9    region: eu-central-1

Configuration Packages introduce a great development process for Crossplane Compositions!

Using Configuration Packages to distribute and consume our Compositions makes a lot of sense. In this post we saw how to structure a Git repository to hold our XRDs and Compositions. We learned that separating Compositions through separate Git repositories makes a lot of sense. Since we're able to build and publish Configuration Packages from each of them, there's no need to leave our Compositions all in one pile.

We also saw comprehensively how to use the GitHub Container Registry for Configuration Package distribution. Additionally everything can be done automatically via a CI/CD tooling like GitHub Actions. Thus every code change can trigger a new Configuration Package version without us bothering about it. As mentioned, we only need to make sure our Compositions are consistently tested.

Finally, using Configuration Packages in our management cluster is pretty straightforward, leveraging a Configuration manifest. Have fun packaging your Crossplane Compositions!