GitHub Actions test a full Tekton CI installation

In my previous blog posts I gave an intro to Tekton and showed how to configure and integrate it. This post will show you a way how to build CI pipelines with built-in testing.
In software development, it is good practice to write tests for the code base. We see the value in our daily work in many places when it comes to further developing the software.
But CI systems, although they make the safety net possible for whole teams in the first place, are often far from the principle. Build and test configurations are all too often developed and used without a net and a double bottom.
This is where I would like to start and take the proven principle even further. The article describes the setup of a tested CI system using the example of Tekton with GitHub Actions.

GitHub Actions

GitHub Actions offer the possibility of building CI pipelines with little effort and use them for projects. My colleague Jonas wrote a wonderful blog post on this topic , which I gladly refer to for more information.
For this article it is important to know that the Actions also allow containerized execution of test code. Moreover, it is even possible to launch additional containers, which allows us to use a minimal Kubernetes cluster in the form of K3d within the test run.
In the demo project used, each push runs through the pipeline and the results of the run are visible under the “Actions” tab on the GitHub pages. The configuration is done as usual with other tools by a file, which is stored in the repository under .github/workflows.

Demo project

The demo project uses GitHub Actions so that a build is started on every commit and push to the repository. The configuration of the pipeline can be found in .github/workflows/create-cluster.yml.
Inside the GitHub action, a k3d Kubernetes cluster is started. K3d starts a minimal k3s cluster which is provided by Rancher. Rancher offers a minimal cluster with this Kubernetes distribution which is optimal for CI operation, among other things. It requires very little memory and also the time to start is extremely low. This temporary cluster is used as a deployment target for the Tekton installation.
After the cluster is available, the further installation starts with Flux , which is a tool used to implement the GitOps approach coined by Weaveworks. The configuration of the target cluster is managed in its entirety in git and is retrieved from there by the respective target cluster. GitOps tools and thus also Flux work pull-based compared to other declarative Infrastructure as Code tools.
Perhaps the question arises: “Why Flux?”. From my point of view, it is a tiny, flexible deployment tool that allows the installation of Helm releases and simple YAML manifests with little effort. Integration into the deployment and the testing process are also quick.
Flux installs the basic components of Tekton and also the desired additional components of the pipelines. Once all the installation steps have been completed, a fully-fledged CI system is available. The installation steps are mapped in the form of customizations which thus define the installation sequence of Kubernetes manifests. In this demo project, the Flux installation starts in clusters/local/ and progresses from there through further references to installation files.
The process outlined above runs autonomously and asynchronously. How can you tell in the CI process when the cluster is deployable? Determining this moment reproducibly, as error-free as possible and precisely is very important for testing, because otherwise there are always false positives in the pipeline, which would greatly reduce its acceptance by the teams using it.
A fundamental problem is that the status of the Flux deployment is made available via a CRD, which itself is installed and made available by Flux.
With on-board means of kubectl wait it is not possible to wait for a status change of a possibly not yet existing CRD, therefore the step “Wait for Flux CRD Deployment” was used. After this run, kubectl wait commands are used to wait for the successful run of the individual Flux customizations. When this has been achieved, the actual testing can begin.

The testing

From here on, it’s all about the test run in the ready-made K3d cluster. We start a Tekton pipeline by creating a PipelineRun as follows:

1apiVersion: tekton.dev/v1beta1
2    kind: PipelineRun
3    metadata:
4      generateName: hello-goodbye-pipeline-
5    spec:
6      pipelineRef:
7        name: hello-goodbye-pipeline

Interesting here is that the PipelineRun is generated according to a simple naming scheme configured via generateName. The prefix specified here is extended by a unique id by the system. The desired pipeline is specified via pipelineRef. For our simple case a sufficient configuration. More complex pipelines can have additional parameters and workspaces added here.

1- name: Execute and verify hello-world pipeline is run to completion
2    run: |
3        PIPELINE_RUN_NAME=$(kubectl create -n tekton-pipelines -f ./hello-goodbye-pipeline-run.yaml -o json | jq -r '.metadata.name')
4        kubectl -n tekton-pipelines wait --for=condition=SUCCEEDED=True --timeout=60s pipelineruns.tekton.dev/$PIPELINE_RUN_NAME
5        tkn pr --namespace tekton-pipelines logs $PIPELINE_RUN_NAME

The PipelineRun shown above is started by the test and the generated name is read from the API with the Swiss Army Knife jq .
Now we just wait for the pipeline to run successfully and report back a success if this is met.

kubectl -n tekton-pipelines wait --for=condition=SUCCEEDED=True --timeout=60s pipelineruns.tekton.dev/$PIPELINE_RUN_NAME

The GitHub Actions job is green when the pipeline has been successfully run.

Recap

With the approach described here, we have automated the testing of an entire CI system from the ground up. No more unpleasant surprises when a small change in the pipeline causes unwanted effects or the system has to be rebuilt. The tests are always running and make it possible for us to develop this part of the infrastructure like a product.
If you have comments or questions feel free to contact me.