Bootstrapping Crossplane with ArgoCD

After going into detail about why the integration of Crossplane and ArgoCD is a great way to unlock a new level of GitOps, I promised to dive into the details of such a setup. Here we are! Let's have a look at the basic steps how to use Crossplane together with ArgoCD.

Crossplane & ArgoCD – blog series

1. From Classic CI/CD to GitOps with ArgoCD & Crossplane
2. Bootstrapping Crossplane with ArgoCD
3. Going full GitOps with Crossplane & ArgoCD
4. Using External Secrets with Crossplane & ArgoCD

If you have read my introductory blog post, you also know what the architectural overview of an integration of Crossplane and ArgoCD looks like (if not: just have a look into the last paragraph Adding Crossplane to apply GitOps to infrastructure provisioning (100% Pull)). In the current post we will concentrate on the pipeline described in the bottom of the full overview: The bootstrapping of a management cluster with ArgoCD and Crossplane:

Prerequisites: a management cluster for ArgoCD and Crossplane

Building a control plane based architecture means: we need a management cluster to rule our infrastructure & deployments. We can circumvent this "chicken and egg problem" (as my colleague Manuel stated :) ) in a variety of ways. One of the simplest solution is to use Kubernetes in Docker (kind) for the management cluster. Therefore we have to install kind together with other tools we need. On a Mac this can be done via homebrew:

1brew install kind helm kubectl kustomize argocd

These other tools include the CLIs of the Kubernetes package manager helm, Kustomize, argocd and mighty kubectl. Also having k9s installed isn't a bad idea. As we're using kind, I assume you also have a running Docker installation.

Now we have everything in place to spin up a local kind cluster

1kind create cluster --image kindest/node:v1.30.4 --wait 5m

Making ArgoCD ready for Crossplane

Before even starting to install ArgoCD, we should be aware of some necessary configuration details in order to let Argo run smootly with Crossplane. The Crossplane docs mention these. But not all require immediate action. For example we can ignore the mentioned health status configuration for now, since the health status for the Providers already work out-of-the box:

"Some checks are supported by the community directly in Argo’s repository. For example the Provider from pkg.crossplane.io has already been declared which means there is no further configuration needed."

So we should focus on the configuration of the annotation based resource tracking in ArgoCD and the exclusion of Crossplane generated ProviderConfigUsage CRDs.

Configure annotation based resource tracking in ArgoCD

Did you ever thought about how ArgoCD tracks resources? There are multiple different ways as the docs state. But with Crossplane we need to use annotation based resource tracking.

You may have already used ArgoCD with resource tracking via the well-known label app.kubernetes.io/instance, which is the default resource tracking mode. But from ArgoCD 2.2 on there are additional ways of tracking resources. One of them is the annotation based resource tracking. This has some advantages:

"The advantages of using the tracking id annotation is that there are no clashes any more with other Kubernetes tools and Argo CD is never confused about the owner of a resource. The annotation+label can also be used if you want other tools to understand resources managed by Argo CD."

The resource tracking method has to be configured inside the argocd-cm ConfigMap using the application.resourceTrackingMethod field:

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: argocd-cm
5data:
6  # Set Resource Tracking Method (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-tracking-method)
7  application.resourceTrackingMethod: annotation

If you ask yourself, where to actually do the ConfigMap configuration: We'll cover that in a moment.

Exclude Crossplane generated ProviderConfigUsage CRDs

The second necessary configuration refers to the exclusion of Crossplane generated ProviderConfigUsage CRDs:

Crossplane providers generates a ProviderConfigUsage for each of the managed resource (MR) it handles. This resource enable representing the relationship between MR and a ProviderConfig so that the controller can use it as finalizer when a ProviderConfig is deleted. End-users of Crossplane are not expected to interact with this resource.

Imagine having a lot of Crossplane Resources that you want to work with. Just like it is shown in the following image. One of the issues is that the ArgoCD UI reactivity can be impacted and become slow and confusing:

And because these resources don't give us anymore insights, we can safely remove them as ArgoCD resources. Therefore we should also configure the ProviderConfigUsage exclusion in the argocd-cm ConfigMap:

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: argocd-cm
5data:
6  ...
7  # Set Resource Exclusion (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-exclusion)
8  resource.exclusions: |
9    - apiGroups:
10      - "*"
11      kinds:
12      - ProviderConfigUsage

We will actually configure all this while installing ArgoCD in a second. But the question is: where exactly can we change parameters of the argocd-cm ConfigMap in ArgoCD?

Install ArgoCD into the management cluster

This question boils down to another question on a higher level: How do we install ArgoCD and change the ConfigMap in a flexible and GitOps-style way? Ideally also in a renovatebot-enabled fashion. I stumbled upon this question already some time ago and found that the use of Kustomize as is a great solution here (which is also described in the Argo docs).

In fact the ArgoCD team itself uses Kustomize to deploy their own ArgoCD instances. A live deployment is available here and the configuration used can be found on GitHub.

Using Kustomize enables a great way of declaritively changing configuration in ConfigMaps, while using the default installation method (which is this install.yaml). And at the same time staying upgradable via Renovate.

You can find the fully comprehensible code for everything in this blog post on GitHub.

So let's first create a directory argocd/install in the root of our repository. Therein we create a file called kustomization.yaml with the following contents:

1apiVersion: kustomize.config.k8s.io/v1beta1
2kind: Kustomization
3
4resources:
5- github.com/argoproj/argo-cd//manifests/cluster-install?ref=v2.12.2
6- argocd-namespace.yaml
7
8## changes to config maps
9patches:
10- path: argocd-cm-patch.yaml
11
12namespace: argocd

Under the resources parameter you can see a link to a ArgoCD installation manifest, followed by the ArgoCD version tag. This is a great way of enabling Renovate to keep our setup up-to-date automatically.

As Kustomize has the ability to use patch files, we can also create a file argocd-cm-patch.yaml in argocd/install. Here we can configure both the annotation based resource tracking mode and exclude the Crossplane generated ProviderConfigUsage CRDs from ArgoCD:

1apiVersion: v1
2kind: ConfigMap
3metadata:
4  name: argocd-cm
5data:
6  # Set Resource Tracking Method (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-tracking-method)
7  application.resourceTrackingMethod: annotation
8  # Set Resource Exclusion (see https://docs.crossplane.io/knowledge-base/integrations/argo-cd-crossplane/#set-resource-exclusion)
9  resource.exclusions: |
10    - apiGroups:
11      - "*"
12      kinds:
13      - ProviderConfigUsage

Additionally to our ConfigMap patch we create another file argocd-namespace.yaml, that will automatically create the namespace argocd for us:

1apiVersion: v1
2kind: Namespace
3metadata:
4  name: argocd

With this simple manifest and it's integration into our kustomization.yaml, we don't need to explicitely run kubectl create namespace argocd.

Now we have everything prepared to install ArgoCD via Kustomize. Simply run a kubectl apply -k aimed to our previously created directory:

1kubectl apply -k argocd/install

Before moving to the next steps, we should wait for ArgoCD to deploy all it's components successfully. Therefore we can run a kubectl wait in the console, which will return a condition met if Argo is fully available:

1kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server --namespace argocd --timeout=300s

Accessing the ArgoCD UI

Since we're using ArgoCD, we should also be able to access it's fantastic UI in our browser. Therefore we first need to obtain the initial password for the admin user on the command line:

1kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

In order to make the argocd-server available outside of our management cluster we have multiple options. One of the simplest is a port-forward:

1kubectl port-forward -n argocd --address='0.0.0.0' service/argocd-server 8080:80

Now we can access the ArgoCD UI inside our Browser at http://localhost:8080 using admin user and the obtained password.

Remember to change the initial password in production environments!

A simple Helm chart

At this point, a question popped up in my head: Is it possible to use a GitOps-style approach right here to install Crossplane? Let's try it.

There are some options to install Crossplane. I wanted to achieve the same level of Renovate-enabled installation for Crossplane, as we implemented it for ArgoCD. As Renovate needs a dependency file to look at, we can create a simple local Helm chart as explained here. This will enable Renovate to automatically update our Crossplane version in our bootstrap setup while having a great ArgoCD integration at the same time.

Therefore let's create another directory in the root of our repository called crossplane. Inside of it we can create our local chart as Chart.yaml:

1apiVersion: v2
2type: application
3name: crossplane-argocd
4version: 0.0.0 # unused
5appVersion: 0.0.0 # unused
6dependencies:
7  - name: crossplane
8    repository: https://charts.crossplane.io/stable
9    version: 1.16.0

As we're going "full GitOps" this Helm chart needs to be picked up by Argo in a declarative GitOps way (not through the UI). But as this is a non-standard Helm Chart, we need to define a Secret first (as the docs state) that defines the Helm repo location:

"Non standard Helm Chart repositories have to be registered explicitly. Each repository must have url, type and name fields."

As we will expand the Crossplane bootstrapping throughout the course of this (and follow up) blog post(s), we should also create a new folder in our already created argocd directory: argocd/crossplane-bootstrap. Inside of this we can now create a manifest for the needed Secret in the file crossplane-helm-secret.yaml:

1apiVersion: v1
2kind: Secret
3metadata:
4  name: crossplane-helm-repo
5  namespace: argocd
6  labels:
7    argocd.argoproj.io/secret-type: repository
8stringData:
9  name: crossplane
10  url: https://charts.crossplane.io/stable
11  type: helm

In order to get this to work we need to apply it to our management cluster:

1kubectl apply -f argocd/crossplane-bootstrap/crossplane-helm-secret.yaml

Bootstrap Crossplane with ArgoCD

It's now time to define an ArgoCD Application for Crossplane. This way we can tell ArgoCD in a declarative way to use our simple Helm Chart to manage the Crossplane installation. Let's create the manifest in argocd/crossplane-bootstrap/crossplane.yaml:

1# The ArgoCD Application for crossplane core components themselves
2---
3apiVersion: argoproj.io/v1alpha1
4kind: Application
5metadata:
6  name: crossplane
7  namespace: argocd
8  finalizers:
9    - resources-finalizer.argocd.argoproj.io
10spec:
11  project: default
12  source:
13    repoURL: https://github.com/jonashackt/crossplane-argocd
14    targetRevision: HEAD
15    path: crossplane
16  destination:
17    server: https://kubernetes.default.svc
18    namespace: crossplane-system
19  syncPolicy:
20    automated:
21      prune: true    
22    syncOptions:
23    - CreateNamespace=true
24    retry:
25      limit: 1
26      backoff:
27        duration: 5s 
28        factor: 2 
29        maxDuration: 1m

As you can see we're using a special finalizer here:

"Without the resources-finalizer.argocd.argoproj.io finalizer, deleting an application will not delete the resources it manages. To perform a cascading delete, you must add the finalizer. See App Deletion."

In other words, if we would run kubectl delete -n argocd -f argocd/crossplane-bootstrap/crossplane.yaml, Crossplane wouldn't be undeployed as we may think. Only the ArgoCD Application would be deleted, but Crossplane Pods etc. would still be running.

Additionally we defined the spec.source to use our repository url via the repoURL parameter. Here you need to configure the Git repository. ArgoCD will later observe for manifests and changes of these over time. Since we created a crossplane directory in that exact repository, we defined the path accordingly.

In the destination field we configured to simply use our management cluster where the server parameter points to our local kind cluster - the home of our installed ArgoCD. As a namespace we use crossplane-system, which is the exact namespace Crossplane wants to be deployed to.

Finally we define some details about how the Crossplane deployment should be handled in the syncPolicy parameters. The first thing here is to fully enable the "100% Pull" GitOps-style behavior in ArgoCD by using syncPolicy:automated. With that our CI/CD pipelines do not need access to our clusters anymore and the GitOps operator pattern will take over any deployments. We also add the prune:ture configuration. This way ArgoCD will automatically delete Crossplane resources that aren't defined in Git anymore.

We also use syncOptions: - CreateNamespace=true here to let Argo create the crossplane crossplane-system namespace for us automatically. Lastly we define some retry configurations for the Crossplane deployment.

Finally it's time to let ArgoCD deploy Crossplane. Simply add our defined ArgoCD Application to the management cluster:

1kubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane.yaml

Now ArgoCD will take care of the Crossplane core components deployment! Here we can look over Argo's shoulder as it does it's work by having a look into the ArgoCD UI:

As you can see all the Crossplane components like the Crossplane operator, webhooks, the rbac-manager etc. are beeing deployed.

Bootstrapping Crossplane with ArgoCD is fun

To wrap up this post we already did the groundwork for a great GitOps setup! We created a management cluster based on kind and had a look into the pre-install configuration requirements to get Crossplane smoothly working together with ArgoCD.

We also installed ArgoCD in an extensible manner with the help of Kustomize, while at the same time making this setup suitable for Renovate. We achieved the same for the Crossplane installation through the use of a local Helm chart. Finally using an ArgoCD Application to let Argo manage the Crossplane installation is pretty cool and adheres to our GitOps principles.

But there are some topics left: We want to also install and configure Crossplane Providers (like the AWS Provider). And extending the Crossplane configuration we will also come to the need for some kind of all-in-one setup for all the moving parts. Finally we want to see some cloud resources beeing provisioned by Crossplane triggered through ArgoCD, right?! We will have a look into how all this can be achieved in the next upcoming blog post. Stay tuned!