Going full GitOps with Crossplane & ArgoCD

In the last post we already deployed Crossplane with ArgoCD in a GitOps-fashion. But what about Crossplane providers and their configuration? And can't we optimize the boostrapping with the ArgoCD App-of-Apps pattern? We can! And we'll also provision some infrastructure in AWS...

Crossplane & ArgoCD – blog series

1. From Classic CI/CD to GitOps with ArgoCD & Crossplane
2. Bootstrapping Crossplane with ArgoCD
3. Going full GitOps with Crossplane & ArgoCD
4. Using External Secrets with Crossplane & ArgoCD

This is the second part of an in-depth guide on how to bootstrap Crossplane with ArgoCD. If you followed along the first post, you should already have Kubernetes in Docker (kind) running as a management cluster including ArgoCD, which manages the Crossplane installation through an Argo Application. But although Crossplane is running, we'll also need to install and configure Crossplane providers in order to get Crossplane ready to provision cloud resources.

So let's enhance the first blog post's setup. Here's a sketch of the idea behind this blog post:

At the end of this post we should be able to provision AWS resources (like an S3 Bucket) with Crossplane and Argo. So let's jump right in and start with the AWS Provider configuration.

Again you can find the fully comprehensible code for everything mentioned in this blog post on GitHub.

Create Crossplane AWS Provider Secret

Throughout this post we will use the AWS provider for S3 as an example of the many available Crossplane providers. Before you get started, be sure to have a working and authenticated installation of the aws CLI.

Now as we need to give the Crossplane AWS provider the possibility to create resources in AWS, we must provide it with credentials of our IAM account. Therefore let's create a file called aws-creds.conf with the following contents:

1echo "[default]
2aws_access_key_id = $(aws configure get aws_access_key_id)
3aws_secret_access_key = $(aws configure get aws_secret_access_key)
4" > aws-creds.conf

Don't ever check this file into source control - it holds your AWS credentials! To prevent that I added *-creds.conf to the .gitignore file in this post's example repository.

Using this file we can now create a Secret that the Crossplane AWS Provider will be able to consume:

1kubectl create secret generic aws-creds -n crossplane-system --from-file=creds=./aws-creds.conf

In an upcoming blog post I will show a different approach to the use of a local Secret by integrating an external secrets provider into the setup.

Install Crossplane AWS S3 Provider with ArgoCD

As the Upbound providers for Crossplane implement the broadest spectrum of Cloud provider resources, we will be focussing on them throughout this post. Therefore let's create a new directory upbound/provider-aws in the root of our repository. This directory will hold every Upbound provider featured in the AWS family (S3, IAM, EC2, EKS, etc.).

If you want to learn more about the migration from community developed Crossplane providers to the Upbound providers and the following split into provider families, have a look at this post.

Inside of upbound/provider-aws let's create another directory provider. In there we can create a the Provider manifest for the AWS S3 provider in the file upbound/provider-aws/provider/upbound-provider-aws-s3.yaml:

1apiVersion: pkg.crossplane.io/v1
2kind: Provider
3metadata:
4  name: upbound-provider-aws-s3
5spec:
6  package: xpkg.upbound.io/upbound/provider-aws-s3:v1.12.0
7  packagePullPolicy: Always
8  revisionActivationPolicy: Automatic
9  revisionHistoryLimit: 1

Now how do we let ArgoCD manage and deploy the provider to our management cluster? Therefore we create a new ArgoCD Application CRD in the already present directory argocd/crossplane-bootstrap called argocd/crossplane-bootstrap/crossplane-provider-aws.yaml with the following contents:

1apiVersion: argoproj.io/v1alpha1
2kind: Application
3metadata:
4  name: crossplane-provider-aws-s3
5  namespace: argocd
6  finalizers:
7    - resources-finalizer.argocd.argoproj.io
8spec:
9  project: default
10  source:
11    path: upbound/provider-aws/config
12    repoURL: https://github.com/jonashackt/crossplane-argocd
13    targetRevision: HEAD
14  destination:
15    namespace: default
16    server: https://kubernetes.default.svc
17  # Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with
18  # 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'
19  syncPolicy:
20    automated: 
21      prune: true

As already used in the previous post we use the spec.source.path to define a directory that contains our manifest(s) for ArgoCD to pick them up.

A crucial point here is to use the syncPolicy.automated flag as described in the docs. As shown with the Crossplane deployment in the first blog post the automated syncPolicy makes sure that the Application's child apps are automatically created, synced, and deleted when the manifest is changed (aka ArgoCD's "true" GitOps feature). Without the syncPolicy configuration the deployment of the Crossplane provider-aws-s3 will give the following error:

1Resource not found in cluster: pkg.crossplane.io/v1/Provider:upbound-provider-aws-s3

We also use the finalizer resources-finalizer.argocd.argoproj.io like we did with the Crossplane core components. This way a kubectl delete -f will also undeploy all components of our Provider upbound-provider-aws-s3.

Let's apply this ArgoCD Application to our cluster now:

1kubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane-provider-aws.yaml

Here we're expecting to encounter an error. Having a look into the ArgoCD UI at http://localhost:8080 you should spot it while Argo tries to sync our Application:

1The Kubernetes API could not find aws.upbound.io/ProviderConfig for requested resource default/default. Make sure the "ProviderConfig" CRD is installed on the destination cluster.

That's no big deal, we'll fix that one in a second.

Install AWS Provider ProviderConfig with ArgoCD

To get our Provider finally working we also need to create a ProviderConfig accordingly that will tell our Crossplane Provider where to find it's AWS credentials. Therefore we create a new directory config inside our already existing upbound/provider-aws folder inheriting a new file upbound/provider-aws/config/provider-aws-config.yaml:

1apiVersion: aws.upbound.io/v1beta1
2kind: ProviderConfig
3metadata:
4  name: default
5spec:
6  credentials:
7    source: Secret
8    secretRef:
9      namespace: crossplane-system
10      name: aws-creds
11      key: creds

Be aware that the secretRef.name and secretRef.key have to match the fields of the provider Secret we already created earlier.

Crossplane resources use the ProviderConfig named default if no specific ProviderConfig is specified. So this ProviderConfig will be the default for all AWS resources.

To let ArgoCD manage and deploy our ProviderConfig we create a new ArgoCD Application CRD inside the directory argocd/crossplane-bootstrap called argocd/crossplane-bootstrap/crossplane-provider-aws-config.yaml. This time the Application spec.source.path points to upbound/provider-aws/config:

1apiVersion: argoproj.io/v1alpha1
2kind: Application
3metadata:
4  name: provider-aws-config
5  namespace: argocd
6  finalizers:
7    - resources-finalizer.argocd.argoproj.io
8spec:
9  project: default
10  source:
11    path: upbound/provider-aws/config
12    repoURL: https://github.com/jonashackt/crossplane-argocd
13    targetRevision: HEAD
14  destination:
15    namespace: default
16    server: https://kubernetes.default.svc
17  # Using syncPolicy.automated here, otherwise the deployement of our Crossplane provider will fail with
18  # 'Resource not found in cluster: pkg.crossplane.io/v1/Provider:provider-aws-s3'
19  syncPolicy:
20    automated: 
21      prune: true

That should be our final bit to get the AWS Provider working in our Setup. Don't forget to apply it via:

1kubectl apply -n argocd -f argocd/crossplane-bootstrap/crossplane-provider-aws-config.yaml

We finally managed to let ArgoCD deploy the Crossplane core components together with the AWS Provider and ProviderConfig correctly:

Using ArgoCD's SyncWaves feature to deploy Crossplane components

While our setup is already fully working in a GitOps fashion, something bothered me. We now have a lot of ArgoCD Application files, that need to be applied in a specific order. And imagine if we would enhance our setup even further (e.g. by integrating the External Secrets Operator or other Crossplane providers). Wouldn't it be much better to have a single manifest defining the whole Crossplane setup incl. core components, Provider, ProviderConfig?

Maybe a naive approach could be to implement an Application that points to a directory where all manifests reside at the same time. But with this aproach we'll run into errors like following:

1The Kubernetes API could not find aws.upbound.io/ProviderConfig for requested resource default/default. Make sure the "ProviderConfig" CRD is installed on the destination cluster.

The issue here is that the order of deployment wouldn't be clear. For example the Provider manifests need to be deployed before the ProviderConfig. Otherwise the deployment fails because of missing CRDs. But wasn't Argo's SyncWaves feature designed for exactly such a scenario?

Sure but using Argo's SyncWaves alone doesn't seem to help here. I tried to use them at Crossplane manifests and got confused, since the deployment didn't work as expected. Finally I understood: To use the SyncWaves feature for our Crossplane ArgoCD deployment, one would need to add the argocd.argoproj.io/sync-wave annotations on every of the Crossplane Provider's Kubernetes objects. And thus alter the manifests to add the annotation... But isn't there another ArgoCD feature we could combine together with SyncWaves to achieve our goal of a single Crossplane deployment manifest?!

App of Apps for Crossplane bootstrapping

Entering ArgoCD's App of Apps pattern. ArgoCD Applications can be used inside ArgoCD Applications too, since they are normal Kubernetes CRDs. So if we would have an Application for every component we want to deploy, we could use SyncWaves to define the exact deployment order. And at the same time avoiding the need to alter any Crossplane manifest itself.

So first we should start to define a new top level Application (aka "App of Apps") that manages our whole Crossplane setup. I created my App of Apps definition in the file argocd/crossplane-bootstrap.yaml:

1# The ArgoCD App of Apps for all Crossplane components
2---
3apiVersion: argoproj.io/v1alpha1
4kind: Application
5metadata:
6  name: crossplane-bootstrap
7  namespace: argocd
8  finalizers:
9    - resources-finalizer.argocd.argoproj.io
10spec:
11  project: default
12  source:
13    repoURL: https://github.com/jonashackt/crossplane-argocd
14    targetRevision: HEAD
15    path: argocd/crossplane-bootstrap
16  destination:
17    server: https://kubernetes.default.svc
18    namespace: crossplane-system
19  syncPolicy:
20    automated:
21      prune: true    
22    syncOptions:
23    - CreateNamespace=true
24    retry:
25      limit: 1
26      backoff:
27        duration: 5s 
28        factor: 2 
29        maxDuration: 1m

As you can see this Application will look for manifests inside the directory argocd/crossplane-bootstrap in our repository. And luckily all our Crossplane components are already defined there as ArgoCD Application manifests :)

Also in our App of Apps we shouldn't forget to define the finalizers finalizers: - resources-finalizer.argocd.argoproj.io again. Otherwise the Applications managed by this App of Apps won't be deleted and will still be running, if we'd only delete the App of Apps!

You might ask: why not use ApplicationSets instead of App of Apps? The Argo devs tell us: App of Apps is not depreciated. "The idea of deploying Applications (which are just Kubernetes resources) from another Application is fundamental to how Argo CD works. It would be difficult to remove even if we wanted to."

Combining SyncWaves with the App of Apps Pattern

Now we have everything in place to combine Argo's SyncWaves feature with our use of the App of Apps pattern. Using SyncWaves we can define the exact order in which an Application (representing a Crossplane component each) needs to be deployed by ArgoCD.

Let's now prepare each Application inside the argocd/crossplane-bootstrap directory to finally use SyncWaves. First we need to deploy the Crossplane Helm Secret, so we add the annotations: argocd.argoproj.io/sync-wave configuration to it's metadata:

1metadata:
2  annotations:
3    argocd.argoproj.io/sync-wave: "0"

We use sync-wave: "0" here, to define it as the earliest stage of our Argo App of Apps deployment. We could use negative numbers though, but for simplicity let's start at zero.

For more info about the Helm Secret and Crossplane core component deployments, I suggest to have a look into the previous post.

Then we need to deploy the Crossplane core components, defined in argocd/crossplane-bootstrap/crossplane.yaml. There we add the next SyncWave as sync-wave: "1":

1metadata:
2  annotations:
3    argocd.argoproj.io/sync-wave: "1"

You get the point! We also add the sync-wave annotation to the AWS Provider in argocd/crossplane-bootstrap/crossplane-provider-aws.yaml (sync-wave: "2") and the ProviderConfig at argocd/crossplane-bootstrap/crossplane-provider-config-aws.yaml (sync-wave: "3").

If we finished adding the argocd.argoproj.io/sync-wave annotations to all our Argo Applications inside of argocd/crossplane-bootstrap, we should have everything in place to use our Crossplane App of Apps in Argo. If you followed along these post's steps, you may need to delete all components previously deployed:

1kubectl delete -n argocd -f argocd/crossplane-bootstrap/crossplane-helm-secret.yaml
2kubectl delete -n argocd -f argocd/crossplane-bootstrap/crossplane.yaml
3kubectl delete -n argocd -f argocd/crossplane-bootstrap/crossplane-provider-aws.yaml
4kubectl delete -n argocd -f argocd/crossplane-bootstrap/crossplane-provider-aws-config.yaml

This overview of manifests to delete makes us aware again of the benefits of having only a single manifest for Crossplane deployment (and just image integrating even more components into the setup).

Now we are able to finally apply our Crossplane App of Apps in Argo:

1kubectl apply -n argocd -f argocd/crossplane-bootstrap.yaml

And it's like magic! All our Crossplane components get deployed step by step in correct order:

If we have a look into crossplane-bootstrap App of Apps we see all the needed components to deploy a running Crossplane installation using ArgoCD (which I found is super nice):

So this looks like a viable setup of Crossplane and ArgoCD! The only thing missing now is to show it can provision real infrastructure in AWS.

Provisioning Cloud resources with Crossplane and ArgoCD

For simplicity reasons let's provision a simple S3 Bucket in AWS (more complex infrastructure will work nearly the same way). The easiest way to provision cloud resources with Crossplane is to use the Managed Resources (MRs) delivered by the Crossplane provider. If you want to learn more about the basic concepts of Crossplane, have a look at this blog post I wrote a while ago.

The Upbound AWS S3 provider docs aid us with some help on how to use the Bucket resource. So let's create another directory in the root of our repository called infrastructure/s3. Therein we then create a new manifest called infrastructure/s3/simple-bucket.yaml with the following contents:

1apiVersion: s3.aws.upbound.io/v1beta1
2kind: Bucket
3metadata:
4  name: crossplane-argocd-s3-bucket
5spec:
6  forProvider:
7    region: eu-central-1
8  providerConfigRef:
9    name: default

In this simple example the only relevant configuration is the AWS region eu-central-1 for our S3 Bucket.

Remember we want ArgoCD to trigger Crossplane, which in turn should provision our S3 Bucket in AWS. So we again need an ArgoCD Application manifest here.

Therefore let's create a new folder infrastructure in the argocd directory, since the Crossplane provisioned infrastructure may not automatically be part of the bootstrap setup (in fact it would not even be part of the repository in a production setup). Inside the argocd/infrastructure folder let's create our Application called argocd/infrastructure/aws-s3.yaml:

1# The ArgoCD Application for all Crossplane Managed Resources
2---
3apiVersion: argoproj.io/v1alpha1
4kind: Application
5metadata:
6  name: aws-s3
7  namespace: argocd
8  finalizers:
9    - resources-finalizer.argocd.argoproj.io
10spec:
11  project: default
12  source:
13    repoURL: https://github.com/jonashackt/crossplane-argocd
14    targetRevision: HEAD
15    path: infrastructure
16  destination:
17    namespace: default
18    server: https://kubernetes.default.svc
19  syncPolicy:
20    automated:
21      prune: true    
22    retry:
23      limit: 5
24      backoff:
25        duration: 5s 
26        factor: 2 
27        maxDuration: 1m

Applying this Application to our management cluster we now tell ArgoCD to trigger Crossplane that in turn should create our S3 Bucket. Let's apply it with:

1kubectl apply -f argocd/infrastructure/aws-s3.yaml

If everything went fine, the Argo app should look Healthy like this:

And inside the AWS console, there should be a new S3 Bucket provisioned:

ArgoCD & Crossplane unlock a new GitOps level

We managed to put the final bits together to get Crossplane completely working with ArgoCD. Therefore we configured authentication to AWS and installed the Upbound AWS S3 provider. The ProviderConfig provides the AWS credentials to the AWS provider.

Although already working, we saw the possible benefits of a single entrypoint to take care of the whole Crossplane installation & provider configuration. Therefore we explored the benefits of combining ArgoCD's SyncWave feature together with the App-of-Apps pattern. Now we have a single manifest to apply to ArgoCD, which now takes over to install and manage all the moving parts of Crossplane. Right now they don't seem to be that many of them, but looking towards integrating multiple Crossplane providers and concepts like external secret stores etc. this should make for a future proof setup.

We also finally verified if our setup fully works by provisioning a S3 Bucket through an Crossplane Managed Resource in AWS.

But as you may already guessed it: There's even more to do! I already often mentioned the external secret store as an alternative to the local authentication Secret our setup currently uses. We should also have a look at provisioning a more complex infrastructure like an AWS EKS cluster and registering that as a deployment cluster in ArgoCD. And maybe even deploy a microservice application onto the newly created cluster... I hope to come up with follow up blog posts for these topics soon.