Time to Renovate

How to keep your IT infrastructure up to date and reduce manual effort to a minimum by using Kubernetes, Helm, GitOps (FluxCD), Continuous Integration (GitLab-CI) and Renovate.

When we moved into our house, everything was new and shiny. Well – it was new and shiny because we had put a lot of effort and money into making it so. But just a few weeks after we moved in, the first scratches and flaws appeared. Having kids only accelerated this slow process of decay – and I (Daniel) have three of them. Oh, how I would have loved to see someone walking through the house regularly, noting everything that had to be renewed and taking care of it (well, someone that wasn't me). But everyone owning a house knows: you're never done. There is always something to do. And it will cost you time, effort and money to keep your house in good shape.

IT infrastructures and software systems are no different. Okay – you (hopefully) don't have kids jumping around in them, touching every white wall with their dirty hands, but the “best before” date of a lot of software components in your infrastructure will be passed rather soonish. In addition to that, there is a constant chance of burglars (hackers) breaking into it if you do not update the lock (authentication server) or windows every few weeks. But as opposed to poor me having no one else taking care of all the renovation work in my house, in IT infrastructures we can leverage automation to be our faithful janitor. I would like to show you how we implemented this and reduced our manual renovation efforts to a minimum by using Kubernetes, Helm, GitOps (FluxCD), Continuous Integration (GitLab-CI) and Renovate.

Kubernetes

No matter if you are a fan of Kubernetes or not, it has become the de facto standard for hosting scalable multi-container environments. I completely agree that Kubernetes is used for a lot of use cases where it should not be used and other methods (some more traditional, some not) would have been a quicker, less complex and more efficient solution. However, once you have mastered the steep learning curve, Kubernetes and the cloud-native landscape have a lot of features, software projects and automations in the offering that can be used to make your day-to-day work a lot less tedious. No more forgotten certificate renewals with cert-manager, readily available monitoring and metrics with Prometheus, log aggregation with Grafana Loki – just to name a few. A huge set of tools ready and waiting to help you manage your scaling infrastructure and application needs. And all those shiny new tools can be installed with the cloud-native package manager: Helm.

Helm

Just like your beloved package managers for DEB, RPM or AUR packages, Helm is the package manager for Kubernetes. Most of the software tools that can be deployed to Kubernetes have Helm charts available that abstract and automate the installation process for you. Each Helm chart packages a single application that can be deployed and configured by just providing the chart, chart version, and a values file that customizes your application deployment according to your needs.

GitOps

“GitOps is the best thing since configuration as code.”

But what is it? GitOps is a declarative way of describing what components should be deployed to your infrastructure and how they should be configured. A good overview of what GitOps is, how it works and how you can get started with it has been provided by my colleague Daniel Henneberg here. In short: a GitOps tool (like Flux) is an agent that runs within your target infrastructure and that fetches your infrastructure or configuration as code definitions from a Git repository as its single source of truth. In a reconciliation loop, the GitOps agent checks if either the code fetched from Git or your infrastructure changed, and it (re-)applies the desired state found in the code to your system. This way you can: Review changes in merge requests upfront. Reduce error-prone manual interactions. Automate the tedious task of manual deployments. Have a versioned view and history of the changes made to your system.

Renovate – the faithful janitor

Having all the tools and concepts above in place is like having a garage full of material and equipment. It’s essential to have, but someone needs to actually use it. Otherwise, it will just pile up and once in a while, you will need a complete makeover of your house. Someone needs to regularly check what has to be done and (ideally) do it without you noticing: enter the faithful janitor. Coming back to IT infrastructures and software systems, our janitor is the open-source software Renovate and it comes for free.

Renovate can be used in multiple ways and on a decent number of platforms. In our case, Renovate runs as a scheduled Gitlab-CI job. Alternatively, you can use Renovate as GitHub Action or even as a (self-hosted) stateful application. When Renovate looks at a Git repository, it discovers what technologies and tools are used in the code and makes proposals on what to update. In each iteration it will automatically discover and (if configured) onboard new repositories, check for updated libraries and dependencies in them, open merge requests with the new versions and even merge them directly if you want it to. Renovate does all that in a fully configurable way – you decide how it should behave to fit your needs. To be frank, the configuration is sort of overwhelming at the beginning. But once you get the idea, you will experience lots of “wow, even that can be done with Renovate” moments. Renovate is not limited to the cloud-native world with containers and charts. It works with a huge number of package managers, e.g. with pip, npm, maven and gomod, to just state a very small selection.

In our case, Renovate takes care of updating all our installed Helm charts. When Renovate initially onboarded our repository, it created a Gitlab Issue called “Dependency Dashboard”. This issue lists all detected dependencies, open or blocked MRs and potential problems. Since the onboarding, Renovate checks every hour (through a scheduled CI/CD pipeline) if new versions of the charts we use are available and if so, it creates new merge requests. Previously created merge requests by Renovate will be rebased and kept up to date if yet another new version is available before the old one was applied. Minor and patch releases will automatically be merged into the branch tracked by FluxCD and applied immediately to our staging environments. Some of our production systems cannot be updated during the working hours because it would cause an interruption of service. For those, Renovate will only create the MR enriched with release notes retrieved from the chart’s source (mostly GitHub). In one of our setups, we install the same application multiple times in the cluster. Another handy Renovate feature: you can decide whether to update all instances at once (in a single merge request) or if you want to update them one by one.

Applying updates with Renovate and Flux

But how does our faithful janitor do its job? How do we get the new and shiny versions of the software applied to our systems? First of all, Renovate needs to check what software we use. Therefore it checks out our Git repository (1) and reads a renovate.json file in the root folder. This file represents Renovate's configuration for our repository and we can configure what Renovate should and should not do with it. Renovate then continues to discover what dependencies it can find in the repository and which version of the respective dependency is currently in use. With this knowledge, Renovate can then check (2) if there are newer versions available by looking up the project sources of the used dependencies on the respective upstreams. If Renovate finds that a newer version is available, it will open a merge request (3) for our project repository with the updated version numbers. Now this is what we wished for, right? A list of things that are to do, well compiled by our faithful janitor, and a sophisticated proposal of how to do them. Now it is up to us to decide what should be done, and if we might want to change the color of the tiles (adapt the configuration) while we are renovating the bathroom (some software). As the operator we now want to carefully read through those proposals (merge requests) and modify or adapt configurations to fit the new versions requirements. Finally we can then merge (4) the merge request and define the new software version as the version we want to have deployed to our system. This is when Flux takes over. In its reconciliation loop, Flux regularly checks for changes in the code (5) and tries to apply the state declared in the code towards our system. Due to the changes we just made by accepting a merge request, it will recognize a difference between the current state in the system and the declared state in code and act accordingly to update the software on the system.

Conclusion

By leveraging automation on both sides of the update process, i.e. sourcing in the updated software artifacts and rolling them out to the target system, we reduced our manual interaction for software system maintenance work to a bare minimum. Yet we keep full sovereignty over the updated chain by being able to accept, postpone or reject merge requests as we want. Our faithful janitor gathers and presents all the maintenance work in separate chunks and provides us with ready-to-use plans for the execution. Unfortunately this free-of-charge renovation full service is only available for software systems, not for our real-world houses. What a shame!