Renovate in Azure DevOps Pipelines with Presets and Custom Managers

Introduction

First, let's take a look at the Renovate tool and why it is useful. Renovate helps to manage project dependencies by providing automated dependency updates. It tries to solve typical dependency problems, like the following:

• Security: Outdated dependencies can contain known security vulnerabilities which are already fixed in the latest version.
• Breaking changes: After a huge dependency upgrade, the code does not work anymore due to breaking changes. Because the upgrade contained multiple versions, it is difficult to find out which exact version caused the trouble and how to fix the code.
• Unknown dependencies: It is unknown what the dependencies of the system are. If the system needs to be installed on a new instance, nobody knows which dependencies to install.
• Deprecations: The installed dependency version is deprecated and will be discontinued. Therefore, the team is forced to migrate, but due to a lot of new versions and breaking changes this might be a time-consuming process.

Renovate tries to mitigate these issues by detecting and updating dependencies automatically. This can help to get an overview of the dependencies and their versions, and it can help to keep them up-to-date.

Renovate supports many typical dependencies out of the box (e.g. npm dependencies in package.json files or python dependencies in requirements.txt files), but for some projects this is not enough. For more specific use cases, Renovate provides comprehensive customization options to, e.g. extend the collection of supported dependencies. In the sections 1-2, a selection of advanced customization options are covered and a complete setup with presets and custom managers is presented.

Renovate and Azure DevOps

If you are working on Gitlab, you can refer to this blog article (german only) for a beginner’s guide to setup Renovate on Gitlab. The customizations presented in the next paragraphs are targeted at Azure DevOps but should be mostly usable “as is” on other platforms as well.

We worked with a self-hosted Renovate instance on the Azure DevOps platform. This includes Git repository hosting, CI/CD pipelines and configuration management. Although Renovate aims to be platform-independent, there are some technical limitations on Azure that we have encountered along the way: Firstly, there is no dependency dashboard, and secondly, the syntax for Renovate presets is limited. Therefore, the article also provides insight into how to handle these Azure limitations.

Part 0: Initial Setup

This part describes how to create an initial setup of Renovate in Azure DevOps. Renovate is intended to track multiple distinct repositories. Therefore, we host the configuration for Renovate in a dedicated repository. There are good tutorials on how to set up Renovate in Azure DevOps, e.g. the official tutorial by Renovate. If you already have a working setup of Renovate on Azure DevOps, just skip to Part 1. Otherwise, here are the details of our setup:

In the Renovate repository the Renovate pipeline is located. Each time the pipeline runs, Renovate checks for dependency updates for all tracked repositories. Each repository that should be tracked by the Renovate bot is added to the repositories list in the Renovate/config.js file and needs a renovate.json5 configuration on its own main branch.

The config.js file holds the self-hosted Renovate configuration. It contains the most general configuration needed to get the Renovate bot up and running. In the hostRules list, the general merge request behaviour is configured and the repositories list allows to add more repositories to the tracked repositories by using the syntax <ProjectName>/<RepositoryName>.

1// Renovate/config.js
2
3module.exports = {
4   platform: 'azure',
5   endpoint: 'https://dev.azure.com/myOrganisation/',
6   token: process.env.TOKEN,
7   hostRules: [
8     {          
9       "azureAutoApprove": false,
10       "automerge": false,
11       "baseBranches": ["main"],
12     },
13   ],
14   repositories: [ // list of repos that should be tracked by Renovate
15     'MyProject/MyRepository',
16   ], 
17 };

The following is the Azure pipeline file. Remember to create an Azure pipeline from it! Then, the Renovate bot runs each day at 3am UTC in a scheduled pipeline. Of course, it is also possible to start the pipeline manually. You can view the Renovate debug logs in the logs of pipeline run. It is possible to change the Renovate log level via the LOG_LEVEL env variable.

1# Renovate/azure.renovate-pipeline.yaml
2
3schedules:
4 - cron: '0 3 * * *'
5   displayName: 'Every day at 3am (UTC)'
6   branches:
7     include: [main]
8   always: true
9
10
11trigger: none
12
13
14jobs:
15 - job: RunRenovate
16   steps:
17     - task: npmAuthenticate@0.230.1
18       inputs:
19         workingFile: .npmrc
20     - script: |
21         git config --global user.email 'bot@renovateapp.com'
22         git config --global user.name 'Renovate Bot'
23         echo "ℹ️ Running renovate..."
24         npx --userconfig .npmrc renovate
25         echo "✅ Renovate finished."
26       displayName: Run renovate
27       env:
28         RENOVATE_PLATFORM: azure
29         RENOVATE_ENDPOINT: $(System.CollectionUri)
30         RENOVATE_TOKEN: $(System.AccessToken)
31         LOG_LEVEL: debug

The following is the Renovate configuration file, which only applies to the respective tracked repository, where it is located. The example configuration below specifies to use the recommended configuration and to pin digests. The file is required to receive dependency updates and must be located on the main branch.

Note that for this example, we always used the .json5 ending for renovate.json5 configurations. This allows to add comments into the configuration file.

1// TrackedRepository/renovate.json5
2
3{
4  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
5  "extends": 
6  [
7    "config:recommended"
8  ],
9  "pinDigests": true,
10}

Finally, the .npmrc file is needed to detect npm dependencies and specifies the npm registry and authentication configuration.

// Renovate/.npmrc

registry=https://registry.npmjs.org
always-auth=true

With this initial setup, a lot of dependencies can already be detected. However, if a lot of repositories use the same or a very similar renovate.json5 configuration, using presets is the way to proceed (see Part 1). Furthermore, if Renovate does not detect all necessary types of dependencies, a custom dependency detection needs to be implemented (see Part 2).

Part 1: Setup Presets on Azure DevOps

Presets are used to share specific configuration values over multiple repositories. They can be used to avoid duplicate code and to distribute changes to the configuration automatically to all tracked repositories. A preset file is usually called default.json or default.json5 and it can have the same content as a renovate.json5 file.

Steps to create a shared preset configuration:

1. Compose a renovate.json5 configuration out of all configuration values that you want to share between repositories.
2. Place this configuration into a file named default.json or default.json5 and place the file into a publicly accessible repository, e.g. the Renovate repository. Note that the filenames renovate.json and renovate.json5 are deprecated for presets.
3. Use the preset in tracked repositories by adding the preset file reference to the extends section in renovate.json5 configurations. In Azure Devops, the file reference in the extends section must be constructed as shown in the scheme below:

1"extends": [
2    "local>ProjectName/RepositoryName:CompletePresetFilePath.json"
3],

There are a lot of different syntax variants for including custom presets, but this is the only syntax variant that works currently in Azure DevOps.

Example: Your Renovate repository is located under the url https://dev.azure.com/myOrg/myProject/_git/Renovate and the preset file is located under the path /default.json5, then you need the following renovate.json5 file in your tracked repositories:

1// TrackedRepository/renovate.json5
2{
3    "schema": "https://docs.renovatebot.com/renovate-schema.json”,
4    "extends": [
5        "local>myProject/Renovate:default.json5",
6    ],
7    // … further configuration values specific to the respective repository
8}

This syntax works also for custom orgs, as the local keyword references the Azure instance dev.azure.com/<org>/ that the project is located in, including the org.

The default renovate.json5 configuration can still be overwritten or abandoned by the tracked repositories, as each holds control over its particular renovate.json5 file. It can be decided for each repository whether to include the default and whether to append additional configuration.

This is the updated setup. The renovate.json5 configuration in each tracked repository now references the default configuration. The change in the azure.renovate-pipeline.yaml file is optional to achieve a shared preset configuration. In the illustration below, all files that are added or changed are coloured, files that did not change are white.

Complete file: TrackedRepository/renovate.json5: See example above.

1// Renovate/default.json5
2{
3  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
4  "extends": [
5    "config:recommended"
6  ],
7  "pinDigests": true,
8  // further configuration values that you want to share
9}

Optional: Validating the Default Configuration

Now, the tracked repositories can use the shared configuration. But what if the file contains an error and is invalid? The Renovate pipeline will fail silently and it might take some time until we detect the problem. Therefore, it is a good idea to extend the Renovate pipeline to validate the default.json5 configuration. This can be done using the renovate-config-validator.

1# Renovate/azure.renovate-pipeline.yaml
2
3echo "ℹ️ Validating the renovate default config..."
4npm install --global renovate 
5validator_output=$(renovate-config-validator default.json5)
6exit_code=$?
7if [[ $exit_code != 0 ]]; then
8  echo "❌" $validator_output
9  exit $exit_code;
10else
11  echo "✅" $validator_output
12fi

1# Renovate/azure.renovate-pipeline.yaml
2schedules:
3 - cron: '0 3 * * *'
4   displayName: 'Every day at 3am (UTC)'
5   branches:
6     include: [main]
7   always: true
8
9
10trigger: none
11
12
13jobs:
14 - job: RunRenovate
15   steps:
16     - task: npmAuthenticate@0.230.1
17       inputs:
18         workingFile: .npmrc
19     - script: |
20         git config --global user.email 'bot@renovateapp.com'
21         git config --global user.name 'Renovate Bot'
22         npm install --global renovate 
23         echo "ℹ️ Validating the renovate default config..."
24         validator_output=$(renovate-config-validator default.json5)
25         exit_code=$?
26         if [[ $exit_code != 0 ]]; then
27           echo "❌" $validator_output
28           exit $exit_code;
29         else
30           echo "✅" $validator_output
31         fi
32         echo; echo "ℹ️ Running renovate..."
33         npx --userconfig .npmrc renovate
34         echo "✅ Renovate finished."
35       displayName: Run renovate
36       env:
37         RENOVATE_PLATFORM: azure
38         RENOVATE_ENDPOINT: $(System.CollectionUri)
39         RENOVATE_TOKEN: $(System.AccessToken)
40         LOG_LEVEL: debug

Part 2: Custom dependency detection

If there are dependencies that Renovate cannot detect automatically, it is still possible to enable Renovate to track them as well. This is because Renovate allows adding custom regex managers and custom datasources. Managers and datasources can simply be appended to your configuration in the renovate.json5 / default.json5 file. Therefore, the dependency detection can be adapted to the specific needs of your projects.

If Renovate should detect dependencies that it supports in general (e.g. npm dependencies), but within a file or format that is not supported (e.g. within a bash script instead of a package.json), a custom manager is needed. If Renovate should detect a completely new type of dependency (e.g. detect API versions in links), then a custom datasource is also required.

In the example used in this blog post, custom managers and datasources are used to detect custom dependencies in bash scripts.

Part 2.1: Regex Manager

In Renovate, managers define in which files to search for the dependencies and how to detect the dependencies within the files. For custom managers, both are described using regular expressions (regex).

Custom managers can be defined in the “customManagers” list in the renovate.json5 configuration file. There are a lot of options to configure the custom manager that can be discovered in the Renovate documentation for custom managers, but most custom dependencies can be detected using the attributes from the example below.

Example: In the example, npm dependencies are detected from bash script files. The manager finds lines like npx package@version … commands in the script and extracts the npm dependencies out of those lines. Note that Renovate only detects a dependency if the version is given!

1// a section of a default.json5/renovate.json5 file
2"customManagers": [
3    {
4        // detect npm dependencies in bash scripts of the form “npx package@version … “
5        "customType": "regex",
6        "fileMatch": ["^.*\.(ba)?sh$"],
7        "matchStrings": ["npx\\s(?<depName>[@]?[^@\\s]*)[@]?(?<currentValue>[^@\\s]*)"],
8        "versioningTemplate": "npm",
9        "datasourceTemplate": "npm"
10    },
11],

• To define that you want to create a custom regex manager, you always need the attribute "customType": "regex".
• The fileMatch attribute defines in which files to search for dependencies. It is a list of regular expressions. In the case below, all files with the ending “.bash”/".sh" are matched.
• The matchStrings attribute defines how to find the dependency and all information about it within the matched files. For this it uses regular expressions. Named capture groups are used to label the dependency information.
  • The match group (?<depName>...) defines what part of the matched string is the dependency name (here: npm package name).
  • The match group (?<currentValue>...) defines what part of the matched string is the current version of the dependency.
• The versioningTemplate attribute defines the versioning schema. It describes how to evaluate whether the detected version is valid or not. If the version is invalid according to the versioning schema, the dependency is skipped (can be viewed in the debug logs). Renovate supports a variety of versioning styles to choose from. If there is no fitting versioning style, it is possible to define a custom versioning by providing a regex or to allow any versioning style by applying the loose versioning scheme.
• The datasourceTemplate attribute defines which datasource to use. The attribute value contains the name of the datasource. The datasource connects to a list of releases and detects what versions are available for the detected dependencies. Renovate supports npm dependencies out of the box, so for the example below I selected the datasource of Renovates list of datasources and did not have to write a custom datasource (see section “Part 2.2: Custom Datasources” below).

This is what the updated setup looks like. Due to the shared preset, only the default.json5 file needs to be changed if the dependency detection should be applied to all tracked repositories. The npm dependencies are now detected in bash files in all tracked repositories, without adapting their respective renovate.json5 configuration.

1// Renovate/default.json5
2{
3  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
4  "extends": [
5    "config:recommended"
6  ],
7  "pinDigests": true,
8  "customManagers": [
9        {
10            // detect npm dependencies in bash scripts of the form “npx package@version … “
11            "customType": "regex",
12            "fileMatch": ["^.*\.(ba)?sh$"],
13            "matchStrings": ["npx\\s(?<depName>[@]?[^@\\s]*)[@]?(?<currentValue>[^@\\s]*)"],
14            "versioningTemplate": "npm",
15            "datasourceTemplate": "npm"
16        },
17    ],
18}

Regex in Renovate configurations

Creating good regexes for the custom manager is probably the most difficult part. There are a few hints that can help to develop it:

• The regex strings need escaping in the JSON file. This means that all regexes need double backslashes instead of single backslashes.
• The regex matches per file and not per line, so be aware of that before using ^ and $.
• Renovate uses the regex parser package uhop/node-re2. It does not support all regex features, e.g. backreferences and lookahead assertions are not supported.
• regex101.com or other regex tools can help to develop the regex faster. The tool provides error messages when the regex syntax is wrong. It is much more convenient to test the regex here than within your repository with a long Renovate pipeline runtime and without error messages. For testing Renovate regex within 101regex.com, select the global regex flag and the ECMAScript (Javascript) regex flavour. Note that while working with this tool you need single backslashes, but do not forget to use double backslashes in the JSON file.
• It is possible to write tests for the regex managers rules in javascript by importing some Renovate packages. This blog post provides an example.

Part 2.2: Custom Datasources

Datasources define under which URL Renovate can find a list of releases and how to extract the versions and deprecation information from that URL using JSONata. Renovate comes with a lot of datasources already, so check out the datasource list before implementing one yourself. However, if the release list should be crawled from a private or a very unpopular spot, a custom datasource is needed. A custom datasource is also needed when Renovate does not support the dependency type at all.

Example: The example below demonstrates how to get all Azure Devops API versions in order to get automatic version updates for the API calls.

1// a section of a default.json5/renovate.json5 file
2
3"customManagers": [
4    {
5        // detect Azure DevOps REST API dependencies in bash files
6        "customType": "regex",
7        "fileMatch": ["^.*\.(ba)?sh$"],
8        "matchStrings": ["https:\/\/dev.azure.com\/.*?api-version=(?<currentValue>.*?)[&\"\s]"],
9        "versioningTemplate": "loose",
10        "depNameTemplate": "azure-rest-api",
11        "datasourceTemplate": "custom.azure-api"
12    },
13],
14"customDatasources": {
15    // get Azure DevOps REST API releases from the internet
16    "azure-api": {
17        "defaultRegistryUrlTemplate": "https://learn.microsoft.com/_api/familyTrees/bymoniker/azure-devops-rest-7.1",
18        "transformTemplates": ["{\"releases\": products.packages[isPrerelease = false].{\"version\": versionDisplayName, \"isDeprecated\": isDeprecated}, \"homepage\": \"https://learn.microsoft.com/en-us/rest/api/azure/devops/\"}"]
19    },
20}

• azure-api is the datasource name, which is to be inserted into the datasourceTemplate attribute of a manager.
• defaultRegistryUrlTemplate contains the URL where the release list can be queried
• transformTemplates is a list of JSONata strings and describes how to transform the URL data to a specific JSON structure that Redocly expects. Note that the JSONata block must be written in one line and quotes need to be escaped. It is recommendable to use an online tool for developing and testing the JSONata transformation. Read the Redocly documentation to find out about required and optional fields.

This is what the JSONata block from the transformTemplates attribute looks like when formatted:

1// The JSONata block using pretty formatting
2
3{
4      "releases":
5              products.packages[isPrerelease = false].{   
6                  "version": versionDisplayName,
7                  "isDeprecated": isDeprecated	 
8              },
9      "homepage": "https://learn.microsoft.com/en-us/rest/api/azure/devops/"
10 }

The JSONata block transforms the data from the url "https://learn.microsoft.com/en-us/rest/api/azure/devops/" (see homepageattribute) into the following structure. This is the (nearly minimal) structure that Redocly expects. The structure consists of a releases list, containing several release objects with version and deprecation information, and the homepage URL.

1// The data structure resulting from the JSONata transformation
2{
3    "releases": [
4            {  
5                "version": <version>,
6                "isDeprecated": <true/false>,	 
7            },
8    ],
9    "homepage": "https://learn.microsoft.com/en-us/rest/api/azure/devops/"
10}

This is the resulting updated setup:

1// Renovate/default.json5
2{
3    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
4    "extends":
5    [
6        "config:recommended"
7    ],
8    "pinDigests": true,
9    "customManagers":
10    [
11        {
12            // detect npm dependencies in bash scripts
13            "customType": "regex",
14            "fileMatch": ["^.*\.(ba)?sh$"],
15            "matchStrings": ["npx\\s(?<depName>[@]?[^@\\s]*)[@]?(?<currentValue>[^@\\s]*)"],
16            "versioningTemplate": "npm",
17            "datasourceTemplate": "npm"
18        },
19        {
20            // detect Azure DevOps REST API dependencies in bash scripts
21            "customType": "regex",
22            "fileMatch": ["^.*\.(ba)?sh$"],
23            "matchStrings": ["https:\/\/dev.azure.com\/.*?api-version=(?<currentValue>.*?)[&\"\s]"],
24            "versioningTemplate": "loose",
25            "depNameTemplate": "azure-rest-api",
26            "datasourceTemplate": "custom.azure-api"
27        },
28    ],
29    "customDatasources":
30    {
31        // get Azure DevOps REST API releases
32        "azure-api":
33        {
34            "defaultRegistryUrlTemplate": "https://learn.microsoft.com/_api/familyTrees/bymoniker/azure-devops-rest-7.1",
35            "transformTemplates": ["{\"releases\": products.packages[isPrerelease = false].{\"version\": versionDisplayName, \"isDeprecated\": isDeprecated}, \"homepage\": \"https://learn.microsoft.com/en-us/rest/api/azure/devops/\"}"]
36        },
37    }
38}

Tips for Debugging and Testing the Renovate Setup

Debugging and testing a Renovate setup can be difficult, because Renovate operates in a multi-repository environment which is not automatically reset after each Renovate test run. In addition, it can be difficult to find explanations for missing expected dependency updates. Therefore, testing can be a bit inconvenient. Here are some tips how to make the debugging and testing of the Renovate setup easier:

• Create a copy of the repository (the one to be tracked by Renovate) for developing and testing your Renovate configuration. The copy does not necessarily need to contain all files, only those that Renovate should check for dependencies. Like this, you do not mess with your actual code and don’t have to be that careful not to merge and break anything. The configuration file of Renovate needs to be on the main branch of your repository, so you also avoid a lot of merges of unfinished Renovate configurations to main.
• Do not set "automerge": true during development, as this configuration will automatically merge dependency updates. With this configuration enabled, the oudated dependency test data might be lost after each Renovate run.
• Use online tools to test the regex and JSONata blocks before testing the resulting configuration in your repository. This saves a lot of time.
• Turn on the DEBUG mode of Renovate by passing the log level via an environment variable. The debug logs provide a lot of information such as which dependencies and versions are detected, and which dependencies have incoming version updates (in the packageFiles with updates section of the logs).
• The Renovate dependency dashboard is not supported for Azure Devops. However, by programmatically parsing the Renovate debug logs, you can also create a custom dependency overview and print it in your pipeline.
• It is possible to test your custom rules in JavaScript using Renovate packages (see section Regex in Renovate configurations).

Summary

Renovate provides a wide range of customization options, way more than are covered in this article. With regex managers and custom datasources, Renovate is extensible for any automated dependency tracking and updating. With custom presets, Renovate configurations can be shared between multiple repositories.

Most of the setup presented in this article can be used "as is" on any other platform too. However, Azure Devops has some special syntax limitations and missing Renovate features. Firstly, we presented the only syntax variant that works for presets in Azure. Secondly, the missing dependency dashboard can be partially replaced by a self-implemented dependency overview based on the Renovate logs.