Keycloak.X, but secure – without vulnerable libraries

TLDR: How to reduce the known CVEs (common vulnerabilities and exposures) to zero by creating your own Keycloak distribution* .

Introduction

Keycloak (see website) will become easier and more robust by switching to Quarkus, at least that’s the promise. We have already shown how to approach a productive setup step by step in the blog post From Keycloak to Keycloak.X with an earlier version of Keycloak.X. In the meantime, version 18.0.0 has been released and the roadmap for the Keycloak project has been further concretized. Among other things, it states that the last Wildfly distribution will be released in September 2022 – from then on there will only be the Quarkus-based Keycloak distribution.

This article describes an approach to improve the performance and security of a Keycloak system by creating a customized Keycloak distribution. This requires complete control over the creation of one’s own Keycloak distribution.

Aspects of a custom Keycloak distribution

Creating your own customized Keycloak distribution can improve the security and/or performance of the running Keycloak system. As a counter-argument, we often hear that having one’s own distribution leads to unnecessary and increased complexity. In addition, there seems to be a general recommendation to use official images so that this part of the responsibility does not have to be borne by oneself. We argue here for the explicit assumption of this responsibility in the case of Keycloak and see great advantages in this step.

A custom distribution can support in the following:

1. Use of an optimized configuration for fast server start-up
2. Support of own extensions and themes
3. Only actually used Quarkus extensions activated
4. Additionally needed Quarkus extensions are supported
5. Libraries can be upgraded to a current patch level

Properties of the standard distribution

To look at the properties of the default Keycloak distribution, we use the following default Keycloak Docker image: quay.io/keycloak/keycloak:18.0.0.

A Docker container with the image can then be started in the following way:

1docker run --rm -it quay.io/keycloak/keycloak:18.0.0 start \
2   --auto-build \
3   --http-enabled=true \
4   --hostname-strict=false \
5   --hostname-strict-https=false

We use the --auto-build parameter to tell Keycloak to apply build-time configuration.

Activated extensions in the standard image

The preceding command outputs the following list of activated Quarkus extensions (Keycloak features) during the Keycloak server start:

12022-05-07 10:44:39,393 INFO  [io.quarkus] (main) Installed features: 
2[agroal, cdi, hibernate-orm, infinispan-client, jdbc-h2, jdbc-mariadb, 
3jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, 
4narayana-jta, reactive-routes, resteasy, resteasy-jackson, 
5smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, 
6vertx]

We see here that Keycloak enables support for many databases by default: MSSQL, Oracle, MySQL, MariaDB, H2 (old 1.x version with many CVEs). We would like to limit this to a single required database in the further course: PostgreSQL.

Missing extensions in the standard image

Quarkus offers a wide range of functionality that can be activated via Quarkus extensions. A pre-selection has already been made in the Keycloak distribution.

A way to activate these functions has already been asked for in a Keycloak discussion and there was already a solution from the community. The procedure described in the Keycloak Discussion works, but may deter users due to its complexity.

Vulnerabilities found in the standard image

In our example, we use the tool Trivy from Aquasecurity to scan Docker images for known CVEs. You can easily run the tool as a Docker container.

We use a small Java CLI wrapper here to run the Trivy scan:

1java bin/scanImage.java --image-name=quay.io/keycloak/keycloak:18.0.0

Result of the Trivy scan with standard Keycloak Docker image as gist .

1quay.io/keycloak/keycloak:18.0.0 (redhat 8.5)
2=============================================
3Total: 104 (UNKNOWN: 0, LOW: 37, MEDIUM: 65, HIGH: 2, CRITICAL: 0)
4 
5Java (jar)
6==========
7Total: 5 (UNKNOWN: 1, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 3)

Note: These results change over time:

• New vulnerabilities are found
• The general CVE scoring changes due to new findings
• There is a re-release of the Docker image with updated OS components

Building your own Keycloak distribution

To build our own Keycloak distribution with the above-mentioned adaptations, we combine parts of the Keycloak.X server distribution with the Keycloak Quarkus server application, which is also used by the Keycloak project in its own distribution. To do this, we create our own Maven project. Using Maven Dependency Management, we include the Keycloak Quarkus distribution as a .zip archive.
This archive is then unpacked with the maven-dependency-plugin into the target directory, whereby we explicitly exclude the lib directory of the distribution. The next step is to include the keycloak-quarkus-server Maven dependency, which allows us to customize the dependencies of the Keycloak Quarkus Server application.

In order to be able to store further configurations in the generated Keycloak distribution, the content of the src/main/copy-to-keycloak directory is copied over the unpacked Keycloak distribution via the maven-resources-plugin.

We can create our own distribution with the following command:

1mvn clean package

After that, we find our own Keycloak distribution in the directory
target/keycloak-18.0.0, which can already be used.

Adding extensions and themes

This approach also allows the use of custom extensions and themes. In the example, we have used our own event listener provider and a custom theme.

Testing with Keycloak Testcontainers

Our own extensions can be tested automatically with the help of the Keycloak Testcontainers library in the form of integration tests. For the sake of simplicity, we use the standard Keycloak Docker Image for the tests. With a little additional configuration and build orchestration, the previously created custom image could also be used here.

Creating a custom Docker image

Our own Keycloak.X distribution can be brought into one’s own Docker Image in the same way as the standard Keycloak.X Docker Image. In our example, we use the fabric8 Maven Docker Plugin for this.

We then start the Docker Image build using the following command:

1mvn clean package docker:build 
2-Ddocker.image=thomasdarimont/custom-keycloakx:1.0.0-SNAPSHOT

Removing unneeded Quarkus extensions

Keycloak uses numerous libraries that are integrated via Quarkus extensions. Depending on the environment, some of these extensions are not needed, e.g. if only a PostgreSQL database is used, then support for Oracle and other databases is not needed. In this case, Quarkus extensions can be removed via appropriate Maven Dependency Exclusions. For example, if we want to remove support for the Oracle database, we can apply the following exclusions to the org.keycloak:keycloak-quarkus-server:18.0.0 Maven Dependency:

1<dependency>
2        <!-- Keycloak Quarkus Server Libraries-->
3        <groupId>org.keycloak</groupId>
4        <artifactId>keycloak-quarkus-server</artifactId>
5        <version>${keycloak.version}</version>
6        <!-- Exclude unused dependencies -->
7 
8        <exclusions>
9            ...
10            <!-- Exclude unused support for: Oracle -->
11                <exclusion>
12                    <groupId>com.oracle.database.jdbc</groupId>
13                    <artifactId>ojdbc11</artifactId>
14            </exclusion>
15            <exclusion>
16                    <groupId>io.quarkus</groupId>
17                    <artifactId>quarkus-jdbc-oracle</artifactId>
18            </exclusion>
19            <exclusion>
20                    <groupId>io.quarkus</groupId>
21                    <artifactId>quarkus-jdbc-oracle-deployment</artifactId>
22            </exclusion>
23            ...
24        </exclusions>
25    </dependency>

This technique can also be used to remove vulnerable libraries that are not needed. For example, Keycloak currently uses an old 1.x version of the H2 database by default, which is affected by numerous CVEs (note: as soon as Keycloak is updated to the new Quarkus version >2.8.2, H2 will also be upgraded to a new 2.x version without known CVEs). However, if you only use Keycloak with another database like PostgreSQL instead, you can also remove the H2 extension.

In order to remove the support for the H2 database, we can apply the
following Maven Dependency Exclusions:

1<!-- Exclude unused support for: H2 -->
2<!-- Note: by default keycloak uses the h2 database as a database for 
3     auto-build. To remove h2, one needs to configure another Database 
4     in src/main/resources/META-INF/keycloak.conf -->
5      <exclusion>
6         <groupId>com.h2database</groupId>
7         <artifactId>h2</artifactId>
8      </exclusion>
9      <exclusion>
10         <groupId>io.quarkus</groupId>
11         <artifactId>quarkus-jdbc-h2</artifactId>
12      </exclusion>
13      <exclusion>
14         <groupId>io.quarkus</groupId>
15         <artifactId>quarkus-jdbc-h2-deployment</artifactId>
16      </exclusion>

In addition, an entry such as db=postgres must be added to the file
src/main/resources/META-INF/keycloak.conf. You have to add an entry like
db=postgres, otherwise the Keycloak distribution build will complain about the missing H2 library.

Let’s start our distribution created in this way as a Docker container (see below) with the following command:

1docker run --rm -it \
2    -p 8080:8080 \
3    -e KEYCLOAK_ADMIN=keycloak \
4    -e KEYCLOAK_ADMIN_PASSWORD=keycloak \
5    thomasdarimont/custom-keycloakx:1.0.0-SNAPSHOT \
6    start \
7   --auto-build \
8   --http-enabled=true \
9   --http-relative-path=auth \
10   --hostname-strict=false \
11   --hostname-strict-https=false \
12   --db=postgres \
13   --db-url-host=172.17.0.1\
14   --db-url-database=keycloak \
15   --db-username=keycloak \
16   --db-password=keycloak

We see in the container log output that the database extensions that are not needed have disappeared and only jdbc-postgresql remains.

12022-05-07 14:27:09,161 INFO  [io.quarkus] (main) Installed features: 
2[agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, narayana-jta, 
3reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation,
4 smallrye-health, smallrye-metrics, vault, vertx]

Integrating additional Quarkus extensions

This approach also allows us to use new Quarkus extensions for Keycloak.
As an example, we want to enable support for centralized logging using GELF in Keycloak.

To do this, we add the following dependencies to our Maven project:

1<!-- Additional Quarkus Features: Begin -->
2<dependency>
3    <groupId>io.quarkus</groupId>
4    <artifactId>quarkus-logging-gelf</artifactId>
5    <version>${quarkus.version}</version>
6</dependency>
7<dependency>
8    <groupId>io.quarkus</groupId>
9    <artifactId>quarkus-logging-gelf-deployment</artifactId>
10    <version>${quarkus.version}</version>
11</dependency>
12 
13<!-- Additional Quarkus Features: End -->

When now build our custom distribution, the new Quarkus GELF extensions will be recognized and activated accordingly.

These Quarkus-specific extensions can then be configured using the
quarkus.properties file in the conf directory of the Keycloak installation.

An example configuration in quarkus.properties for GELF:

1# Configure log streaming via gelf
2quarkus.log.handler.gelf.enabled=true
3quarkus.log.handler.gelf.host=localhost
4quarkus.log.handler.gelf.port=12201
5quarkus.log.handler.gelf.facility=iam

Let’s start our distribution created in this way as a Docker container again:

1docker run --rm -it \
2    -p 8080:8080 \
3    -e KEYCLOAK_ADMIN=keycloak \
4    -e KEYCLOAK_ADMIN_PASSWORD=keycloak \
5    thomasdarimont/custom-keycloakx:1.0.0-SNAPSHOT \
6    start \
7   --auto-build \
8   --http-enabled=true \
9   --http-relative-path=auth \
10   --hostname-strict=false \
11   --hostname-strict-https=false \
12   --db=postgres \
13   --db-url-host=172.17.0.1\
14   --db-url-database=keycloak \
15   --db-username=keycloak \
16   --db-password=keycloak

We see that the desired logging-gelf extension has been recognized by the Quarkus runtime.

12022-05-07 14:27:09,161 INFO  [io.quarkus] (main) Installed features: 
2[agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, logging-gelf, 
3narayana-jta, reactive-routes, resteasy, resteasy-jackson, 
4smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, 
5vertx]

Patching used libraries

As already mentioned, CVEs are known for some Java libraries used by the current Keycloak distribution. Compatible patch versions already exist for some of these libraries. With the approach shown, these libraries can be easily updated via Maven Dependency Management. The new dependency versions are then updated accordingly when the dependencies are resolved in the build of the own Keycloak distribution and raised to the latest (compatible) patch level.

The latest available Keycloak release 18.0.0 contains several vulnerable libraries, for example a version of the XStream library (1.4.18) which we can update with a managed Maven dependency override:

1<dependencyManagement>
2  <dependencies>
3<!-- CVE Patch overrides: Begin -->
4     <dependency>
5        <groupId>com.thoughtworks.xstream</groupId>
6        <artifactId>xstream</artifactId>
7        <version>1.4.19</version>
8     </dependency>
9 
10   </dependencies>
11</dependencyManagement>

Note: In our example on GitHub, we were able to successfully mitigate all CVEs through dependency upgrades.

Note: Since each new Keycloak version is accompanied by new versions of libraries, it is recommended to remove the overwritten managed dependencies after upgrading the Keycloak version and to run a new image scan. After a new image scan, you may receive a new list of vulnerable libraries that you can then patch again in the way shown.

Security vulnerabilities found in base image

Through appropriate dependency upgrades and dependency exclusions, we can bring all Java libraries to a currently secure state.
No more CVEs are reported for Java libraries. However, the ubi8-minimal Docker image still contains vulnerable components.

We can perform the Docker image scan with the following command:

1java bin/scanImage.java 
2--image-name=thomasdarimont/custom-keycloakx:1.0.0-SNAPSHOT

Result of the Trivy scan with custom ubi8-minimal image in a gist .

1thomasdarimont/custom-keycloakx:1.0.0-SNAPSHOT (redhat 8.5)
2===========================================================
3Total: 104 (UNKNOWN: 0, LOW: 37, MEDIUM: 65, HIGH: 2, CRITICAL: 0)
4 
5Java (jar)
6==========
7Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

If we also want to get rid of the reported CVEs from the base image, then one possibility is to exchange the base image for one without CVEs, for example image based on Alpine. According to Trivy scan, the image alpine:3.15.4 currently does not contain any known CVEs. Using the following command, we can build an Alpine based Docker image:

1mvn clean package docker:build -Ddocker.file=keycloak/Dockerfile.alpine

A new scan of the new Docker image with Trivy then delivers pleasing results: 0 CVEs \o/.

1java bin/scanImage.java --image-name=thomasdarimont/custom-keycloakx:1.0.0-SNAPSHOT

Result of the Trivy scan with Alpine Docker image as gist .

1thomasdarimont/custom-keycloakx:1.0.0-SNAPSHOT (alpine 3.15.4)
2==============================================================
3Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)
4 
5Java (jar)
6==========
7Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

Summary

In this article we have presented an approach for creating your own Keycloak distributions. This approach makes it possible to simply deliver your own extensions and themes instead of doing this, for example, during deployment at runtime. Furthermore, Quarkus extensions that are not needed can be removed and new Quarkus extensions can be added to Keycloak.

Another customization option is fine-grained upgrades of libraries without known security vulnerabilities. By additionally using a different Docker base image, we were able to create a Docker image with a Keycloak distribution that does not contain any known CVEs.

Besides the higher security due to a reduced attack surface, the footprint is again improved due to the reduced amount of extensions.

This approach allows dynamic packaging of Keycloak distributions according to one’s own requirements. It would be desirable for the Keycloak project to support this or a similar approach out of the box to enable more secure and streamlined Keycloak installations.

The example for creating your own Keycloak distribution can be found here on GitHub .
In the branch keycloak-custom-server/zero-cves you will find the version of the example we used for the scans.

Disclaimer

Keycloak is a complex open-source software product that relies on a large number of libraries. Unfortunately, this does not prevent CVEs from being discovered – but that is the case with every larger software project. We are very happy about our achievement: producing a custom Keycloak distribution without any known security vulnerabilities*. Other approaches like a searching / replacing / deleting vulnerable libraries had the same goal, but always felt quite fragile. We’re looking forward to your feedback.

*) Zero CVEs refers to the result of an image scan with Aquasec/Trivy and is a snapshot at the time of the experiment. A scan with other tools, e.g. Docker Scan , at another time could reveal new CVEs if new CVEs become known in the meantime. We recommend performing continuous vulnerability scans of generated artifacts such as custom libraries and Docker images.