Migrating a Spring Boot application to Java 17 – the hard way: Day 2

Welcome back to my article on migrating a Spring Boot application to Java 17 – the hard way.

On day 1 we:

In this post we will cover

• Spring Cloud
• Spring Data
• Spring Kafka

Day two

We’re off to a good start, but we aren’t done yet. Let’s recompile everything and see where we stand:

1$ mvn clean verify
2
3[ERROR] java.lang.IllegalStateException: Failed to load ApplicationContext
4Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configurationPropertiesBeans' defined in class path resource [org/springframework/cloud/autoconfigure/ConfigurationPropertiesRebinderAutoConfiguration.class]: Post-processing of merged bean definition failed;
5Caused by: java.lang.ClassNotFoundException: org.springframework.boot.context.properties.ConfigurationBeanFactoryMetadata

Looks like we have an issue with one of our integration tests, so let’s dig into Spring Cloud.

Spring Cloud

Spring Cloud provides a number of tools for developing distributed systems running in the cloud. In our project, we use two modules; Spring Cloud Kubernetes and Spring Cloud Netflix .

We are currently using Spring Cloud Hoxton , specifically the Hoxton.RELEASE version.

According to the compatibility matrix , Hoxton does not support Spring Boot 2.5.x. We need to upgrade to at least Spring Cloud 2020.0.3 (notice the new version scheme being used here too).

Searching through GitHub, the class org.springframework.boot.context.properties.ConfigurationBeanFactoryMetadata was removed in 2.4 .

Let’s go ahead and update our Spring Cloud version to 2020.0.4 (the latest fix version as of writing this article).

1<project>
2  <properties>
3    <spring-cloud.version>2020.0.4</spring-cloud.version>
4  </properties>
5
6  <dependencyManagement>
7    <dependencies>
8      <dependency>
9        <groupId>org.springframework.cloud</groupId>
10        <artifactId>spring-cloud-dependencies</artifactId>
11        <version>${spring-cloud.version}</version>
12        <type>pom</type>
13        <scope>import</scope>
14      </dependency>
15    </dependencies>
16  </dependencyManagement>
17</project>

After upgrading, my IDE reports that a dependency we use can no longer be resolved. We’ll look into that below.

Release notes

For reference, here are the release notes for Spring Cloud 2020.0 for all fix versions.

Spring Cloud Kubernetes

Spring Cloud Kubernetes helps developers run applications on Kubernetes. Although it has a number of cool features, we use its externalised configuration support.

Our application configuration – you know, the application.properties|yml that configures your Spring Boot application – is stored in a k8s ConfigMap, and Spring Cloud Kubernetes makes that external configuration available to the application during startup.

Getting back to the code, our IDE complains that the spring-cloud-starter-kubernetes-config dependency cannot be resolved.

1<dependency>
2  <groupId>org.springframework.cloud</groupId>
3  <artifactId>spring-cloud-starter-kubernetes-config</artifactId>
4</dependency>

According to the release notes , 2020.0 introduced a restructuring of the existing spring-cloud-kubernetes modules and introduced a second client based on the official Kubernetes Java Client . The existing fabric8 implementation was renamed (to make it clear which client is being used).

Users of Spring Cloud Kubernetes can now choose between two implementations:

1. the renamed fabric8 starters , or
2. the new Kubernetes Java Client

I looked for guidance when to use one over the other, but didn’t find anything in the documentation, only the release notes. I found this blog post by Rohan Kumar who wrote up a pretty good comparison of the two. Be sure to check out his blog for some very good posts on k8s.

What comes next represents only my experience and lessons learned. You may have different experiences, and I’d love to hear from you about them.

First attempt – using the new client

Let’s use the new official Kubernetes Java Client, switching from the existing fabric8 client. Nothing against the fabric8 client, I just prefer to use official-looking things. Besides, we don’t need any features that only the fabric8 client provides.

I removed the spring-cloud-starter-kubernetes-config dependency and added this new one:

1<dependency>
2  <groupId>org.springframework.cloud</groupId>
3  <artifactId>spring-cloud-starter-kubernetes-client-config</artifactId>
4</dependency>

At first everything looked promising. Locally, the project compiled and the unit/integration tests were green. Fantastic, I thought, that was easy. Too easy, it turns out.

Then came Jenkins

I committed my changes in a branch and pushed to Bitbucket. I’ll admit, I’m a big fan of feature branches and proud of it. I know some of my colleagues are going to give me heck for that (looking at you Thomas Traude ). A few minutes later I received a notification that my Jenkins build was red.

1[ERROR] java.lang.IllegalStateException: Failed to load ApplicationContext
2Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'kubernetesKubectlCreateProcessor': Unsatisfied dependency expressed through field 'apiClient'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultApiClient' defined in class path resource
3
4Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultApiClient' defined in class path resource
5
6Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.kubernetes.client.openapi.ApiClient]: Factory method 'defaultApiClient' threw exception; nested exception is java.io.FileNotFoundException: . (Is a directory)
7Caused by: java.io.FileNotFoundException: . (Is a directory)

But it builds on my machine!

Looks like we have some flaky tests. Depending on the environment, the application context may fail to load. This is the very definition of frustrating, but don’t worry, I enjoy these kinds of challenges.

In case you’re asking yourself why the tests fail if the builds run in Kubernetes, that’s because they don’t. Our Jenkins jobs don’t run in Kubernetes, since we make extensive use of Testcontainers . If you don’t use them, be sure to check them out, awesome. And their new cloud solution looks very promising.

Disabling Spring Cloud Kubernetes in tests

Spring Cloud Kubernetes can be disabled in tests using the property spring.cloud.kubernetes.enabled. Drop that property into your tests like so, and you’re good to go (or at least it used to work).

1@ActiveProfiles("test")
2@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
3        properties = {"spring.cloud.kubernetes.enabled=false"})
4class ApplicationIT {
5}

I didn’t understand the issue at first, it should have been disabled. We’ve been using the new Kubernetes Java Client successfully in other projects, and there the tests aren’t flaky. I took another look, turns out that our projects are using different versions. Version 2020.0.1 works as expected.

A change introducing additional configuration properties in the Kubernetes Java Client had an unintentional side-effect; the property spring.cloud.kubernetes.enabled no longer works as expected. There is no longer a single property to disable Spring Cloud Kubernetes.

Issues have been reported here and here , with the fixes scheduled for 2020.0.5. Unfortunately, as of writing this article, version 2020.0.5 hasn’t been released. This enhancement was included in Spring Cloud 2020.0.2, which explains why the version 2020.0.1 worked in other projects.

According to the documentation, these new features can be disabled.

And note that you can disable the configuration beans by setting the following properties in your Spring context:

kubernetes.informer.enabled=false # disables informer injection
kubernetes.reconciler.enabled=false # disables reconciler injection

What to do? Instead of disabling those additional properties in our tests, I opted for another solution.

Second attempt – using the existing fabric8 client

Let’s switch back to the fabric8 client. Replace the spring-cloud-starter-kubernetes-client-config dependency for this one:

1<dependency>
2  <groupId>org.springframework.cloud</groupId>
3  <artifactId>spring-cloud-starter-kubernetes-fabric8-config</artifactId>
4</dependency>

Locally the build is green. Push to Jenkins, and wait. Crossing my fingers always helps, so that’s what I did. And what do you know, worked like a charm; no issues, nothing, zero, zip. I love when things just work.

I should have known better. The fabric8 client has been serving us well for many years. Don’t mess with a running system!

Lessons learned updating Spring Cloud Kubernetes

It would appear the Spring Cloud Kubernetes Java Client is not yet ready. The Kubernetes Java Client introduced their own Spring integration which doesn’t integrate with the Spring configuration properly. Hopefully the two projects are cooperating, and we’ll get a nice clean Spring-based configuration in the future. Once version 2020.0.5 is released, I’ll give it another try.

However, this does raise an important topic of mine; trust and confidence in the libraries we depend on and software we deliver.

On the one hand, this project performed a major update of Spring Cloud, so I expect things to break. On the other hand, considering the change occurred in a fix release, I wouldn’t have expected this to occur. And that raises concerns for me. Since there was an easy work-around, why bother mentioning this at all? I feel it’s important to discuss and give our feedback. When changes like this occur in fix releases, it can damage trust and erode confidence. Especially when users expect a different behavior.

According to Spring’s own statement the release trains follow calendar versioning (just learned this about this myself), while the projects use semantic versioning .

Given a version number MAJOR.MINOR.PATCH, increment the:
1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards compatible manner, and
3. PATCH version when you make backwards compatible bug fixes.

I interpret that as a commitment to try and avoid situations like above. You may interpret that differently. I also understand that s**t happens. In situations like this, I’m reminded of the old saying don’t touch running software. In the world of cloud, we have to be ready and able to update our software whenever needed. And therein lies the challenge. Why Spring Cloud Kubernetes bumped the Kubernetes client to 11.0.0 in a fix release, I don’t know.

We have a solution which works, so let’s move onto the next Spring Cloud project.

Spring Cloud Netflix

Spring Cloud Netflix is a collection of widely popular and successful OSS projects, donated by Netflix to Spring.

Spring Cloud Netflix Zuul

Our API gateway application uses Spring Cloud Netflix Zuul to provide routing to backend systems, along with authentication and authorization services using OpenID Connect.

It turns out, Zuul entered maintenance mode back in 2018 and was removed from spring-cloud-netflix in this release. It is superseded by Spring Cloud Gateway .

Migrating from Zuul to Spring Cloud Gateway is going to take longer than a day. We decided to leave this migration for another day, so we can get a running system by the end of this day. To do that, we refactored the POMs, so our API Gateway application remains on Java 11 and continues to use the 2.3.3.RELEASE Spring Boot version. Remember, we didn’t set out to upgrade Spring Boot, but enable Java 17. If Zuul cannot be used with Java 17, then so be it.

Hopefully we can cover this in a separate blog post in the future. We will have to migrate Zuul soon, since it’s EOL.

We’ve now completed the Spring Cloud upgrade, let’s move on to the next Spring module in our project.

Spring Data

Spring Data is a collection of projects, providing data access in the familiar Spring-based way.

As stated in the release notes , Spring Boot 2.5.x updated to Spring Data 2021.0 . Specifically, Spring Boot 2.5.7 updated to Spring Data 2021.0.7.

There’s no need to import a BOM, the spring-boot-starter-parent manages the Spring Data dependencies for us.

Release notes

For reference, here are the release notes for Spring Data 2021.0 . They don’t contain much information, but the blog article “What’s new in Spring Data 2010.0 ” does provide a decent overview.

Spring Data Rest

Our application uses Spring Data Rest to expose JPA entities as REST APIs . That’s right, simply define your JPA entities, mark the Repository and voilà, you have a simple CRUD application up and running in less than 5 minutes.

1@RepositoryRestResource(path = "entities")
2public interface EntitiesRepository extends PagingAndSortingRepository<MyEntity, String> {
3}

Unfortunately, upgrading wasn’t as fast. Compiling our application, we get the following error:

1$ mvn clean verify
2
3[ERROR] /../src/main/java/de/app/config/CustomRepositoryRestConfigurer.java:[12,5] method does not override or implement a method from a supertype

The following class no longer compiles:

1@Component
2public class CustomRepositoryRestConfigurer implements RepositoryRestConfigurer {
3
4    @Override
5    public void configureRepositoryRestConfiguration(RepositoryRestConfiguration config) {
6        config.exposeIdsFor(MyEntity.class);
7    }
8}

Looks like the RepositoryRestConfigurer interface changed. I tried to track down some release notes for this, without any luck (the Spring Data release notes are not particularly thorough).

Looking at the code on GitHub, the method was deprecated in 3.4 M2 (2020.0.0) and removed in 3.5 M1 (2021.0.0) . As we skipped Spring Boot 2.4.x, we never saw the deprecation notice in Spring Data 2020.0.x. Otherwise, we could have migrated our code before it was removed. Another example of why it’s better to update frequently.

The fix is easy, CorsRegistry was added to the configureRepositoryRestConfiguration method. Our class now looks like this:

1@Component
2public class CustomRepositoryRestConfigurer implements RepositoryRestConfigurer {
3
4    @Override
5    public void configureRepositoryRestConfiguration(RepositoryRestConfiguration config, CorsRegistry cors) {
6        config.exposeIdsFor(MyEntity.class);
7    }
8}

Our code now compiles, but we have some failing tests.

The repository rest controllers

Some of our tests fail with the following error:

1[ERROR] java.lang.IllegalStateException: Failed to load ApplicationContext
2Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'restHandlerMapping' defined in class path resource [org/springframework/data/rest/webmvc/config/RepositoryRestMvcConfiguration.class]:
3Caused by: java.lang.IllegalStateException: Spring Data REST controller de.app.EntitiesRestController$$EnhancerBySpringCGLIB$$bcf6b665 must not use @RequestMapping on class level as this would cause double registration with Spring MVC!

Something else changed in Spring Data Rest. Again, I found nothing in the release notes, but tracked down the commit “Prevent duplicate controller registrations through class-level @RequestMapping” which changed the behavior.

When we detected @BasePathAwareController and @RepositoryRestController instances, we now reject types that use @RequestMapping on the class level as doing so causes an inevitable registration of the controller with Spring MVC.

Turns out we’ve been doing just this:

1@RepositoryRestController
2@RequestMapping("/entities")
3@Validated
4public interface EntitiesRestController {
5    @GetMapping(value = "/{id}", produces = APPLICATION_JSON)
6    ResponseEntity<MyEntity> getObject(@PathVariable("id") final String id);
7}

We customize the rest data endpoints using a @RepositoryRestController. This is still possible, but the code must be adapted. The @RequestMapping annotation on the class must be removed, and the path added to each method. Luckily our API only has a few methods, but I can imagine this is frustrating for larger APIs.

1@RepositoryRestController
2@Validated
3public interface EntitiesRestController {
4    @GetMapping(value = "/entities/{id}", produces = APPLICATION_JSON)
5    ResponseEntity<MyEntity> getObject(@PathVariable("id") final String id);
6}

I haven’t verified the behavior in our existing application, but I interpret the issue so. With the previous handling, “our application would actually have 2 rest endpoints, one served by Spring Data Rest another by Spring MVC”. But like I said, I haven’t verified this.

After making that change, those tests are green, but we now have another issue.

Customizing the repository Rest controller media type

Another batch of tests are now failing after this change. In some cases, the default Spring Data Rest endpoints had been customized and no longer match, so we receive either 404 or 405 errors. It seems the customized endpoints have to match the default Spring Data Rest endpoints enough, otherwise they’re not recognized.

I assume it used to work, because of the @RequestMapping(“/entities”) annotation, which got picked up by Spring MVC and treated as a regular endpoint. However, I haven’t been able to verify, and I will update this article if and when I have more information.

By default, Spring Data Rest endpoints use a different content type application/hal+json. By changing the Repository Rest API configuration, the default media type can be changed to application/json and “most” of our tests started passing again.

Remember the CustomRepositoryRestConfigurer class from above? Let’s add some additional configuration:

1@Component
2public class CustomRepositoryRestConfigurer implements RepositoryRestConfigurer {
3
4    @Override
5    public void configureRepositoryRestConfiguration(RepositoryRestConfiguration config, CorsRegistry cors) {
6        config.exposeIdsFor(MyEntity.class);
7        config.setDefaultMediaType(MediaType.APPLICATION_JSON);
8        config.useHalAsDefaultJsonMediaType(false);
9    }
10}

That fixes some of the test cases, but not all of them.

Versioning the Repository Rest controller endpoints

Unfortunately we ran into an issue with our versioned Repository Rest controllers. We attempt to version the API using different media types, e.g. application/json for version 1, and application/vnd.app.v2+json for version 2.

FYI – Spring Boot Actuator supports versioning like this; application/json, application/vnd.spring-boot.actuator.v2+json and application/vnd.spring-boot.actuator.v3+json.

Some of our tests fail with this error:

12021-11-26 11:19:32.165 DEBUG 60607 --- [main] o.s.t.web.servlet.TestDispatcherServlet  : GET "/entities/1", parameters=\{\}
22021-11-26 11:19:32.173 DEBUG 60607 --- [main] o.s.d.r.w.RepositoryRestHandlerMapping   : Mapped to org.springframework.data.rest.webmvc.RepositoryEntityController#getItemResource(RootResourceInformation, Serializable, PersistentEntityResourceAssembler, HttpHeaders)
3
42021-11-26 11:19:32.177 DEBUG 60607 --- [main] o.j.s.OpenEntityManagerInViewInterceptor : Opening JPA EntityManager in OpenEntityManagerInViewInterceptor
52021-11-26 11:19:32.199 DEBUG 60607 --- [main] .m.m.a.ExceptionHandlerExceptionResolver : Using @ExceptionHandler org.springframework.data.rest.webmvc.RepositoryRestExceptionHandler#handle(HttpRequestMethodNotSupportedException)
6
72021-11-26 11:19:32.208 DEBUG 60607 --- [main] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/json', given [/] and supported [application/json, application/\*\+json]
8
92021-11-26 11:19:32.208 DEBUG 60607 --- [main] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Nothing to write: null body
10
112021-11-26 11:19:32.209 DEBUG 60607 --- [main] .m.m.a.ExceptionHandlerExceptionResolver : Resolved [org.springframework.web.HttpRequestMethodNotSupportedException: Request method 'GET' not supported]

This worked with Spring Boot 2.3.3-RELEASE, and I can only assume so because it was handled by Spring WebMVC, not Spring Data Rest. We never found a solution for this using Spring Data Rest, so we refactored the API into a Spring WebMVC Rest Endpoint. If anyone reading this knows how to achieve this using Spring Data Rest, please contact me, I would love to learn how.

That being said, it may not make sense to even do this. I cannot ask the developers why it was done this way, they’re not here anymore. This project’s story can only be told through its Git history.

Lessons learned updating Spring Data Rest

Updating Spring Data Rest wasn’t easy, but that had little to do with Spring Data Rest itself. I suspect we’re using Spring Data Rest wrong, incorrectly mixing WebMVC concepts. If we hadn’t done this from the beginning, things would have run much smoother.

We are now done with the Spring Data Rest migration. It’s time to move onto our next Spring module, Spring Kafka.

Spring Kafka

Spring Kafka, or rather Spring for Apache Kafka , is a great way to use Kafka in your Spring projects. It provides easy-to-use templates for sending messages and typical Spring annotations for consuming messages.

We use Kafka for communication between our applications.

Configuring the consumers

Running our Kafka test cases, we get the following error:

1[ERROR] java.lang.IllegalStateException: Failed to load ApplicationContext
2
3Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'consumerFactory' defined in class path resource [de/app/config/KafkaConsumerConfig.class]:
4
5Caused by: java.lang.NullPointerException
6at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
7at java.base/java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852)
8at org.springframework.kafka.core.DefaultKafkaConsumerFactory.<init>(DefaultKafkaConsumerFactory.java:125)
9at org.springframework.kafka.core.DefaultKafkaConsumerFactory.<init>(DefaultKafkaConsumerFactory.java:98)
10at de.app.config.KafkaConsumerConfig.consumerFactory(AbstractKafkaConsumerConfig.java:120)

It turns out, we had been configuring the consumerConfigs bean and setting null values in its properties. The following change from HashMap to ConcurrentHashMap means we can no longer configure null values. We refactored our code and now tests are green. Easy-peasy.

Kafka messages with JsonFilter

Another test case was failing with this error:

1[ERROR] org.apache.kafka.common.errors.SerializationException: Can't serialize data [Event [payload=MyClass(Id=201000000041600097, ...] for topic [my-topic]
2
3Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot resolve PropertyFilter with id ‘myclassFilter'; no FilterProvider configured (through reference chain: de.test.Event["payload"])
4at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)

Some of our Java Beans use a @JsonFilter to manipulate the serialization and deserialization. This requires a propertyFilter to be configured on the ObjectMapper.

Spring for Apache Kafka made a change to the JsonSerializer , introducing an ObjectWriter . When the ObjectWriter instance is created, the ObjectMapper configuration is copied, not referenced. Our test case was re-configuring the ObjectMapper with the appropriate propertyFilter after the ObjectWriter instance was created. Hence, the ObjectWriter didn’t know anything about the propertyFilter (since the configuration was already copied). After some refactoring, changing how we create and configure the JsonSerializer, our test cases were green.

Running our build $ mvn clean verify finally resulted in a green build. Everything is working as it should. We pushed our changes to Bitbucket and everything built like a charm.

Lessons learned updating Spring Kafka

Updating Spring Kafka was very easy and straighforward. Wish everything was this easy.

Lessons learned during Spring Boot upgrade

Spring and Spring Boot do a great job documenting their releases, their release notes are well maintained. That being said, upgrading was challenging, it took quite a while before everything was working again. A big part of that is on us, for not following best practices, guidelines, etc. A lot of this code was written when the team was just starting out with Spring and Spring Boot. Code evolves over time, without refactoring and applying those latest practices. Eventually that catches up with you, but we use this as a learning experience and improved things. Our test cases are now significantly better, and we’ll keep a closer eye on them moving forward.

Migrating Spring Boot to Java 17 – Summary

This article chronicled our migration story and may or may not represent yours. Depending on the Spring Boot version you’re coming from, the features you use and the Spring modules you integrate in your applications, your migration will look very different.

In the end, migrating the application to Java 17 was a matter of updating our Spring Boot version. I’m sure this wasn’t a surprise to everyone, but this article was about the hard way, not the easy way.

It is as simple, and as hard, as keeping our dependencies up to date. We know this is a best practice, yet it still isn’t done. I can completely understand. Before joining codecentric AG, I was in product development for almost 20 years and am fully aware of the competing priorities. If we’ve learned anything over the past week, it’s how dependent and vulnerable we are on OSS. Being able to move fast and update quickly is so important.

We should get comfortable updating our applications continuously and the Spring Boot version at least every six months. The update process is smoother when moving from one version to another, without skipping versions. And keep in mind, Spring Boot versions are supported for about one year before reaching EOL.

Thankfully there are tools to automate this process, such as Dependabot , Renovate , Snyk . These tools automatically scan your dependencies, continuously look for new versions and create pull-requests when a new version is available. If you use GitHub, you’re most likely already familiar with Dependabot .

Keep an eye out for a future post where I will provide some migration tips and tricks.

[Update] December 28, 2021: Fixed a minor typo and Snyk was spelt incorrectly, my apologies.