Charge your APIs Volume 25: Contract Testing

I feel the way we do integration testing is sort of like setting your house on fire to test your smoke alarm. It is excessive, tiresome and way too costly.

This is not a quote from myself. I typically don't come up with such good ideas when I need them. I stumbled upon it during my research on this topic, and – for me – it perfectly illustrates the issue with API testing.

APIs typically undergo testing more or less late in the overall development process and compared to unit tests, these integration and end-to-end tests, they are costly. You wonder why? Let's start with a bit of theory upfront to show you why. (If you feel confident with the theory, you can proceed directly to the chapter on contract testing.)

For the german-speaking audience, my colleague Sebastian Tiemann has also written a related article on the topic Warum schlechte APIs teuer sind.

Test Pyramid

An important concept is the so-called Test Pyramid for software tests. It states that different types of tests in the development of applications should be weighted according to their position within the pyramid. Specifically, tests at the lowest level – the unit tests – should contribute the largest proportion of all tests. Accordingly, each subsequent level should represent a smaller proportion, as higher levels involve greater integration between systems, resulting in increased effort, decreased speed, and therefore higher costs.

There are various interpretations of the Test Pyramid tailored to individual project environments. However, for the understanding of this article, a simple version based on Mike Cohn's concept is sufficient.

Let's define shortly the different levels.

Unit Tests

The bottom level of the pyramid is formed by unit tests. They represent the functionality of the smallest testable units of an application. They are typically executed in isolation from other units. Unit tests are an essential part of modern software development. The advantages are clear. They are usually quickly created or adjusted, can be automated and executed within a short period, and reveal a large portion of errors early on.

Inte⁠gration Tests

While unit tests focus on testing isolated fragments, integration tests aim to verify the correct interaction between multiple, often independently developed, connected components.

End-to-End Tests

End-to-End tests are a comprehensive form of software testing aimed at verifying the entire functionality of an application by simulating the entire process from start to finish. Unlike integration tests, which focus on the interaction between modules, end-to-end tests evaluate the behavior of the application in a realistic environment. These tests simulate typical user interactions on the UI to ensure that all components and interfaces function correctly, and the application as a whole delivers the expected results.

What's the problem?

So, in theory, we now know what an ideal testing world should look like. However, in reality, the pyramid is often, quite literally, turned upside down. Time and resources are limiting factors and tempt both developers and project managers to downgrade unit tests to something like "nice to have" and push them to an unspecified future date under the motto of "technical debt". And even if we have a broad base of unit tests, the human pursuit of security leads to a significantly higher number of integration and/or end-to-end tests being defined and created – because better safe than sorry, and there's also the good feeling of having tested everything with systems that are as close to production as possible.

At this point, the pain typically begins, which may not be noticeable at first, especially in new projects where you're only dealing with a small number of interfaces. At this stage, it may still feel like we have a complete overview of everything. However, even these integration tests already assume that the systems to be tested are available in appropriate test environments and ideally kept up to date with the latest, mutually compatible versions. If teams start to drift apart due to delays or differences in their overall development speed, this can potentially lead to incompatibilities between the test environments. This may or probably will slow down the working speed of other teams and create a backlog.

But not only the working speed of the teams can be a problem. Integration or end-to-end tests are comparatively slow because each request must pass through all the involved systems, and preparing the tests, such as provisioning the necessary data foundations, can also require a big amount of time.

As we continue to create more and more test scenarios for integration and end-to-end tests, we end up with a considerable mountain of tests which we are naturally very proud of because all these tests give us even more confidence. However, unfortunately, our project leader comes up with some "minor adjustments" that require us to revise all tests. While this may not be a problem for a few tests, the effort increases not linearly but exponentially as the number of tests grows larger.

Searching for errors in the integration and end-to-end testing environment can also be challenging, as it's not always immediately apparent whether a flaw is due to one's own shortcomings or an issue in another involved system. This can lead to numerous questions. Have we used the correct test data in the appropriate environment? Has the version of one of the involved services changed, or has an error been built into them? So you enter into a chat with the tester, potentially discussing what actions were taken to reproduce the error, and you begin looking into the log data. In the best-case scenario, if one can even calls it that, it's discovered that an error in our own code led to the undesired result, and the tester's time was well spent. In the worst-case scenario, one receives a cryptic error message from one of the invoked external systems and has no idea what actually went wrong. Thus, the investigation continues, and further individuals must be involved. The entire process has now consumed a significant amount of time and likely created frustration for all people involved, despite the initial intention just to ensure that the application integrates into the overall system as expected. Just as with the effort for tests, it's worth noting that the risk increases more significantly with rising complexity when it comes to errors.

In summary, it can be concluded that there are many factors contributing to the challenges associated with integration and end-to-end testing, which can have a negative impact on our development cycle. And it is precisely at this point that contract testing comes into play.

Contract Testing

Contract testing, like the previously mentioned unit, integration and end-to-end testing, is a method in software testing. Interestingly, the concept of contract testing is not new, but it has seen a noticeable surge in popularity lately.

In this method, interfaces are tested based on contracts. A contract represents an agreement between a consumer and a provider (not always both as we will see soon). It typically includes input and output parameters, formats, errors and protocols that the interacting parties must adhere to for the test to be positively validated.

Once contract testing is setup you are able to test solely against the defined contracts and you do not depend on any implementations nor have to setup any environments for testing while having the security to always align to the definition. This leads to a Shift-Left because tests which have been created and executed in late stages before may now executed early in the process.

Out there in the contract testing universe exist multiple different strategies and methodologies.

Strategies

Regarding strategies, contract testing classically consists of the consumer-driven and the provider-driven approach. The main difference is: who is setting the expectations into the contract or in modern language: who is making the rules.

In consumer-driven contract testing the consumer of a service describes in the contract for every provider what the interface should look like, resulting in perfectly tailored solutions without overhead.

In provider-driven contract testing typically presents the whole repository of interfaces of the provider in a contract.

A third strategy, perhaps slightly divergent from the other two, is the provider contract test mentioned in Matt Fellows's article on schema-based contract testing, which ensures that an API provider aligns with its own published schemas and does not rely on any dependencies to other teams.

Methodologies

In terms of methodologies, code-based contract testing involves validating contracts at a code level. This means that actual code needs to be executed on both the consumer and provider sides for this approach to work. Its brings multiple advantages into play:

• Code-based contract tests ensure that the code and contracts remain aligned, minimizing the risk of divergence between them
• Tests generated within the code reflect real-world message exchanges, enhancing the authenticity and reliability of the testing process
• Offers valuable insights into the inner workings of the implementation, aiding in understanding and troubleshooting
• The detailed nature of code-based contract tests increases the likelihood of detecting bugs related to interfaces
• Once established, code-based contract tests provide a robust foundation that teams can depend on during subsequent code modifications

In contrast, schema-based contract testing verifies that sent messages adhere to a given schema. It brings multiple advantages compared to code-based contract testing:

• Many API developers are already used to schema validation, which reduces the learning curve and therefore allowing them to adapt quickly
• Offers a short ramp-up time that enables teams to start testing quickly
• They typically require less maintenance over time, as they focus primarily on validating schema differences rather than code implementations. This also results in swift test runs
• Removes the complexity of creating test data
• The method of creating schemas doesn't matter as long as the schema remains valid. There's a wide array of tools available to support developers in creating and managing schemas, offering flexibility and convenience
• Schema-based testing allows for both black-box and white-box testing approaches fostering a broader base of test authors

Sounds good? I think so! But nevertheless there are some disadvantages when using schema-based contract testing compared to the code-based apporach.

• Schemas may still require manual creation or at least manual review to ensure accuracy and completeness
• Incomplete or inaccurate schema declarations may lead to uncertainties, for example regarding the expected codes and attributes in specific situations
• Just because a definition exists in a schema doesn't guarantee that it's implemented in the same way within the system
• The effectiveness of schema-based validation hinges on the quality of the schema itself. Poorly written schemas may provide a false sense of security
• Automatically generating code from schemas may introduce overhead by including information that the actual process doesn't actually require

What's better now?

It depends! If you prioritize comprehensive testing with a very high guarantee on perfectly working interfaces, consumer-driven contract testing could be your choice. But keep it mind it may contain a lot of obstacles to overcome. You have to convince all parties to accept and work with this concept because it doesn't make sense to start without these agreements.

On the other hand, the schema-based contract testing approach provides a swift start without a steep learning curve and you are not dependent on any other parties while still providing you great confidence on having stable APIs.

The "new" Pyramid

The introduction of contract testing has "altered" the testing pyramid adding contract tests as a new layer right above the unit tests.

Since this new layer has the capability to fetch most interface related issues early on it reduces the effort we have to spent on the following layers. Those tests now can focus on important business cases instead of interface-related topics and this reduces the overall complexity. The graph which has been grown exponentially before now has a linear growth regarding effort for adjustments and risk. (Of course we are talking about a perfect world scenario with linear growth, but I hope the intention becomes clear)

Possible Complications

As already said, code-based contract testing presents a significant hurdle: convincing all parties involved of its value as an extension to your usual testing stack. This is because it requires an upfront definition of contracts and the establishment of the technical base. It's crucial to persuade people of its long-term benefits. This can be challenging, particularly with project managers or executives who may only see short-term additional effort and tend to prioritise tasks which return an immediate success.

Meanwhile schema-based contract testing might be the approach most projects can way easier start working with.

Tools

There are multiple tools on the market regarding contract testing. A good starting point for code-based solutions are Pact or Spring Cloud Contract.

For the schema-based approach, Portman is a great choice.

While you can run all tests locally with these tools, they unfold their full potential when integrated into a CI/CD pipeline.

Conclusion

Contract testing is a valuable addition to your usual testing repository. It reduces the risk of late findings of interface errors and supports a fast deployment lifecycle by pushing the confidence of the team. Ultimately it will save time and therefore money – or at least the time usually used for fixing these unnecessary errors that can be spent in a better way with tasks that create business value!

If you want to give it a try but don't want to spent a long time on learning and ramping up, schema-based contract testing is a great way to start optimizing your API testing capabilities.

References

• https://www.amazon.de/-/en/Mike-Cohn/dp/0321579364 (Book)
• https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
• https://martinfowler.com/bliki/ContractTest.html
• https://pactflow.io/blog/contract-testing-using-json-schemas-and-open-api-part-1/
• https://pactflow.io/blog/contract-testing-vs-integration-testing/
• https://pactflow.io/what-is-contract-testing-page/
• https://martinfowler.com/articles/practical-test-pyramid.html#ContractTests