Charge your APIs Volume 33 - Definition-Based API Mocking, Simulation, and Testing with Microcks - Part 3: Contract Testing and Consistent Mock Environments

Key Takeaways

• This article is the third and last one in a three-part series about definition-based API mocking, simulation, and testing with Microcks (make sure you have read the first and second article)
• The previous articles focused on (i) Microcks’ architecture; (ii) its capabilities in automated mock derivation and provisioning; (iii) mock data specification; and (iv) subsequent mock refinement. The present article goes into more advanced core features of Microcks, i.e., mock-guided contract testing and interaction with Microcks’ own API for the provisioning of consistent mock environments.
• Contract testing verifies the conformance between API definition and implementation, and can be automated with Microcks to further increase the value of mocks
• The provisioning of consistent mock environments is a prerequisite for mock-based integration testing, and the article shows how to automate the provisioning process and even enable its developer-local execution

Introduction

This article is the last one in a three-part series about definition-based API mocking, simulation, and testing with Microcks. The first article presented Microcks’ architecture and an example API used throughout the article series to illustrate a selection of Microcks’ core features. Its follower article dealt with Microcks’ capabilities for mock data specification and mock refinement using parameters constraints and dispatchers. In the following, mock-guided contract testing with Microcks and the usage of the tool’s own API for the provisioning of consistent mock environments for integration testing are described and exemplified.

Contract Testing

Contract testing verifies that the implementation of an API conforms to the API’s contract. In general, an API contract is an agreement between an API provider and one or more of its consumers. It is typically expressed in the form of a certain definition format, and determines input and output parameters, protocols and data formats, as well as errors. A provider and a consumer successfully conclude an API contract (i) when the provider agrees to adhere to the contract upon exposing the corresponding API’s implementation; and (ii) when the consumer agrees to respect the contract when interacting with the corresponding API’s implementation.

In Microcks, contract testing assumes the following prerequisites:

• Existence of operation mocks, e.g., derived from OpenAPI example objects, for the API whose contract shall be tested.
• Existence of an implementation of that API against which Microcks can run contract tests based on operation mocks.

If both prerequisites are satisfied, contract test execution is feasible from Microcks' web UI by navigating to the mocked API’s details page, which holds a block called "Tests":

The Conformance Index metric indicates how well the contract of a mocked API is covered by sample mock requests and responses, and thus how well an API contract is testable. Microcks calculates the Conformance Index from a variety of characteristics of a mocked API including the number of operation mocks and the complexity of dispatchers. The metric then informs about the maximum possible Conformance Score that is achievable from contract testing using the mocked API. For instance, the example API reaches conformance index A or, in numbers, 87.5% maximum achievable conformance when using its mocks for contract testing. That is because each of its operations has at least one example for mock derivation assigned and its dispatchers aim to prescribe the behavior of potential implementations as detailed as possible. Finally, the concrete contract conformance achieved by an API’s implementation from testing it with the respective mock counterpart in Microcks is communicated by the Conformance Score metric.

Microcks' web UI allows for interactive contract testing as follows:

1. Clicking the "+ New Test" button below the Conformance Score metric
2. In the subsequent dialog, specifying the test endpoint, i.e., the entrypoint URL of the running API implementation against which the test shall be performed
3. In the same dialog, selecting the test runner, which depends on the API type and desired test scope:
   • HTTP: Smoke tests HTTP operations by checking that their response statuses range between 200 and 299, which signal successful execution of API operation implementations. 
   • SOAP_UI: For API implementations that are expected to conform to definitions modeled with SoapUI, checks that test case assertions are met. 
   • POSTMAN: Executes test scripts from Postman Collections for conformance checking of API implementations.
   • OPEN_API_SCHEMA: Sends mock requests derived from OpenAPI example objects to the API implementation under test and verifies that received responses conform to the mock responses derived from the example objects.

Microcks' documentation lists further runners, which seemingly become available in the conformance testing dialog when mocked APIs were uploaded from definition files that follow certain formats such as AsyncAPI.

The OPEN_API_SCHEMA runner is predestined for checking the conformance of an implementation of the example API because the API’s defining "customers.yaml" file is expressed in OpenAPI. However, the runner cannot be executed as-is due to the prerequisites that are imposed on mock requests by the various dispatchers for the API’s Microcks-based mock. For instance, several operations such as GET /customer_kinds expect a certain value in the HTTP header api_key, and apply a Mickrocks parameters constraint to ensure this value is present and valid. For such situations, Microcks supports fine-grained configuration of contract test executions, e.g., from the web UI in the "New Test" dialog:

Subsequent to contract test execution, Microcks' web UI displays a button to add the test to a CI/CD facility such as Jenkins or GitHub Actions. After hitting the button and selecting the desired CI/CD technology, Microcks generates a technology-specific test execution command to be copied into CI/CD automation scripts. Hence, this feature enables to leverage Microcks for continuous API validation, e.g., to detect breaking changes and the resulting degradation of API implementation conformance.

The Maven module "api-implementation-app" in the Git repository of the example API comprises an implementation of the API and therefore allows further examination of Microcks' contract testing capabilities.

Provisioning of Consistent Mock Environments Using Microcks' Own API

Microcks comes with an HTTP-based API that supports automation of mock setup, test execution and reporting before, during, or after application tests are run. The Maven module "integration-tests" in the Git repository of the example API implements several tests that execute against Microcks-based mocks of the API within a test environment that builds upon Microcks' Testcontainers module. For illustrative purposes, the Maven module resorts to Microcks' own API in test environment setup to configure parameters constraints, e.g., the HTTP header api_key for the mock of the GET /customer_kinds operation, and dispatchers, e.g., the SCRIPT dispatcher for the mock of GET /customer. As an alternative to the programmatic usage of Microcks’ own API, such information is declaratively specifiable in secondary artifacts via Microcks' API Metadata format.

In order to configure parameters constraints and dispatchers for mocks of the example API’s operations, the "integration-tests" Maven module interacts with a previously started Microcks testcontainer by means of Microcks' own API during test environment setup. In this step, the module invokes the following operations, which are organized under the mock tag in the OpenAPI definition of Microcks' own API:

• GET /services/{id}: Microcks subsumes mocked APIs under the term "Services" and this operation allows retrieval of information about Microcks Services. The test environment setup in the "integration-tests" Maven module relies on the operation to gain access to Microcks' descriptions of the mocks for the example API’s operations. The module then utilizes these descriptions to install parameters constraints and dispatchers prior to test case execution against the operation mocks.
• PUT /services/{id}/operation: This operation allows modification of operation mocks from their Microcks descriptions. The "integration-tests" Maven module employs this operation to extend the mocks for the example API’s operations with the required parameters constraints and dispatchers. Since the operation follows HTTP PUT semantics,  it overrides existing operation descriptions with given ones. Therefore, information that was derived from API definitions, e.g., inferred dispatchers, must be always incorporated into operation descriptions that are passed to the operation. The curl command for adding the JSON_BODY dispatcher and api_key header constraint to the mock of the POST /login operation via Microcks' own API is as follows:

DISPATCHER_RULES_AS_STRING=$(echo '{
  "exp": "/password",
  "operator": "equals",
  "cases": {
    "123456": "some_customer",
    "default": "invalid_credentials"
  }
}' | jq '. | @json')

curl -X PUT \	
http://localhost:33047/api/services/66b5.../operation?operationName=POST+%2Flogin \
  -H "Content-Type: application/json" \
  --data '{
    "dispatcher": "JSON_BODY",
    "dispatcherRules": '$DISPATCHER_RULES_AS_STRING',
    "parameterConstraints": [{
      "name": "api_key",
      "required": "true",
      "recopy": "false",
      "mustMatchRegexp": "\\QeyJhbGciOiJIUzI1NiJ9....\\E",
      "in": "header"
    }]
  }'

Note that the command assumes the Microcks testcontainer to be running at http://localhost:33047 and the ID of the Microcks Service of the example API retrieved with GET /services/{id} to be “66b5...”.

Discussion

In the following, we discuss potential use cases and further characteristics of Microcks to support reasoning about its applicability in your own projects. However, it should be noted that Microcks is a CNCF sandbox project and as such underlies continuous improvements, which is why it makes sense to frequently consult Microcks' blog to learn about them. For instance, this article series primarily focused on core features of Microcks and omitted advanced, yet powerful, features like stateful mocks from Microcks' recent 1.10.0 release.

In general, projects that are looking for approaches to increase API development speed and quality should have Microcks on their list of tools for further examination. That is because of Microcks' versatility resulting from (i) its support for the most common API definition formats; (ii) the several options for definition import; (iii) features like parameters constraints and dispatchers to bring API mocks to life without no or low code; (iv) eased integration with modern test frameworks and CI/CD pipelines; (v) the possibility to specify contract tests, either in an automated or interactive fashion; and (vi) an open and passionate community with low response times and comparatively quick bug fixes. Furthermore, while Microcks enables automation via its CLI tool and own API, it also comes with a streamlined web UI.

However, due to its versatility, the integration of Microcks also incurs a certain learning curve. For example, to integrate with existing projects spanning several teams and/or roles, one should get familiar with the configuration and operation of Keycloak. Furthermore, the effective application of features like JSON_BODY and especially SCRIPT dispatchers require engagement with additional technologies that are not as widespread as everyday programming languages or frameworks like Java and Spring. Concerning the latter kind of dispatcher, debugging capabilities are currently rather limited and its utilization can counteract Microcks’ focus on no code/low code for even mildly complex APIs with only a few interaction constraints. In addition, Microcks adopters should be aware that the tool sometimes provides several ways to achieve the same outcome. For example, one might employ OpenAPI example objects or Postman Collection examples to describe mock requests and responses. Similarly, Microcks' own API can partially fulfill the same purposes as its Metadata format.

For the more technical subtleties, while Microcks supports the most common API definition formats, it inherently assumes that existing definitions conform to those formats for automatic mock derivation. Additionally, Microcks requires definitions in certain formats to comprise versioning information. Also, OpenAPI-based definitions must be accompanied by Postman Collections. Another Microcks feature that the article series did not discuss are Direct APIs. It enables the interactive creation of API definitions from scratch using Microcks’ web UI, but seems currently rather rudimentary as it only generates basic CRUD operations for one resource. Consequently, APIs that are not resource-driven or require more sophisticated operations cannot immediately be modeled as Direct APIs. Furthermore, the subsequent modification of generated Direct API definitions does not seem possible from Microcks' web UI. In any case, Microcks heavily depends on the quality of provided data, i.e., the quality of API definitions, and, at the time of writing, does itself not provide means to actively assist in improving quality deficiencies in mocked APIs besides their detection by mock tests.

Regarding contract testing, Microcks' capabilities in test failure reporting make a veritable target for future improvement. For instance, technical details about failed tests like a missing header in a mock request are not visible from Microcks’ web UI but only from technical logs, e.g., by issuing the docker logs command against a running Microcks testcontainer. Moreover, URLs that are suffixed by a redundant forward slash (/) can cause contract tests to fail.

Lastly, while there is always a community member and even core developer of Microcks to reach out for support, the documentation itself is still sparse, especially for advanced topics, which is likely due to Microcks' fast development cycles and rapid inclusion of new features from community feedback.

Conclusion

This three-part article series (see the first and second article) presented and discussed some core features of Microcks—a CNCF sandbox project that develops a multi-protocol tool for API mocking, simulation, and testing. Microcks is able to derive and serve mocks from API definitions based OpenAPI or AsyncAPI, among others. It aims to address the concerns of (i) API implementers by enabling comparatively lightweight integration and contract testing; (ii) API owners and quality assurers with API simulation for conformance assessment; and (iii) platform engineers who are in charge of development and test environment provisioning. In order to support the requirements of this rather broad target audience, Microcks provides several features that allow for mock refinement, e.g., constraint specification for message headers and query parameters of mocked API operations, custom dispatchers to influence mock routing, and automated or interactive contract testing of API implementations guided by previously derived API mocks. The article illustrated these features by means of a Git repository comprising (i) the OpenAPI definition of a coherent example API; (ii) the example API’s instrumentation for mock-based integration testing leveraging Microcks’ Testcontainers module and own API; and (iii) an implementation of the example API for experimenting with Microcks’ capabilities in contract testing. To summarize, Microcks is a very versatile and flexible tool for API mocking, simulation, and testing that bears a broad range of innovative features. On the other hand, teams that need to rely on mature and long-proven technology could be put off by the experimental mindset which Microcks yet implies in some areas and which is expectable from a growing CNCF sandbox project.