Contract Testing: Preventing Breaking API Changes Without Slowing Delivery

Key Points

• Your contract tests run at the unit test level using generated mocks, not in a shared integration environment.
• Contract testing reduces the risk of breaking changes in microservice architectures by testing each provider and consumer pair against the contract before either service is released.
• Contract testing accelerates delivery by avoiding tight coupling between consumer and provider teams and reducing reliance on expensive integration or end-to-end testing environments to verify compliance.

Quick Answer

At its core, contract testing is a way of managing changes across independently deployed services. It is best suited to testing applications built on microservice architectures that consist of many internal services that interact with each other. It is better suited to applications whose services interact frequently with each other than to applications with only a few services or those that interact primarily with external, public APIs.

What is Contract Testing?

Contract testing is a test approach that verifies that two services, consumer and provider, meet a contract that specifies how they must interact.

Recall from our article on software testing strategies that a test approach consists of a test type, test practice and test level. We can use these definitions to classify how contract testing fits among other test approaches in your organization.

• The purpose of contract testing is to verify the functionality of a service, specifically that consumer and provider are aligned with their contracts, so contract tests are of the functional test type.
• Contract tests run directly on the code of the service the development team is working on, so they are at the unit test level.
• Contract tests run against a test double (such as a mock or a stub) of another service the team’s service interacts with. Unlike traditional unit tests, where the test cases and the test double are written by the development team, in contract testing, the test double is generated from the public API contract of the other service, meaning teams that write contract tests do not need to manually maintain those doubles. Contract testing can therefore be considered a generator-based test practice.

Contract tests are therefore functional, generator-based unit tests for services that interact with other services via contract.

Like unit tests, contract tests are focused, cheap, and fast. Unlike other unit tests, the test doubles are generated from the contract of the service being tested, and so are potentially more reliable over time.

Next, we’ll look at why contract testing evolved, and what problems it solves.

Why Contract Testing?

Contract testing emerged because of the move to microservices and Application Programming Interfaces (APIs), which allowed development teams to deliver small increments of value more quickly than with monolithic applications. As microservices evolved, and service dependencies multiplied, breaking changes increased, and the need to create contracts between services was recognized.

In 2006, Ian Robinson described the design pattern of Consumer-Driven Contracts. In 2009, J.B. Rainsberger wrote Integrated Tests Are a Scam, which influenced the development of Pact, the first contract testing tool, in 2013 by a team that included Ron Holshausen, Brent Snook and Beth Skurrie.

Before contract testing, development teams often tried to mitigate the risk of breaking changes in services by adding more unit tests, integration tests or end-to-end tests.

Shailendra Singh of Hypertest, another testing tool that supports contract testing, discusses a use case that illustrates the type of integration issues and breaking changes that contract tests can catch but other testing approaches may not. The issue occurred at Atlassian, a software company, where a Billing Service for one of its products failed. It took the team an entire weekend to determine the cause, a breaking change in the API of a User Service that was called by the Billing Service, something developers working on either service would not have caught through unit tests alone.

They had unit tests that were fast and tended to be very reliable, but in this case did not catch these problems. Why did this happen? Because the engineers who wrote these tests forgot to update mocks.

Shailendra Singh

Below is a comparison of contract testing to unit testing, integration testing, and end-to-end testing. The table compares what the testing approach validates, how quickly it gives feedback, the scope of failures, maintenance burden, and where it fits in the test strategy.

How Contract Testing Works

Figure 1 shows an example of the entities and processes of contract testing. In production, a consumer service sends a request to a provider service and awaits a response, forming a standard request and response interaction. For unit testing, the consumer creates a contract, typically in OpenAPI format, with the information and data it needs from the provider. This contract is used to generate a mocked dependency of the provider for the unit tests of the consumer service.

Similarly, the provider creates a contract following the OpenAPI specification with the information and data it needs from a consumer service. This contract is used to generate a mocked dependency of the consumer for the unit tests of the provider service.

If the API of either the provider or the consumer are updated, the corresponding contract is updated and the mocked dependency is generated again automatically by a contract testing tool, preventing integration issues before they reach production. This ensures that the mocks for other services are always up to date.

Consumer-Driven vs Provider-Driven Contract Testing

The process just described is called consumer-driven contract testing because it allows the consumer to tell the provider what it needs. Several contract testing tools use this approach, including Pact, Hypertest and Spring Cloud Contract.

As previously mentioned, Pact was the first contract testing tool. It includes an open source version and a commercial version called PactFlow. Pact supports Java, TypeScript, Go, .NET, Kotlin, Swift, PHP, Ruby and Python. HyperTest supports Java and Node.js, while Spring Cloud Contract supports Java Spring Boot applications.

Such tools need access to the source code of both the provider and consumer. This enables the contracts and test doubles (mocks or stubs) on both sides to be updated, if needed, before unit tests and contract tests are run. A benefit is that the contract tests become part of the quality gate for the build. For the build to succeed in a continuous integration / continuous delivery (CI/CD) pipeline, the full test suite, including unit tests and contract tests, must all pass.

Consumer-driven contract testing can be done manually without specialized contract testing tools. An example process would be for developers to update the contract for their service whenever they change the API, then to notify other teams, then each other team would update the test doubles in their unit tests. This might work fine for a few services, but for products encompassing tens or hundreds of services, for every team to carry out such a manual process risks becoming a coordination challenge.

Organizations could build internal automation into their CI/CD pipelines for some or all of these steps. But then maintaining this automation could become a maintenance challenge.

Consumer-driven contract testing is also effectively limited to internal services within your organization. What if your product uses external services with public APIs? For this case, provider-driven contract testing is needed.

Figure 2 shows the entities and processes of provider-driven contract testing. The production scenario is the same as before, a request and response scenario where a consumer service sends a request to a provider service and awaits a response, including any data exchanged between them. For unit testing, the consumer mocks the provider based on the provider’s public API. However, the consumer has no way to tell the provider what it needs. Also, there is no mechanism for automatically notifying the team developing the consumer that the provider API has changed and that they need to update the mock of the provider in their unit tests. Consumer-driven tools solve exactly this problem by keeping both teams in sync automatically.

Internal automation could be added to build pipelines to check the API periodically against the previous version of the API. Without such automation, keeping unit tests current becomes a manual coordination challenge.

Provider-driven contract testing does not require the specialized tools that are needed for consumer-driven contract testing. Standard API testing tools can be used instead, making it a more accessible option for teams already familiar with API testing. For example, contract testing can be done with Bruno, a lightweight API framework.

Introducing a new test approach and deciding how to implement contract testing into an organization that hasn’t done it before can seem like an overwhelming task. BairesDev offers software quality assurance services trusted by over 1,500 companies, with experience across a wide variety of testing approaches and tools.