Skip to content

Contract Testing

Jump to bottom
Leslie Fung edited this page Jan 28, 2020 · 11 revisions

Spot Contract Testing

Many of the concepts and examples presented here have been directly sourced and/or repurposed from the articles found here: Pact Foundation

  1. Overview
  2. Contract Tests vs Functional Tests
  3. Contract Testing In Provider Integration Tests

Overview

A contract is only useful if we can verify that both the consumers and provider adhere to it.

Consumers

The consumers of a contract are clients that rely on the contract. Spot contracts are hand written, so clients should rely on a client SDK generated from the contract to ensure conformance. The consumers of a Spot contract are effectively the code generation utilities that use the contract. Pragmatically speaking, consumers are conformant as long as the Spot contract itself is valid.

Provider

The provider service of a Spot contract must be tested to ensure conformance to the contract. This ensures a provider service can confidently release without breaking any reliant client applications. The following sections will detail how Spot contracts can be tested.

Backwards Compatibility: Contract testing also enables services to confidently remain backwards compatible. See Backwards Compatibility.

Contract Tests vs Functional Tests

The goal of contract testing is to verify that the shape of data returned by a provider meets consumer expectations. Contract tests help to identify:

  • potential bugs in consumer applications
  • misunderstandings about provider endpoints or responses

The confidence gained from contract tests allows a consumer application to gracefully handle any response returned by the provider service.

With this in mind, contract tests should focus on how a provider responds, not why it responds a certain way. Functional testing of complex business logic should be left to the provider to test.

Assumptions

  1. A reasonable assumption must be made that the provider uses the same serializeration format for a response status code returned by a given endpoint.
  2. Consumers must trust that the provider service has implemented its business logic correctly.

Case Study

Consider a User Service that allows clients register users. A happy path test case may look like:

POST /users
{
  username: "johnsmith",
  email: "[email protected]",
  firstName: "John",
  lastName: "Smith"
}

Expect: response.status === 201

To avoid a misunderstanding of how the provider behaves, we must also test other response codes. A typical failure test case may look like:

POST /users
{
  username: "",
  email: "[email protected]",
  firstName: "John",
  lastName: "Smith"
}

Expect: response.status === 400
Expect: response.body === { error: String }

The provider team have now indicated that the username must not exceed 20 characters in length. At this point it may become very tempting to add a new test case:

POST /users
{
  username: "username_with_21_char",
  ...
}

Expect: response.status === 400
Expect: response.body === { error: String }

We are now in the realm of functional testing. The test case is testing that the User Service has implemented its validation rules correctly. These tests should be covered in the User Service's codebase.

But why?

Let's compare the expectations between the two failure cases:

Expect: response.status === 400
Expect: response.body === { error: String }

Expect: response.status === 400
Expect: response.body === { error: String }

They are identical! The consumer application will react to both in the same way. We don't really care about the individual business rules, we care about how the User Service responds when something goes wrong.

But more testing is good right?

Consider now the scenario where the business rules change. The provider service has now increased the maximum allowed length for the username to 25 characters. Now the test case fails:

POST /users
{
  username: "usernamewith_21_chars",
  ...
}

Expect: response.status === 400
Expect: response.body === { error: String }

Received: response.status === 201

Consumer applications should be relatively unaffected by such a change as the contract has not changed. However the provider service will cause the contract tests to fail by loosening validation rules.

Best Practice

  • For happy path test cases, construct requests which are unlikely to ever cause errors
  • For failure test cases, construct requests that break some rule that is likely to never change
  • Provide one good test case for every response code

Contract Testing In Provider Integration Tests

Provider integration tests often involve simulating client requests and checking resulting responses. Contract tests can form a part of the integration tests by checking that the request/response pairs conform to the specified contract.

Spot Validation Server

The Spot provides a validation server for providers to integrate with during integration tests to perform contract testing.

To start the validation server:

$ yarn spot validation-server api.ts

The validation server should be started and available during integration tests. The validation server provides a GET /health (see health.ts) endpoint for checking validation server readiness.

POST /validate

The validation server exposes a POST /validate endpoint (see validate.ts) which accepts a recorded HTTP request/response payload and returns validation errors. Example:

// POST http://localhost:5907/validate

// REQUEST
{
  "request": {
    "method": "POST",
    "path": "/company/123/users",
    "headers": [{ "name": "x-auth-token", "value": "helloworld" }],
    "body": "{}"
  },
  "response": {
    "status": 200,
    "headers": [{ "name": "a", "value": "b" }],
    "body": "{}"
  }
}

// RESPONSE
{
  "interaction": {
    "request": {
      "method": "POST",
      "path": "/company/123/users",
      "headers": [{ "name": "x-auth-token", "value": "helloworld" }],
      "body": "{}"
    },
    "response": {
      "status": 200,
      "headers": [{ "name": "a", "value": "b" }],
      "body": "{}"
    }
  },
  "endpoint": "CreateUser",
  "violations": [
    {
      "type": "request_body_type_disparity",
      "message": "Request body type disparity:\n{}\n- # should have required property 'data'",
      "type_disparities": ["# should have required property 'data'"]
    },
    {
      "type": "response_body_type_disparity",
      "message": "Response body type disparity:\n{}\n- # should have required property 'name'\n- # should have required property 'message'",
      "type_disparities": [
        "# should have required property 'name'",
        "# should have required property 'message'"
      ]
    }
  ]
}

We recommend building a custom test matcher in your testing framework to transform performed interactions into the required format and sending them to the validation server. Any violations can be reported by your test matcher.

Violations

The Spot validation server is strict with request validation. Extra headers, query parameters and body fields will return violations. This encourages providers to construct realistic requests during testing and reduces the chance of extra data resulting in a false positive. The validation server is not strict with response validation as any extra data should be ignored by clients.

Returned violations come with a type field. The type can be used to filter violations. This can be useful to ignore some expected violations. For example - in order to produce a 400 response, sometimes it may be required to construct a request body that does not conform to the contract.

Type Reason
undefined_endpoint the provided request path does not match an endpoint defined on the contract
undefined_endpoint_response an endpoint was matched but the provided status code does not match any response defined on the contract
required_request_header_missing the provided request headers do not contain a required request header defined on the contract
undefined_request_header a provided request header is not defined on the contract
request_header_type_disparity a provided request header's value does not conform to the type defined on the contract
path_param_type_disparity a path parameter does not conform to the type defined on the contract
required_query_param_missing the provided query parameters do not contain a required query parameter defined on the contract
undefined_query_param a provided query parameter is not defined on the contract
query_param_type_disparity a provided query parameter's value does not conform to the type defined on the contract
undefined_request_body a request body was provided but no request body is defined on the contract
request_body_type_disparity the provided request body's value does not conform to the type defined on the contract
required_response_header_missing the provided response headers does not contain a required request header defined on the contract
undefined_response_header a provided response header is not defined on the contract
response_header_type_disparity a provided response header's value does not conform to the type defined on the contract
undefined_response_body a response body was provided but no response body is defined on the contract
response_body_type_disparity the provided response body's value does not conform to the type defined on the contract
Clone this wiki locally