Integration Tests for Third Party APIs

Pondering how to write proper integration tests for third party API? The following post might show you a new approach. We will be using Consumer Driven Contract Testing (CDCT).

Let’s start with what a Third Party API is supposed to be. We regard it as API which is not under our control. We cannot change it ourselves. Asking for a change might take weeks or months. Examples range from the Google Maps APIs to some Legacy CRM system internal to our company.

Until last year I have experienced three ways of dealing with these APIs from a testing perspective:

  1. Not automating the tests of the integration at all
  2. Hitting the production API
  3. Check in the API responses (e.g. .json files) into source control

The first one is by far the easiest. We make sure the application still works by manually checking if the application does what its supposed to do and whether the correct values from the APIs are used.

With the second approach we make sure the API is properly integrated without the need to manually checking it. The downside is that these APIs might be slow to respond, leading to slow tests and tests which will fail from time to time. For example if our CI tool loses the network connection. One might also run into rate limiting issues when the API is called too often in a short amount of time. We might even incur a cost for every request since not every API is free of charge.

The third approach is the one I usually encounter. With it we are neither subject of additional costs per request, slow tests or rate limiting issues. The production API is only ever hit once during the creation of the initial .json file. There are even tools which help us with the workflow such as Ruby VCR.

Still, one issue arises. We would not notice right away if there is a mismatch between the response in the .json file and the actual response. Aside from sloppiness and not checking in the correct response, e.g. with missing structure, we would also not notice directly whether the API provider breaks the API by removing fields. The latter might be improbable when working with Google APIs but is certainly possible with company internal systems or the API of smaller vendors.

To mitigate the issues above a colleague introduced the idea of using Consumer Driven Contract Testing (CDCT) for third party API. Since we’re already using it for API we do have control over the tooling was already in place and the developers were well versed in its usage.

CDCT is often called TDD for microservices and used as such. The API consumers (e.g. an app or another microservice) writes contracts on how they require the API to behave and thus which fields of a response is used.

As an example say the API returns the following:

{
"id": 42,
"age": 28,
"address": "Some street in some City",
"phone_number": "+49 123456789"
}

One consumer might be interested in the age of the user, another might only care about the address. Thus consumer A only cares whether the age field is present while consumer B only cares whether the address field is present.

Consumer A is ok as long as the response is at least:

{
"age": 28
}

Consumer B is fine if the response looks as follows:

{
"address": "Some street in some City"
}

Both state these facts in a contract.

On the provider side (a service) there will be a test to make sure the contract is never violated. Thus if the provider removes a field from a response, which is not mentioned in any contract, all tests will still pass. Because thefield was save to be removed. No consumer was using or expecting it. Thus for the example above one would not be able to remove the age or the address. As a result if the business decides customers will not be called anymore removing the phone_numberwould be possible. No contract depends on it. Even the id could be removed.

Of course we cannot give these contracts to the Google Maps API or to our legacy CRM system and force them to abide. These providers presumably neither care nor have the tooling in place to support CDCT. Thus on first glance using CDCT for third party APIs seems odd.

What we can do is create another service which acts as a replacement for the actual service during automated testing. The service will hold the contracts defining the required fields from the actual API. We call these services proxies. They never proxy requests but act as an intermediate between the API and the implementation during automated testing. The proxy project will have two goals:

  1. Make sure the API responds as expected (Similar to using the production API during tests)
  2. Provide stubs to consumers (Similar to checking in the JSON response files)

Thus the best of both approaches without the drawbacks.

Let’s have an example. We want to display how long it takes to drive from Stuttgart, Germany to Berlin. With the Google Distance Matrix API we can get the following result. Here were using HTTPie.

http https://maps.googleapis.com/maps/api/distancematrix/json \
origins==Berlin destinations==Stuttgart

The call will result in

{
"destination_addresses" : [ "Berlin, Germany" ],
"origin_addresses" : [ "Stuttgart, Germany" ],
"rows" : [
{
"elements" : [
{
"distance" : {
"text" : "636 km",
"value" : 635736
},
"duration" : {
"text" : "6 hours 18 mins",
"value" : 22651
},
"status" : "OK"
}
]
}
],
"status" : "OK"
}

We learn it takes about 6 hours and 18 minutes to drive from Berlin to Stuttgart at the time the request was made.

At first glance the only piece of interest to us is the value 22651. It’s the duration in seconds. Our consumers, e.g. an Android App, might want to decide how they want to format the value for the user.

We should make sure the value field is included in the response. Of course the documentation states it’s included but we want to be extra careful.

Using a Groovy DSL of Spring Cloud Contract we can define the following contract.

org.springframework.cloud.contract.spec.Contract.make {
request {
method GET()

The contract only contains the parts of the response we care about and the request used to create the expected response. As a result the following test, asserting on the relevant fields, will be generated automatically by the library.

@Test
public void validate_shouldProvideDistanceBetweenTwoCities() {
// when:
Response response = webTarget
.path("/maps/api/distancematrix/json")
.queryParam("origins", "Berlin")
.queryParam("destinations", "Stuttgart")
.request()
.method("GET");

If we provide the specified parameters in the request (the whenpart), we expect the specified response (the then part).

The DSL in the assertions is JSON Assert, a fluent interface for JsonPath. A great side-effect is how we have now documented in code which fields we use. Maybe later during development we find a cheaper or faster API filling our requirement to have these durations calculated. We would only have to compare if they offer the same functionality we already require.

The generated contract tests pass without us writing any implementation code. Google already did all the work for us. If we use CDCT for our own services we usually have to implement the functionality after defining the contract. That’s why CDCT was called TDD for microservices above. We’re done with the first step. We can verify the API behaves as expected by hitting the production API.

Aside from the generated tests, which some might regard as useless in as “Why would you want to test API you don’t own?”, we get some stubs. These stubs, which are mostly the response with some metadata as JSON file, can be used by our consumers to develop against. The library is using WireMock, a HTTP mock service, which returns the response defined in the contract when hit with the request defined in the contract. After all it was one of our initial goals to not permanently call the real API and to have stubs which respond as the real API does.

These stubs are made available to other projects via the local .m2 repository, a git repository or from our nexus or artifactory instance. An example using the local .m2 repository can be found in the sources at the end.

In our consumer project we will need a Client to make the appropriate HTTP calls to get our desired duration. Using the generated stubs we can test drive its functionality.

@SpringBootTest
@RunWith(SpringRunner.class)
@AutoConfigureStubRunner
public class GoogleDistanceMatrixClientTest {

Straightforward. With the actual implementation, using Feign, we should be able to make the test pass. JsonPath is used for brevity.

@FeignClient(value = "google-distance-service")
public interface GoogleDistanceMatrixClient {

To tell the test to hit the stub instead of the real API we need a service mapping configured below. In production you will want to point google-distance-service to https://maps.googleapis.com.

stubrunner:
ids: 'co.hodler:scdcproxy:+:stubs'
stubsMode: LOCAL
ids-to-service-ids:
scdcproxy: google-distance-service

Our tests are passing. The API is beautifully integrated. Marvellous. Ship it. We schedule a build to run the tests of the proxy project in an interval of our liking. For example once a day. The tests for the consumer projects can be run as often as we like. They never call the real API.

Are we done?

Not really. All seems fine until a road gets blocked and suddenly the duration takes one second more and we receive 22652 instead of 22651. Subsequently leading to a failing test. Instead of just changing the value to 22652 and waiting for it to change again on a future date we should improve the resilience of the test.

The updated syntax of the value part would be

value: $(stub(22651), test(regex('\\d+')))

Thus our stub (WireMock) will always provide the value 22651 to a consumer. The actual tests (the assertions) against the real API will only require the API to respond with at least one digit. That’s what the regex('\\d+') is for. Thus our generated assertion will change to

assertThatJson(parsedJson).array("['rows']")
.array("['elements']").field("['duration']")
.field("['value']").matches("\\d+");

The real API might return 12345 and the test would not care. Nor should it. Subsequently the test on the consumer side, which is expecting 22651 would not break.

Seems were done…

Not really, try to create the following request

http https://maps.googleapis.com/maps/api/distancematrix/json \
origins==Stuttgart destinations=="Gotham City"

The API does not always return the duration. Definitely not when using a fictional city. We could, and should, create a second contract covering the error case and make sure our Client can deal with the error. Currently it can’t. It would run straight into NullPointerException. Having contracts, and thus stubs, for these edge cases is a great way to harden the Client to deal with more than the happy path.

For the examples above everything looks great. Until we get to a requirement of preparing the API to respond with something specific. Say to fetch something via GET request we might need to first POST something. We also did not take into account adding a personal API key to the request. A way to address these issues will be part of a future post.

To sum up, using the above technique, we made sure

  • The API behaves as we expect it to
  • Aside from the proxy project, our tests do not call the real API
  • We make sure there is no mismatch between the response we expect and the actual response
  • We have documented which fields of an API we use

Hopefully this might be of use to others too. It surely is for us. We use the approach for every integrated third party API.

The code for the proxy project and the consumer project can be found on GitHub.

Building things. Usually by writing code. www.hodler.co. Software Engineering @porschedigital

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store