Traffic mirroring: test APIs without tests

Integration and load testing are in the list of best practices in our sector, but synthesizing good inputs for such tests and analyzing the results are usually two hard things to achieve. This post will show you how we suggest our customers set up a shadow testing system with minimal impact on the production environment and proper harness.

Quick overview

As pointed out at https://cloud.google.com/architecture/application-deployment-and-testing-strategies

Testing new backend features by using the production load.

When used with tools such as Diffy, traffic shadowing lets you measure the behavior of your service against live production traffic. This ability enables you to test for errors, exceptions, performance, and result parity between application versions.

In basic terms, our proposal includes two scenarios:

• Performance regression testing to answer the question “is the new version worst than the current one?”

• Feature regression testing to answer the question “does the new version honor the expected behavior defined by the current one?”

Both scenarios require mirroring the incoming traffic. Still, the second one also requires adding one more piece to the puzzle, which can analyze responses and process the differences between environments/versions.

Notice: Setting up the network connection from production (where your mirrored traffic comes from) to staging could be tricky, depending on how you deal with domain names and inter-cluster connectivity.

Testing the performance

The easiest way to ensure your next release is as performant as the current one is to deploy it in an identical environment as your current version is and clone the incoming traffic.

At the KrakenD configuration, we can add a mirror backend next to every safe, actual request so that KrakenD can mirror the traffic. The safe part in the last statement is vital since KrakenD will reject your endpoint otherwise because the test should not have side effects. The KrakenD documentation site offers a complete description of the traffic shadowing or mirroring feature

The following configuration snippets will demonstrate the process:

Original

    {
      "endpoint": "/shadow",
      "backend": [
        {
          "url_pattern": "/destinations/1.json",
          "host": [ "http://service1:8080" ]
        }
      ]
    }

Final

    {
      "endpoint": "/shadow",
      "backend": [
        {
          "url_pattern": "/destinations/1.json",
          "host": [ "http://service1:8080" ]
        },
        {
          "url_pattern": "/destinations/1.json",
          "host": [ "http://stg-service1:8080" ],
          "extra_config": {
              "github.com/devopsfaith/krakend/proxy": {
                  "shadow": true
              }
          }
        }
      ]
    }

So with this simple change, you can compare the performance metrics of your environments and quickly identify any possible issue.

Feature regression test

A lean solution for building such a service is to create a cluster of shadow testing instances for each internal API consumed by the gateway. These shadow testing instances monitor the current and the subsequent release candidate, sending the incoming requests mirrored from the production gateway to the backends deployed at both environments (production and staging) and inspecting the responses, looking for differences to report.

Diffy is a fantastic tool that allows teams to harness their API releases without writing a set of synthetic tests just by proxying the incoming traffic to several backends and comparing the responses, looking for noise and differences between them.

An excellent overview and introduction to Diffy is this post from Twitter’s engineering team.

Canary testing with the Diffy service

The proposed flow for all the endpoints consuming resources from the backend service1 (notice the naming convention used to identify not- production-ready environments) is:

This is the minimal required Diffy configuration for such environment definition, ignoring the .tld suffix (that we have removed for clarity at the local docker configuration). In this example, the service1 name is used as primary and secondary master, delegating the decision to some 3rd party, but they could also be instance hostnames, IP…

docker run -ti \
  -p 8880:8880 -p 8881:8881 -p 8888:8888 \
  diffy/diffy \
    -candidate=stg-service1:8080 \
    -master.primary=service1:8080 \
    -master.secondary=service1:8080 \
    -service.protocol=http \
    -serviceName="Service1" \
    -proxy.port=:8880 \
    -admin.port=:8881 \
    -http.port=:8888 \
    -rootUrl=diffy-service1:8888
    -summary.email=''

Mirroring the incoming traffic

With that infrastructure in place, we can make a small change in our KrakenD configurations to send the mirrored traffic to the proper Diffy service instead of sending it directly to the staging environment.

Again, the following configuration snippets will demonstrate the process:

• sending the mirrored traffic to staging

    {
      "endpoint": "/shadow",
      "backend": [
        {
          "url_pattern": "/destinations/1.json",
          "host": [ "http://service1:8080" ]
        },
        {
          "url_pattern": "/destinations/1.json",
          "host": [ "http://stg-service1:8080" ],
          "extra_config": {
              "github.com/devopsfaith/krakend/proxy": {
                  "shadow": true
              }
          }
        }
      ]
    }

• sending the mirrored traffic to Diffy

    {
      "endpoint": "/shadow",
      "backend": [
        {
          "url_pattern": "/destinations/1.json",
          "host": [ "http://service1:8080" ]
        },
        {
          "url_pattern": "/destinations/1.json",
          "host": [ "http://diffy-service1:8880" ],
          "extra_config": {
              "github.com/devopsfaith/krakend/proxy": {
                  "shadow": true
              }
          }
        }
      ]
    }

A complete environment is available at this repo.

Validating the next candidate

After deploying the new KrakenD configuration, requests will start hitting the Diffy services, and you can start monitoring the results using the Diffy GUI and your tracing backend (check the telemetry section for more details).

In addition to that, if you are using the B3 propagation standard in your backends, you will also see the traces at the backend level nested depending on the trace id assigned by the gateway. So you can compare the performance of both versions without delay added by the Diffy proxy.

The following screenshot shows how the actual client is not affected by the generation of the shadow request to the Diffy instances.

Diffy considers differences between responses from primary and secondary upstreams as noise, so it allows you to remove it to see the actual differences between the current version and the one from the staging.

You can inspect differences between baselines and candidates in a dedicated pop-up window.

With the differences already highlighted for you

Diffy is smart enough to know that such difference has no value, and it considers these kinds of scenarios as noise and allows you to ignore them.

Observations

Restrictions

The feature regression testing system is just limited to check use cases that we don’t update frequently. If sending the same request one second after or not could hugely impact the response, then the use case is not a good candidate for the feature regression test. These scenarios also require a longer training time before being able to interpret the deviation of the results.

Impact on load

It’s critical to keep in mind that, by using the Diffy trick, the load of the defined backend endpoints will increase by 200%. Here are some easy options to keep the extra load under control:

1. set up the shadow traffic feature in a subset of KrakenD instances, so not all the incoming traffic gets mirrored.

2. schedule a couple of blue-green deployments per day, week, or whatever your deployment cycle is, enabling and disabling the shadow backends.

3. switching from pure mirroring (Diffy off) to shadowed harnessing your versions (Diffy on).

4. add a new environment cloned from production with the current version and use it as a source of truth in your Diffy tests, so your production environment doesn’t receive the extra load. This option could require a read-only replica of your production databases to keep the source of truth up to date.

Privacy concerns

One final thought regarding privacy: Diffy will check the processed responses’ contents and store sensitive data without obfuscating it. Depending on your requirements, maybe you’ll need some extra work in that area.

Conclusions

We have seen one powerful application of the traffic mirroring feature KrakenD API Gateway offers out-of-the-box: shadow testing your API releases. With minimal changes to the gateway, we can mirror the production traffic to a testing environment where we can evaluate new release candidates in terms of performance and functionality without writing a single synthetic test.

Start doing your experiments now by downloading the source code of these examples.

Thanks for reading! If you like our product, don’t forget to star our project!