Enhance your .NET Testing #8: Contract tests with Verify

The most common way to test the values of an object is to use assertions for each property to check exactly what the object contains.

This can be done for types, classes, enumerables. The more aspects of the object that are tested, the more certain you can be that everything works as expected.

If you didn't read the previous article, I recommend you to do it:

Assertions can be made quickly with small classes. It gets really annoying when your class contains a class, which contains a class... and so on. Nesting creates a huge amount of validations.

The complexity can be solved by splitting the current class into minor classes to avoid manipulation, and by reducing the scope of each method, but this is not always possible, mainly when we want to validate an end-to-end object update workflow.

In this case, we want to be sure that the resulting object is exactly as expected, as in API response or data ingestion updates. Let's start with an assert validation of the following code. It creates a goat with a name and a color, then adds a habit.

public Task Test_Verify()
{
    var goat = new Goat
    {
        Name = "Bicky",
        Color = "Grey"
    };
    goat.AddHabit(".NET", 3); // First param is Name and second is Difficulty

    Assert.Equal("Bicky", goat.Name);
    Assert.Equal("Grey", goat.Color);

    Assert.NotNull(goat.Habits);
    Assert.Equal(1, goat.Habits.Count);
    Assert.Equal(".NET", goat.Habits.First().Name);
    Assert.Equal(3, goat.Habits.First().Difficulty);
}

As you can see, it is quite redundant and not really interesting to do with only 3 properties included in a list.

The job is done, all properties are properly tested.

If you pay attention, something is not tested here! Have you noticed what it is?

Imagine that this class has been working fine for 2 years, the team of developers has changed (or you may still be on this project but you have to do other things in this period) and the Product Owner asks you to add the property Age on the goats.

The first thing you do is update the Goat class, add the implementation requirements, create tests to validate and push to the main branch.

There's still something unpleasant here.

You remember our test above. It's still green, when it shouldn't be. We only tested the properties we knew about when the test was created, and the code won't raise any red code since the assertions are still true.

Imagine exposing unwanted properties on your API without noticing it...

Verify to save us all

Presentation

Verify, a .NET library, helps us overcome this problem by generating a JSON file modeling the asserted object.

It works on all C# test libraries: xUnit, MSTest, NUnit and Expector for F#.

Each test using Verify will generate a file containing the test result. This file will be compared to the verified file of the test with the diff comparison.

If the two files are identical, the tested method returns the expected result. If not, something has changed and an error is raised. It is very simple!

The only difference is the first run because there is no file to compare with. The test will be red, and you will have to merge the received file into the verified file which is currently empty.

The repository contains an excellent diagram of how this works.

We will take a simple example to show how Verify works. It is possible to go much further by using the extensions :

Ensure the completeness of an API response (code, content, headers and cookies)
Validate the Moq configuration inside the test file to notice any changes of the initial state
Track EF changes and SQL query creation
Check the log flow

Quick example

The test has been updated to remove the assertions and replace them with the Verify() function.

The goat argument will be the serialized and compared result.

public Task Test_Verify()
{
    var goat = new Goat
    {
        Name = "Bicky",
        Color = "Grey"
    };
    goat.AddHabit(".NET", 3);
    
    return Verify(goat);
}

Once the test launched, everything is red. 2 files are also creating, one for the verification and one containing the received object.

A pop-up should open to show the difference between the two files. As expected, the verified file is empty. Merge the contents to have the first verified file completed.

If you run the test again, it should be green.

Add extra settings

Verify can be parameterized to modify the verification operation using the VerifySettings class.

To understand why, let's take a few examples.

The first, which can be annoying, is to disable the popup that opens every time a comparison fails, using the DisableDiff() option.

The second, for the sake of organizing files, is to define the folder in which verified files are stored with the UseDirectory() option. It's rather annoying to have a mixture of test class files and verified results. For simplicity's sake, we recommend creating a sub-folder with results only.

The third problem is when the content of the result contains a random part such as a date or a GUID generated when an element is created (which is often the case in an API). Verify's built-in solution is to use AddScrubber() to modify the content of the result, either with constants or regexes.

_settings.AddScrubber(builder => {    
	var modifiedContent = Regex.Replace(builder.ToString(), "MYREGEX", "MYREPLACEMENT");    
    builder.Clear();    
    builder.Append(modifiedContent);
});

Use Verify for contracts tests

Now that Verify's configuration is out of the way, it's time to put it to practical use: contract testing.

Contract testing is useful when setting up an exposure, whether HTTP or something like GraphQL, to ensure that the system doesn't regress from one version to the next. This is all the more important on public APIs, where other players wired into our system expect it not to change from one version to the next.

To ensure this, it's important to check 2 elements: input requests and output responses.

One way of making sure that this is the case is to use a WebApplicationFactory to load the entire application and start a HttpClient. If you're unfamiliar with this, I suggest you read this article before continuing:

The test is divided into several parts: configuring Verify, sending the HTTP request and retrieving the response to be validated.

[Fact]
public async Task ShouldXXX()
{
	// Configure Verify to replace Bearer
    var settings = new VerifySettings();
    settings.AddScrubber(builder => {
        var pattern = @"Bearer\s[^\s]+";
        var replacement = "Bearer REPLACED_BEARER_TOKEN";
        var modifiedContent = Regex.Replace(builder.ToString(), pattern, replacement);
        builder.Clear();
        builder.Append(modifiedContent);
    });
            
    // Configure HttpClient to run again the API or the WebApplicationFactory
    var client = new HttpClient(); 
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "MYBEARER");
    var uri = new Uri(client.BaseAddress, "/API/ROUTE/?param=test");
    
    // Send request
    var response = await client.GetAsync(uri);
    
    // Extract and verify contract from request
    var contractContent = await ExtractContent(response);
    await Verify(contractContent);
}

The HttpResponseMessage contains all the request information, both from the request and the response. All you need to do is extract and serialize it, so that Verify can use it.

Very important point: the contents of both the request and the response must be retrieved with ReadAsStringAsync to be sure of having the raw version of the contents. If you use ReadAsJsonAsync or a serializer, you run the risk of modifying the raw result, and therefore of having a mismatch between what you're testing and the real return.

protected virtual async Task<string> ExtractContent(HttpResponseMessage response)
{
    var requestContent = response.RequestMessage?.Content is null ? "" : await response.RequestMessage.Content!.ReadAsStringAsync();
    
    var content = new { 
        Request = new {
            Headers = response.RequestMessage.Headers,
            Method = response.RequestMessage.Method,
            RequestUri = response.RequestMessage.RequestUri!.PathAndQuery,
            Version = response.RequestMessage.Version,
            Content = requestContent
        },
        Response = new {
            Headers = response.Headers,
            StatusCode = response.StatusCode,
            Content = await response.Content.ReadAsStringAsync()
        }
    };
    return JsonSerializer.Serialize(content, new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping });
}

This gives the following result:

{
  "Request": {
    "Headers": [
      {
        "Key": "Authorization",
        "Value": [
          "Bearer REPLACED_BEARER_TOKEN
        ]
      }
    ],
    "Method": {
      "Method": "GET"
    },
    "RequestUri": "/API/ROUTE/?param=test",
    "Version": "1.1",
    "Content": ""
  },
  "Response": {
    "Headers": [],
    "StatusCode": 200,
    "Content": "{\"fieldA\":[],\"fieldB\":1,\"fieldC\":null}"
  }
}

We are now certain that the /API/ROUTE/?param=test route call in GET must always contain a bearer and must always have an HTTP 200 code with contents equal to "{"fieldA":[], "fieldB":1, "fieldC":null}".

If one of these elements changes (case included), the test will detect it.

Conclusion

Verify is a powerful library for simplifying the assertion of complex data models and documents such as API responses or log/PDF files.

Beware, Verify does not replace all property assertions. Personally, I see a huge use for API, logs and full object manipulation, but this is only a small part of my validation testing.

Last but not least, all *.received.* files must be excluded from source control if you want your git commits to stay clean. If you want to delete them recursively with a PowerShell script, you can use the following command :

Get-ChildItem -Path . -Recurse -File | Where-Object { $_.Name -like "*.received.*" } | Remove-Item

To go further:

Have a goat day 🐐

#Tests #.NET