vendredi 26 avril 2019

Does xUnit support "same tests, different setups"? Or how can I write tests that run locally for dev, and in a pipeline to test deployment?

TL;DR: As per title: How can I run the same tests but have different setups?

I'm working in C#, Visual Studio, deploying via Azure DevOps (nee VSTS) to a WebApp in Azure.

The software under development configures resources in Azure, and identities in Azure Active Directory.

This means I have tests like:

[Fact]
public void ShouldCreateAzureDevOpsProject()
{
   _azureDevOpsHelper.GetAzureDevOpsProject(_projectName).Should().NotBe(null);
}

[Fact]
public void ShouldPutDefaultFileInRepo()
{ 
  _azureDevOpsHelper.GetDefaultFileFromRepo(_projectName).Should().NotBe(null);
}

[Fact]
public void ShouldEnableAllMicrosoftResourceProviders()
{
   _azureSubscriptionHelper.GetMicrosoftResourceProviders().Select(x => x.RegistrationState).Should().NotContain("NotRegistered");
}

I want to run these tests against the code as I write it. The code runs on my laptop, so the setup (which I currently have in an xUnit Fixture) is:

new EngineOrchestrator.EngineOrchestrator().RequestInstance(userSuppliedConfiguration);

But those tests are equally applicable to being run in our deployment pipeline, to check for regression after deploying to our test environment. For those purposes, the set up would involve creating a HTTP client, and hitting the endpoint of the application.

The point is that the tests are identical, regardless of the setup. The values-to-test-for are sourced from a json configuration file in both the 'local' and 'pipeline' cases; isolation is achieved by subbing in a different configuration file for the tests during deployment.

Put another way; I'm trying to work out how I can encapsulate the setup, so that two different setups can share the same tests. That's the reverse of what fixtures etc do, where multiple tests can share the same setup.

I have tried:

  • Switching the "setup" depending on what machine is running the test
if (Environment.MachineName.StartsWith("Plavixo"))
{
    new EngineOrchestrator.EngineOrchestrator().RequestInstance(userSuppliedConfiguration);
}
else
{
    HttpEngineHelper.RunOrchestrator(userSuppliedConfiguration, authenticationDetails);
}

This is my current solution, but it feels brittle, and makes the test artifact huge because it necessarily includes all the source to be able to new up the Engine, even when it's going to run on the build machine.

  • Making the Tests Abstract, and inheriting to concrete classes that have their the specific setup in their constructors.
public class LocalBootstrap : BootstrapTests.BootstrapTests
{
   public LocalBootstrap():base()
      {
         //do specific setup here

public abstract class BootstrapTests
{
   [Fact]
   public void ShouldCreateAzureDevOpsProject()

This sort of worked, but the set up ran before each and every test, which make sense: "xUnit.net creates a new instance of the test class for every test that is run, so any code which is placed into the constructor of the test class will be run for every single test."

  • Making the Fixture abstract

A fixture runs once and is shared between tests. I tried making the fixture abstract, and having a concrete class for each of my set ups.

xUnit throws a System.AggregateException : Class fixture type may only define a single public constructor. This is referenced by a github issue that has been closed as "By Design"

  • Running my local tests on a localhost

This is my next option to investigate. Is this a good idea?

Are there any other things I should try?

Aucun commentaire:

Enregistrer un commentaire