mercredi 20 novembre 2019

Dynamic Test with live fetched test data? Is this a good or bad idea?

So I was thinking about a framework that provides test data fetched live from the production environment.

To scope the concept down here is a small pseudo code in java for this:

class MyApplicationTest {
  private MyApplication myApplication;

  @DynamicTestData(from="MyApplication", method="doStuff")
  private String dynamicTestData;

  @DynamicTest(range = DataRange.LATEST50)
  public void test() {
    myApplication.doStuff(dynamicTestData));
    // e.g. AssertJ logic
  }
}

class MyApplication {
  @TestDataCollector(for = "MyApplication")
  public void doStuff(String someData) {
    // business logic
  }
}

So the idea is the following. When the application is running in a the production environment all calls to the doStuff method will be proxied.

@TestDataCollector(for = "MyApplication")

Defines what method to collect the argument values as test data from. The proxy logic will send messages to a TestDataCollectorServer that is saving the arguments given to the doStuff method.

Now when you define a DynamicTest now (should not be seen as a unit or integrationt test) you can tell the test for what application you are requesting test data from. Done by:

 @DynamicTestData(from="MyApplication", method="doStuff")

Then by defining a dynamic test like that:

 @DynamicTest(range = DataRange.LATEST50)
  public void test() {
    myApplication.doStuff(dynamicTestData));
    // e.g. AssertJ logic
  }

It will run like a parameterized test with the DataRange.LATEST50 entries collected by the TestDataCollector Server.

Assumed Benefit

So the benefit I as a normal skilled developer see is that you have real used data to run your tests. I see the problem that those dynamic tests might fail in one case and might succeed in another. But still I see the advantage of having real life cornercases explicitly crashing test cases and therefore one can write a test case for that post mortem.

What are the problems?

I already discussed this in a small circle and people explained several problems to me. I'd like to have this thing evaluated by a greate audience and therefore am posting it here as a question.

Aucun commentaire:

Enregistrer un commentaire