jeudi 10 décembre 2020

Is there a standard name for testing the number of tests, or other calculated parameters?

This is a general question about testing, independent of language.

One method I often use when writing tests, which I've never heard of before although I surely didn't invent it, is the following:

Suppose that I'm testing something multiple times, e.g. in a loop; e.g., it could be artificially generated test data. E.g., it might be calling a func(...) which returns some data containing a list/vector/array etc. etc. of results.

I have at the start of the test something like

int total_size = 0

...and with each calculated result:

data = func(...)

do_some_test(data)

total_size = total_size + data.size()

Then, finally at the end of the test, I'll do

TEST_THAT(total_size == 10000)

...or whatever the value is. Of course, it might not be size that I test; it could use any discrete or integer-valued property of the data objects.

So, this could be regarded as a very weak kind of integer-valued hash of test data, input and/or output, although human-readable.

There are at least four reasons to do this:

(a) Check that we are actually testing something; it's all too easy for bugs in test code which mean that we're not testing anything, or as much as we thought.

(b) If something changes unexpectedly, due to the algorithms or tests, even if the tests still pass, then we have a chance of spotting that it changed.

(c) Although it could give a false sense of security, it's always nice to see that a test has run through 1000 times, or tested 5000 things, etc. etc. rather than just 5, say. If people ask you how well tested something is, you can immediately say "it has been tested on 2000 objects, of total size 10000", or similar.

(d) If running the same code on multiple platforms or compilers, it helps to find differences between them.

So, my questions (sorry, two rather than one!):

(1) Is this a standard method with a well-known name?

(2) Assuming it is well known, is this regarded as good practice (if done sensibly, of course?)

Aucun commentaire:

Enregistrer un commentaire