samedi 13 avril 2019

Automatic low level testing generation based on high level desired behavioral examples

Top Level Problem

Our team has inherited a very large and brittle python 2 (and C,C++, few others) codebase that is very difficult and costly to update. Tons of dependencies. Very few tests. Adding behavior improvement and converting to python 3 both have appeared to be monumental tasks. Even making small changes for a new release we've had to revert many times as it's broken something.

It's a story of insufficient testing and its major technical debt.

Still, the project is so big and helpful, that it seems a no brainer to update it than re-invent everything it does.

Sub Problem

How to add a massive amount of missing small tests. How can we automatically generate even simple input/output acceptance unit tests from the high level user acceptance tests?

Attempted Solution

There are about 50 large high level behavioral tests that this codebase needs to handle. Unfortunately, it takes days to run them all, not seconds. These exercise all the code we care the most about, but they are just too slow. (Also a nerdy observation, 80% of the same code is exercised in each one). Is there a way to automatically generate the input/output unit tests from automatic stack examination while running these?

In other words, I have high level tests, but I would like to automatically create low level unit and integration tests based on the execution of these high level tests.

Mirroring the high level tests with unit tests does exactly zero for added code coverage, but what it does do is make the tests far faster and far less brittle. It will allow quick and confident refactoring of the pieces.

I'm very familiar with using TDD to mitigate this massive brittle blob issue in the first place as it actually speeds up development in a lot of cases and prevents this issue, but this is a sort of unique beast of a problem to solve as the codebase already exists and "works" ;).

Any automated test tool tips? I googled around a lot, and I found some things that may work for C, but I can't find anything for python to generate pytests/unittest/nose or whatever. I don't care what python test framework it uses (although would prefer pytest). I must be searching the wrong terms as it seems unbelievable a test generation tool doesn't exist for python.

Aucun commentaire:

Enregistrer un commentaire