mercredi 26 juin 2019

Reproducible random tests

I've recently run into the following piece of documentation. A cursory search revealed just a gist. So I decided to bring these ideas to life.

The first one is far from being used as is. The second one is too wordy and doesn't work in case of failure (failureException is supposed to be a class, the corresponding method can be commented out).

Before I describe my solutions, let me ask the questions. 1. Am I doing anything wrong? This sort of task must have been simplified ages ago. At least in my book. 2. Is there any way to improve my solutions?

The first one (branch factory-boy-state) stores random generators' state in files, that can later be used to pass state to the tests:

class MyTestRunner(DiscoverRunner):
    def setup_test_environment(self):
        self.handle_rnd_state(faker.generator.random, 'faker.state',
            'FAKER_RANDOM_STATE')
        self.handle_rnd_state(factory.random.randgen, 'factory.state',
            'FACTORY_RANDOM_STATE')
        super().setup_test_environment()

    def handle_rnd_state(self, rnd, fname, env_var):
        state = self.retrieve_rnd_state(fname, env_var)
        if state:
            rnd.setstate(state)
        else:
            self.store_rnd_state(fname, rnd.getstate())

    def retrieve_rnd_state(self, fname, env_var):
        state = os.environ.get(env_var)
        return pickle.loads(b64decode(state.encode('ascii'))) if state else None

    def store_rnd_state(self, fname, state):
        encoded_state = b64encode(pickle.dumps(state))
        with open(fname, 'w') as f:
            f.write(encoded_state.decode('ascii'))

Usage:

$ ./manage.py test --testrunner p1.tests.MyTestRunner
$ FAKER_RANDOM_STATE=`cat faker.state` FACTORY_RANDOM_STATE=`cat factory.state` ./manage.py test --testrunner p1.tests.MyTestRunner

Another one (branch factory-boy-seed) seeds the random generators, but the seed can be overriden via an environment variable:

class MyTestRunner(DiscoverRunner):
    def setup_test_environment(self):
        seed = os.environ.get('FACTORY_SEED')
        if seed:
            seed = seed.encode('ascii')
        if not seed:
            seed = b64encode(os.urandom(16))
        print('-- seed: %s' % seed.decode('ascii'))
        factory.random.reseed_random(seed)
        super().setup_test_environment()

Usage:

$ ./manage.py test --testrunner p1.tests.MyTestRunner
$ FACTORY_SEED=IPnHCxdWehAxz6ctbNvyPQ== ./manage.py test --testrunner p1.tests.MyTestRunner

Test runner can be put into the settings.

Aucun commentaire:

Enregistrer un commentaire