How do I collect tests in py.test that are the result of dynamically creating unittest TestCases and TestSuites?

tl;dr: Is there a way to collect dynamically generated unitest.TestSuite contained tests with py.test or am I out of luck?

I've developed a tool called gabbi which allows declarative testing of HTTP APIs using a YAML format. It's similar in concept to pytest-yamlwsgi but specifically designed to work in the testrepository and subunit driven world of OpenStack development.

However before I entered that world I used pytest and I still prefer it. I'd like my tool to be usable with it. This is proving more difficult than I hoped.

In gabbi, each YAML file is an ordered sequence of HTTP requests. Each request is a TestCase, the entire set of tests is kept in a subclass of TestSuite which has two purposes:

• to run the tests in order


• to establish any so called "fixtures" that are stated in the YAML files

On the unittest side of the world these tests are built and loaded by the load_tests protocol.

This branch of TiddlyWeb show a sample of me adding gabbi tests to TiddlyWeb. I've experimented in that branch to get it working in py.test. I can almost but not quite get it working by adding something like the following to the test_tiddlyweb file:

def test_generate_tests():
    from unittest import defaultTestLoader, TestResult
    tests = load_tests(defaultTestLoader, None, None)
    # when calling a unittest.TestCase we need to provide a
    # TestResult to track the results
    result = TestResult()
    for test in tests:
        yield test, result

(I also need to make sure that the underlying TestSuites are hashable with a __hash__ function.)

This will actually run the tests but there are issues:

• Failures are not tracked: py.test says everything passes.


• The count of tests is the count of TestSuites, not count of TestCases

I've tried using py.test hooks in various ways during collection, running and reporting to with some impact on collection but no success on running and reporting. The problem seems to be that the unittest handling built into py.test is not ready to deal with nested nor yielded TestSuites. (load_tests returns a TestSuite containing one TestSuite per YAML file, each HTTP request in a YAML file is a single TestCase.)

Note that the ideal solution would not change the existing gabbi code that is creating the TestSuites and TestCases, instead it would transform the results of that creation into something usable. If that's not possible, I'd love to hear about more invasive solutions as well. Thanks!