mardi 29 novembre 2016

CodedUI: Logging inside AssemblyCleanup

We have a Coded UI test framework that allows test authors or result analyzers to mark a test as failing to a specific bug. Basically, if a test is marked this way, we store the test name and the bug it fails to in our TestContext object, and eventually at the end of the run we want to output a list of tests that failed to bugs.

We had been printing this information on a test-by-test basis using TestCleanup, but this leads to confusion. As more tests executed inside the run, the list would increase when it came across expected failures, and later tests would output more data than previous tests. For example, if I ran 10 tests, and test 2 and 8 failed to an expected bug, the first result log would omit that section entirely, the 2nd through 7th result logs would show one expected failure, and the 8th through 10th would show two expected failures. In other words, some would have a section that looked like this:

Test A failed to Bug 1

And some would have that same section that looked like this:

Test A failed to Bug 1

Test B failed to Bug 2

This made analyzing a run confusing, since the true list of expected failures would show up only on the last test that ran, whatever that one happened to be.

So doing this in TestCleanup is impractical, and I don't want to do it in ClassCleanup because we have a ton of classes that hold our tests -- it's our primary method of organizing tests. That leaves me with AssemblyCleanup.

I got AssemblyCleanup to execute, and I can call logging code in it. However, I can't find the logged output anywhere. If I run from the command line using vstest.console.exe and generate a .trx file, the run details view doesn't show me any logged output. If I run a test individually within Visual Studio, I don't even get a run details view. And the AssemblyCleanup logging output doesn't get put into individual test logs, as I would expect.

Ultimately my question is this: is there any way to log information inside the AssemblyCleanup method of a Coded UI test and view it somewhere, or is that a completely pointless thing to do?

Aucun commentaire:

Enregistrer un commentaire