jeudi 16 février 2017

TFS Automated tests with multiple itterations show as passed even when the second itteration fails

I am working with an TFS 2017 environment with test agent 2015. Before this we had an TFS 2013 environment with test agent 2013 and MTM (this worked fine).

On the moment we have the following problem:

We run a set with around 40 tests, all of them have multiple iterations. if the first iteration fails we see this in tfs, the test status is set to failed this is perfect. However if the first iteration succeeds and the second fails the test case is set to passed in TFS. But if the second iteration fails we want the whole test to be set to failed. The way it is now it looks like almost all our tests pass however sometimes a lot of later iterations fail what means that we get false reporting.

When I open the .TRX file belonging to one machine I can see what iterations failed and which one succeeded.

so the problem in a nutshell:

if the first iteration of a test passes and the second one fails the whole test is set to Passed in stead of failed what gives us false reporting.

I have absolutely no idea what we are doing wrong. But now it gives is false information about our runs.

Is there anyone here that has experienced the same problem?

Any help would be really appreciated as I have not been able to find any information about this subject on google.

Aucun commentaire:

Enregistrer un commentaire