The term "brittle" testing is thrown around a lot, but I can't actually find it defined anywhere. I've found plenty of opinion pieces arguing whether brittle testing is good/bad/inherent but what I'm really wondering is what IS brittle testing?
I assume brittle tests are tests that break easily, but even that doesn't answer my question. I'm not sure if a "broken" test is (1) a test that fails in the test environment when you change some underlying dependency, or (2) a test that succeeds in the test environment despite changing some underlying dependency.
For example:
(1) You change a string validation method to return the string on success, when it previously returned a boolean. This then causes tests for all of your account signup methods to fail, because they all call the string validation method (which now returns something new). Is this a "brittle" test?
(2) You change a string validation method to return the string on success, when it previously returned a boolean. The tests for all of your account signup methods continue to succeed in the test environment, because they all stub the string validation methods to return the value they expect (even though this value is now different than what the method actually returns). Is this a "brittle" test?
I guess it's important to note that I'm thinking in terms of unit testing here, but if the definition warrants talk of other types of testing then I'm all ears.
Aucun commentaire:
Enregistrer un commentaire