lundi 4 janvier 2016

JS code coverage across multiple test classes

Background

I'm working on a single-page application with a JavaScript code base that is well-tested at the unit level, and want to add a functional test suite to verify application behavior closer to the level of what the user sees. At this time, I'm not looking at full end-to-end testing (I'd expect to stub/mock interactions with the back-end) but that may be a goal later on.

Our build is run by npm; gulp/grunt are not currently in use but may be adopted soon as our build complexity grows. We are using Jasmine and Blanket for testing and coverage currently, but for purposes of this question I'm fairly tool-agnostic.

Goal

What I'd like to do is generate code coverage that is comprehensive across both categories of unit tests. That is, if I have a line of code that is exercised during functional testing but isn't exercised in unit tests, my reporting should still mark that as "covered." (If I can get more or deeper information about how code gets exercised, great, but not needed.)

The underlying goal (since XY problems are so common on SO) is to create a forcing function for test completeness ("code coverage must be above X%") without forcing redundancy (as might happen if coverage was measured separately for both) or introducing a false preference for one category of testing (as might happen if coverage was measured only for one category of testing.) There's value to testing the same code in different circumstances, of course, but we are resource-limited and want to focus on the tests that will add the most value.

Question(s)

The short question is "how do I do this?" I haven't found any documentation or examples of this being done. I feel like the answer is "just stick istanbul in front of X which runs both kinds of tests", but don't really know what X looks like.

Some more specific questions I have in mind:

  • Are there any examples which could be followed of comprehensive coverage being obtained for testing across multiple categories?
  • What are the constraints one must accept to make this kind of testing feasible?
  • Am I wrong to think of unit tests and functional tests as separate suites being run in different ways? (The problem becomes trivial if functional and unit tests are not separate suites, but are distinctions one makes on a test-by-test basis. But that's not the sort of test suite I'm used to seeing.)
  • Is there a reason not to do this? (The fact that I can't find examples of this being done feels conspicuous to me.)

I'm reasonably familiar with the tools that are out there in this category (Karma, Mocha, Jasmine, Blanket, Istanbul, etc...) so I'm not looking for recommendations, per se, but rather some guidance on how to achieve this kind of configuration. However, I'd certainly be interested to hear if some tools are better- or ill-suited for this sort of usage.

Aucun commentaire:

Enregistrer un commentaire