I am new to testing in Java and I would like to understand better how test quality is measured and how to measure it properly. My background is silicon design-verification (SystemVerilog/UVM, verification of Verilog). In SystemVerilog/UVM tests of hardware are divided into mulitple subcategories of coverage such as code coverage, toggle coverage (for registers), assertions, coverpoints where you can cover transitions (state machines etc.), divide tested values into bins of particular ranges, etc.).
In silicon verification everything starts from the verification plan that defines the coverage you need. The verification Plan is a link between product requirements (specification, functionality) and actual mesurable coverage metrics. This type of coverage provides reasonable high level of confidence. You can present this type of coverage metrics to customer and they will commonly understand it. It is an industry-wide accepted approach, understood and followed.
In Java testing I fail to fully understand the metrics and planning of tests; In discussion with customers - what metrics would you use?. Based on various stackoverflow discussions I would be biased to say that there is no strong common agreement as to what is a comprehensive testing of Java and there is no clear expectations. In my view it boils down to a couple of significant differences between software and silicon developement and the major one could be cost of a single bug.
software silicon
--------------------------------------------------------------------------
cost of bug: small-large | large (requires another tape-out)
soonest next release: few weeks | few months (tape-out)
basic unit of test: testcase | testcase
coverage metrics: code, | code (block, toggle, expression, FSM),
any other? | functional (covergroups, coverpoitns),
| assertion/formal
suppliers of tools: many | two-three main suppliers
cost of toolS: many free | expensive
methodologies: ??? | Specman 'e', UVM; constrained
| SystemVerilog constrained-randomized
| test
It appears to me that there are no mainstream approaches as how to test. Code coverage is something that most often mentioned but from my silicon practice it does not necessairly truly reflect what you want and really NEED to test. So just to summarize my question:
- What are the most commonly used test metrics you would show to your customers?
- Is it fair to say that there is no common expectations from software customers about coverage metrics?
Many thanks for unfolding the mystery.
Aucun commentaire:
Enregistrer un commentaire