I'm working on integrating performance tests with CI/CD infrastructure. My performance testing tool is JMeter and CI server is Jenkins. Both can do their jobs, but when it comes to integrating performance tests in CI/CD pipeline, things are no longer so trivial.
To have proper deployment pipeline CI server needs to know when a performance test build should be considered passed or failed. Verification of mean average of response times is not a good option - completely different SLAs can apply to different types of transactions executed as part of the same JMX file. Asserting on mean average of response times for particular transaction type is a far better option, but it is still far from a perfect solution. This won't tell us e.g. if response times for same type of transaction are increasing (which can has something to do with memory leaks) or decreasing (which can be a blessing of server-side cache). For that reason, relying only on mean avg response times can create false confidence in software quality.
I analysed a couple of tools, including JMeter Maven Analysis Plugin and Jenkins Performance Plugin. None of them seems to offer what I'm looking for.
In pre-CI era, performance tests were executed late in the development lifecycle and analysed by a human being. I wonder if anyone came across any advance enough tool, which can make it possible for CI server to reliably determine if perf test build should be marked as passed or failed, without verification of results by a human?
Aucun commentaire:
Enregistrer un commentaire