dimanche 28 janvier 2018

Does JBehave based negative-scenarios grow exponentially?

In behavior based testing, it looks like the number of error scenarios grow exponentially.

As per Aslak Hellesøy, BDD was created to combine automated acceptance tests, functional requirements and software documentation.

In 2003 I became part of a small clique of people from the XP community who were exploring better ways to do TDD. Dan North named this BDD. The idea was to combine automated acceptance tests, functional requirements and software documentation into one format that would be understandable by non-technical people as well as testing tools.

Software development teams use JBehave as a tool for BDD testing (thanks to Dan North).

As there can be a lot of possible negative options; it looks like the number of negative scenarios in a JBehave test suite can grow a lot in numbers. Time taken to run test suite as well as to modify the product increases with these kind of growing scenarios. Specially, I feel that it is becoming hard to maintain as a documentation of the product.

I am not exactly sure whether this is an abuse of BDD/JBehave concepts due to misunderstandings from different teams; or may be that is the way it should be.

Let me explain this concern with an example.

Say an application has a behavior to order an item via a REST service.

PUT /order
{ 
    // JSON body with 3 mandatory parameters and 2 optional parameters
}

Happy scenario

  1. Invoke REST endpoint with correct values for all 3 mandatory parameters
  2. Invoke REST endpoint with correct values for all 5 parameters

Negative scenarios

There are a lot of negative scenarios that we can come up with.

Input value based scenarios

  1. Mandatory parameter 1 is set to null, with correct values for other two mandatory parameters (3 possible scenarios with each mandatory parameter)
  2. Mandatory parameter 1 is set to empty, with correct values for other two mandatory parameters (3 possible scenarios with each mandatory parameter)
  3. Mandatory parameter 1 is set to a value in invalid format, with correct values for other two mandatory parameters (3 possible scenarios with each mandatory parameter)
  4. Mandatory parameter 1 & 2 are set to null, with correct value for other mandatory parameter (2 possible scenarios)

Likewise, we can write 3^3 scenarios just for those three parameters; which grows exponentially with the number of parameters.

Then we can combine, optional parameters also into the equation and come up with more scenarios (say optional parameter with null, empty and invalid-format values).

Payment ability based scenarios

Based on available money, there will be scenarios.

Delivery location based scenarios

Based on delivery possibilities, there will be scenarios.

Question/Concern

I would like to learn more on whether all these negative scenarios (+more) should be part of JBehave based test suite? If that is the case, any advice/thoughts on how to make it more maintainable?

Aucun commentaire:

Enregistrer un commentaire