lundi 30 avril 2018

TestNG 6.10+ priority more important than preserve-order

I'm just working with some legacy code using TestNG framework in the 6.9.x version which I have to upgrade to the newer version of the framework - at least 6.11 . The problem is, many @Test annotations are marked with extra priorities attributes (@Test(priority = x)). But I've got a problem after change introduced in TestNG in the 6.10 release, namely:

New: Hierarchy on order features (from less important to more important): groupByInstance, preserveOrder, priority, dependsOnGroups, dependsOnMethods

Let's see an example of two test classes each containing three test methods with defined priorities:

First class:

public class TestClass1 {

    Logger LOG = Logger.getLogger("logger1");

    @Test(priority = 2)
    public void methodB() {
        LOG.info("Method 1B");
    }

    @Test(priority = 3)
    public void methodA() {
        LOG.info("Method 1A");
    }

    @Test(priority = 1)
    public void methodC() {
        LOG.info("Method 1C");
    }
}

The second one:

public class TestClass2 {

    Logger LOG = Logger.getLogger("logger2");

    @Test(priority = 1)
    public void methodC() {
        LOG.info("Method 2C");
    }

    @Test(priority = 2)
    public void methodB() {
        LOG.info("Method 2B");
    }

    @Test(priority = 3)
    public void methodA() {
        LOG.info("Method 2A");
    }

}

... and XML:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
  <test name="Test" preserve-order="true">
    <classes>
      <class name="com.test.radek.testngtest.TestClass2"/>
      <class name="com.test.radek.testngtest.TestClass1"/>
    </classes>
  </test> <!-- Test -->
</suite> <!-- Suite -->

After running such an XML suite, tests execution order is various for different TestNG releases:

For 6.9.x:

Apr 30, 2018 11:46:19 AM com.test.radek.testngtest.TestClass2 **methodC**
Apr 30, 2018 11:46:19 AM com.test.radek.testngtest.TestClass2 **methodB**
Apr 30, 2018 11:46:19 AM com.test.radek.testngtest.TestClass2 **methodA**
Apr 30, 2018 11:46:19 AM com.test.radek.testngtest.TestClass1 **methodC**
Apr 30, 2018 11:46:19 AM com.test.radek.testngtest.TestClass1 **methodB**
Apr 30, 2018 11:46:19 AM com.test.radek.testngtest.TestClass1 **methodA**

For 6.10 and higher:

Apr 30, 2018 11:47:37 AM com.test.radek.testngtest.TestClass2 **methodC**
Apr 30, 2018 11:47:37 AM com.test.radek.testngtest.TestClass1 **methodC**
Apr 30, 2018 11:47:37 AM com.test.radek.testngtest.TestClass2 **methodB**
Apr 30, 2018 11:47:37 AM com.test.radek.testngtest.TestClass1 **methodB**
Apr 30, 2018 11:47:37 AM com.test.radek.testngtest.TestClass2 **methodA**
Apr 30, 2018 11:47:37 AM com.test.radek.testngtest.TestClass1 **methodA**

So on these two example we can see that starting from TestNG 6.10 the preserve-order is completely ignored by priorities.

I saw a discussion regarding this preserve-order / priority correlation and now I'm no really sure how to handle this problem.

Main question:

Does TestNG have any mechanism which allows to execute tests according to priorities defined in @Test but grouped by a class (and the class order is defined in the XML suite) - so exactly as it was before TestNG 6.10 ?? What is the most correct way to migrate legacy tests relying on priorities and preserve-order to the new TestNG ?? Replacing the "priority" logic with dependsOnGroups could be very time-consuming for 1000+ test methods :/

Aucun commentaire:

Enregistrer un commentaire