samedi 4 avril 2015

Test all build options permutations in CTest

The main CMakeList in my project has lots of options to enable and disable optional features (mostly booleans, a few strings), which look like:



option(Feature1 "Enable feature 1" OFF)
option(Feature2 "Enable feature 2" OFF)
option(Feature3 "Enable feature 3" OFF)
...


The main test suite for the project tests all the features at once, hence it is built with all features enabled, which is easy to do. The problem is that the project should build with any combination of options. I would like to integrate that to my testing, using something like:



ctest_start(Nightly)
foreach(optionset ${ALL_PERMUTATIONS})
set(CTEST_BUILD_OPTIONS ${optionset})
ctest_configure()
ctest_build()
endforeach()
ctest_submit()


What is the best way to plough through all the combinations in a CTest script? I currently have about 7 options (~400 permutations), and there will be more in the future.


My current thoughts are either:



  1. Have one massive loop nest in the CTest script, which is not ideal at all

  2. Create a bash or python script to generate a ctest script with all the permutations

  3. find (or create) a cmake script/module/tool to do this


A good solution should report the broken combinations elegantly and make it very easy to add/remove options for testing.


Is there an existing tool to do this or do you know a project which does this kind of testing?


Aucun commentaire:

Enregistrer un commentaire