vendredi 15 janvier 2016

Approach for determining winner in A/B test with more than 2 version?

I've heard people take the top performing variation (say there are 4 variations total), and then do either a t-test or a chi-square test against each of the other versions.

My issue with this is first the multiple comparison problem, i.e. that the Familywise Error Rate gets, but there are way to correct for this.

The other issue is that selecting the top performer creates some sort of bias and might violate the assumptions of the test (it wasn't randomly selected). So if I have 4 versions of a test, what is the correct approach to compare their performance for purposes of trying to identifying the best performer with statsitical signficance?

Aucun commentaire:

Enregistrer un commentaire