|Will a winner be determined before I stop the experiment?||Honn||6/15/12 1:46 AM|
I have an experiment where it claims that one page has 96% chance to beat the original, but it still says that a winner cannot be determined yet.
Is this because it never determines a winner while the experiment is still running, or is it because it requires a higher visitor count or probability before it determines a winner?
Or to speak plainly, when should you stop a test? When analytics says it has found a winner or when the chance to beat the original is high enough?
|Re: Will a winner be determined before I stop the experiment?||TylerKatzenberg||6/15/12 1:05 PM|
Google has a little information on this topic here: http://support.google.com/analytics/bin/answer.py?hl=en&answer=1745154
Unless your site gets a ton of traffic and you have seen in past A/B tests that you can get meaningful results sooner, I would recommend sticking to the two week minimum. I've seen cases on my site where a variation had a greater than 90% chance of beating the original after four or five days, only to have that advantage completely wiped out a few days later.