I have been developing the A/B testing procedure that I want to use in an upcoming experiment at work. Generally speaking I'm on team Bayes when it comes to statistical matters, and that's the approach that I will take in this case as well. In this (long) post I'll outline a method for Bayesian A/B testing that is largely built on some excellent blog posts by Evan Miller, Chris Stuccio, and David Robinson. In this post I'll summarize the results of Miller and Stuccio, ending up with a decision function for A/B testing based on the gain expected from making the change that is under consideration. I go a bit further than the referenced posts by deriving a measure of uncertainty in the expected gain that can be used to determine when a result is significant and overcome the "peeking" problem discussed in Robinson's post. Lastly I calculate the expected difference between the two procedures under test (as opposed to the gain which is the difference but only when the new procedure is better). Throughout I present some simulated examples to give a sense of the impact of the different parameters in the model and to illustrate some aspects of the model that one should be aware of.

more ...