Test Engineering Post

But now the analyst faces a choice of metric by which to compare pA and pB. E.g., in order to

trigger a recommendation, do we want the difference between success probabilities to exceed 0.1

1Paraphrasing from Klugman et al “Loss Models”, 2nd Ed. Wiley (2004), pg. 419: “…the process must end with a

winner. While qualifications, caveats etc. are often necessary, a commitment is required.”.

2

“Fail Fast and Learn”, pg. 79, A/B Testing, Dan Siroker and Pete Koomen, Wiley (2013)

1

—i.e. pA −pB > 0.1— or do we want the success probability to lift by 20% —i.e. pA > 1.2 ∗ pB?

Different metrics (and acceptance values) define different lines in two-dimensional probability

space {(pA, pB)} (See Fig.1, where we have plotted the lines defined by a metric value for each

of three metrics: Probability Difference, Probability Lift and Odds Factor, which we will discuss

shortly.). We recommend A over B if the observed probabilities lie below and to the right of the

Figure 1: Probability Comparison Metrics on 2D Probability space

chosen line.

from NerdWallet
https://www.nerdwallet.com/blog/engineering/test-engineering-post/