Easy as ABC: A Quick Introduction to Bayesian A/B Testing in Python (Will Barker)

https://www.youtube.com/watch?v=nRLI_KbvZTQ&ab_channel=PyConCanada 

 

If you are just trying to break in the field like as data analyst, AB testing is one of the Bread and butter skills that you should know 

 

1. Bayesian Intuition

- It's pretty similar to how we naturally reason about things.

- We update our prior beliefs with new information.

- Describe hypotheses and data in terms of probability distributions.

- Data can be used as it comes - good for A/B testing

 

2.  How can we do Bayesian A/B Testing?

 

1. Understand our prior belief

    a. In the context of this A/B Test, it's simply asking "What do we think the click rate of our banner ads is typically like?"

    b. This can be subjective

 

2. Simulating an A/B Test

import numpy as np 
np.random.seed(42)
from scipy.stats import beta

group_size = 1000 # 1000 trials in both A and B groups
A_group, B_group = np.random.rand(2, group_size)

A_successes = sum(A_group < 0.15) # Our A variant has a true success rate of 15%
B_successes = sum(B_group < 0.20) # Our B variant has a true success rate of 20%

A_failures = group_size - A_successes
B_failures = group_size - B_successes

# A_posterior = prior + A's data
A_posterior = beta(A_successes + 8,
                   A_failures + 42)
                 
B_posterior = beta(B_successes + 8, 
                   B_failures + 42)

3. Calculate a p-ish value with Monte Carlo Simulation

 

import pandas as pd 

# 100,000 trials
n_trials = 100000

# draw 100k samples from A dist
A_samples = pd.Series([A_posterior.rvs() for _ in range(n_trials)])
B_samples = pd.Series([B_posterior.rvs() for _ in range(n_trials)])

# how many times did B outperform A?
B_wins = sum(B_samples > A_samples)

# percentage of B wins
B_wins / n_trials

~ 0.98, which can be likened to a one-side p-value of 0.02

 

3. Next Steps

- Python's Data Science stack makes A/B testing easy, not many lines of code required at all.

- Free A/B testing course offered by Google on Udacity.

- Check out Cameron Davidson-Pilon's Bayesian Methods for Hackers for more hands-on Bayesian projects with PyMC3

- Will Kurt's Count Bayesie blog is another great introduction to Bayesian stats. Do the projects in Python!