Cracking A/B Testing Problems in Data Science Interviews | Product Sense | Case Interview

https://www.youtube.com/watch?v=X8u6kr4fxXc 

Outline

1. What is AB Testing?

2. How long to run an AB Test?

3. Multiple testing problem

4. Novelty and primacy Effect.

5. Interference between variants

6. Dealing with interference

 

 

1. What is AB Testing?

a. A/B Tests (a.k.a Controlled Experiments)

    i. Used in industry to make decisions

    ii. Simplest form : Control A, Treatment B

        - Control group : Existing Features

        - Treatment group : New Features

    iii. Evaluate features with a subset of users

 

b. Appear frequently in Data Science Interviews

    i. Typically asked together with metric questions

    ii. Can include any A/B Testing component

        - Developing new hypothesis

        - Designing A/B Tests

        - Evaluating Results

        - Making decisions

 

 

2. Designing An AB Test

a. How long to run an AB Test?

    i. Use sample size and number of users

         - Eg 0.1% increase in revenue

    ii. Round the duration by weeks

 

b. Determine the sample size

 

c. How to estimate parameters?

        - We don't know before experiments

        - Use minimum detectable effect

 

d. Minimum detectable effect ( 1 - type 2 error)

    i. The smallest different matters in practice.

        (Eg. 0.1% Increase in Revenue)

 

 

3. Multiple Testing Problem

a. Test multiple variants of a feature

    i. Colors

    ii. Homepage Design

 

b. Sample Question

    i. 10 Tests are running with different landing page

    ii. 1 case won and p value < 0.05

    iii. Should you make the change?

 

c. The answer is NO 

    i. Should not use the same significance level

    ii. More than 2 variants

    iii. Probability of false discovery increases

 

d. Eg. There are 3 groups, what is the chance of at least one false positive?

    i. Pr(no false positive) = ( 1 - 0.05 ) ^ 3 = 0.95^3 = 0.857

    ii. Pr(at least 1 false positive) = 1 - Pr(no false positive)

                                                   = 0.143

    iii. Type I error over 14%