Question

Question 1 (4pt) Л fundamental requirement in A/B testing is that treatment and control samples are drawn at random from the same population. This means that the number of users in the samples should satisfy the expected ratio. If the actual ratio is different than the expected it means something is wrong with the sampling process. We call this situation Sample Ratio Mismatch (SRM). Most of the time SRM implies a severe selection bias, enough to render the experiment results invalid ("A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments", Pavel et al KDD 2017), Suppose you are running an A/B experiment for an e-commerce company and want to check for sample ratio mismatch (SRM). You are running an A/B experiment designed for 50% of the users to use the control group (current system) and 50% to use the treatment group (system with a new exciting feature). You are measuring how many users click in the buy button. NA 64454 (number of users in the control group) N 61818 (number of users in the treatment group) This situation can be modeled with a binomial distribution. The binomial distribution with proportions can be approximated to a normal distribution with a large number of users The standard deviation of the population can be calculated as: p(1-p) Where p is the expected probability for selecting a group if the randomization process is correct, and N is total number of users that were randomized 2a) Calculate the probability of the users being selected to the control group and the probability of the users to be selected to the treatment group 2 group is in the expected range. 2 with a confidence level of 99.9%. 0 b) State the null hypothesis to check if the randomization of the control c) Check if the control group has sample ratio mismatch using z-test )

Answer 1

a. pc- probability of the users being selected to the control group = 64454/(64454 61818) 0.5104 pt probability of the users being selected to the treatment group - 1 - pc-1- 0.5104 0.4896 b. Let pe-true proportion of the users being selected to the control group, Null hypothesis. Ho.р.-0.5 Alter native hypothesis, Hi : Pc 0.5 (c) 2-statistic = Pc0.5 , where, N 6445461818 126272 z-statistic = 7.3928 p-value = 2P(Z > 7.3928|Z ~ Ņ(0, 1)) = 2(1-Φ(7.3928)) 0.0000 < 0.001 99.9% confidence interval for pc s 1 - Pc el(0.5058,0.5151), Pc 20.00051 where, Z0.005 . 3.2905 Since p-value < 0.001 and 99.9% CI of pc does not contain 0.5 hence we reject Ho at 0.001 level of significance and conclude that the control group has sample ratio mismatch with expected ranqe