I wish you had mentioned the feedback would be on another website at the beginning of the interview. Also, is this person who is being interviewed a real candidate or someone who is already working as a DS at Amazon (or somewhere else?)
He A/B tested it and found out that more peole will watch the video, and more people will move to paid subscription. Major dick move, but this part isn't measured and not important.
Although it would seem more straightforward at first blush, it wouldn't be a particularly favorable metric in this case. See, if a certain user makes several searches and/or purchases (which is inevitable, in practice), the underlying observations that contribute to the metric are correlated with one another, leadning to a violation of the t-test's assumption of observations' independence. Think of an extreme case to understand the intuition; e.g., if all purchases and recommendations were to come from a single customer.
What does total recommended products means what if treatment one is able to recommend tax more products and the purchase rate out of these recommended product is X2 system one the control group is able to recommend only 1/10 of the product and the purchase rate is half of the system too so in this case the purchase rate for system one is higher but more users have purchased in the total revenue is higher in system B
Z-tests are statistical calculations that are generally used to compare population mean to a sample mean. The z-score tells how far, in terms standard deviations, a data point is from the mean or average of a data set. So a z-test compares a sample to a defined population. Like z-tests, t-tests are calculations used to test a hypothesis, but they are used mostly when we need to determine if there is a statistically significant difference between two independent sample groups. In other words, a t-test asks whether a difference between the means of two groups is unlikely to have occurred because of random chance.
