A/B Testing Mistakes and Solutions: How to Excel in Your Data Science Interview!

Подписаться 56 тыс.

Просмотров 27 тыс.

50% 1

In this video, I'm going to talk about a few mistakes that people often make when analyzing A/B testing results. You may often encounter these questions during an interview for data scientist positions. Want to know what these mistakes are and how to correctly solve those questions? Stay tuned!
A/B Testing Playlist
• A/B Testing in Data Sc...
🟢Get all my free data science interview resources
www.emmading.com/resources
🟡 Product Case Interview Cheatsheet www.emmading.com/product-case...
🟠 Statistics Interview Cheatsheet www.emmading.com/statistics-i...
🟣 Behavioral Interview Cheatsheet www.emmading.com/behavioral-i...
🔵 Data Science Resume Checklist www.emmading.com/data-science...
✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: www.emmading.com/coaching
// Comment
Got any questions? Something to add?
Write a comment below to chat.
// Let's connect on LinkedIn:
/ emmading001
====================
Contents of this video:
====================
0:00 Overview
0:56 Sample Ratio Mismatch
3:48 Debug Sample Ratio Mismatch
6:03 Violation of SUTVA
7:58 Changes in Users' Behaviors

Опубликовано:

3 авг 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 19

@TalebNassim888 3 года назад

We use chi-square to test if T:C =1:1

@ricardolee5597 3 года назад

Thank you for sharing. Super helpful. This is a really great video, especially for people new to AB testing

@snowguo1786 2 года назад

Thank you! Somehow I missed this video, this has a lot of info and content, I've write them all done. May come back and watch again.

@saeidsamizade6257 3 года назад

Great knowledge sharing. Thank you 👍

@tjc7524 3 года назад

Hi Emma, thanks for the great A/B testing series. Can you elaborate more why sample ratio mismatch will cause the invalidity of the test results from statistical perspective? Also, can we design the sample size rather than 1:1 in reality?

@poopah4497 2 года назад

Thank you

@saiteja1997 2 года назад

Great and Useful videos. While you have explained few ways to identify the causes for sample ratio mistach, What are ways to deal Sample Ratio Mismatch ? Is it required to re-run the experiment after resolving the bugs/issues ? Or can we make random sample to make both groups equal?

@dwardster 2 года назад

Hey Emma, great video! Quick question, for tiered significance levels, is it safe to have a lower significance level for a guardrail metric than for your primary metric? Based on your video, if my primary metric is CTR, and I expect that to increase, I would use a significance of 0.05, and if I track a guardrail metric like bounce rate and I don't think it will be affected I would use a significance level of 0.001. To me that doesn't seem safe because I could get a significant p=0.04 for CTR and an unsignificant p=0.003 for bounce rate, and the conclusion would be that the experiment should be implemented. I guess what I'm asking is how confident should I be in how a metric might change to be able to group it into a group using a smaller significance level?

@anirbansen7132 2 года назад

Thanks for the great video. I have a question w.r.t the sample size that you mentioned. With a 50:50 split on a website, there will be numerous sessions coming in. So, is the sample size the minimum number of sessions we need on each side to run a test. Or do we randomly sample X samples from all the incoming sessions, X being the sample size?

@hello-pd7tc 2 года назад

Hi Emma, thank you for your video! Can you help explain why would a segment of population (ios, android) would cause multiple testing?

@SuperLOLABC 3 года назад

Hey Emma, is reading Trustworthy Online Controlled Experiments book enough for an entry-level data scientist interviews? If not what else should I pair the book with for interview preparation? Amazing content as always!

@Crtg17 3 года назад

I have the same question. From my interview experience so far, I feel it is very important to learn how to tie the theory to the context. The book only gives you all the framework and potential caveats, but you have to think about how to "tailor" them according to the case in the interview. (Looking for Emma's answer as well!)

@junqichen6241 3 года назад

Hi Emma, I have a question about covariate imbalance for A/B test. If covariate imbalance was observed after the experiment ended, how would you tackle this issue? Thanks in advance!

@jt007rai 2 года назад

How do we use t-test for SRM? I thought we can only use chi-squared

@ekaterinavolkova4348 5 дней назад

"Trustworthy Online Controlled Experiments" by Ron Kohavi, Diane Tang, Ya Xu - Thanks for your recommendation

@sitongchen6688 3 года назад

Hello Emma, Thank you very much for this insightful video! I have follow-up questions for geo-based randomization to make control and treatment groups more independent. 1. For example, if we put all the SF users in control and Dallas users in treatment groups in case of Uber app. The feature based on the test wins, but how can we roll out this feature to all the markets, since the test is only done within those two specific markets? or we firstly roll out to the markets which are comparable to these 2 markets? 2. Do you mind doing a video explaining the common observational causal studies in case that the firm can not use AB tests to establish the casualty? Thanks a lotttt!!

@MrJioYoung 2 года назад

This is individual heterogeneity estimation. Causal inference methods might be useful. Or time randomization can be used for each location and the control / treated groups are split based on date.