This is really helpful. I learned a lot. Learning more about A/B testing for UX and the natural variance is a new and very interesting concept to me. Thank you!
Could you do a video explaining lift? I’ve only seen it used in the context of data mining and association rules, but it seems like you’re using it different here and I’m not sure what this measure represents here.
If larger changes are made, results are less likely to be close. From my experience people running tests will make little tweaks to the site and get results that are very close. The bigger the change the bigger the difference in the result for most tests.
If you are peeking at your data for the right reasons, then you aren't suffering from peeking, but you are benefiting from peeking. Check out this video which explains that more. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-cS072qIYhBg.html
Are you talking about running the natural variance test? You would want to get more data than just a day, but one day is better than nothing. I guess it depends on your risk tolerance for having an unknown higher or lower variance amount. Me personally, I prefer more data so I can be more confident in the numbers I am using.
It really depends on your organization. Some organizations have have a cross functional team of experts to support this work. In most smaller organizations a person might wear multiple hats and do many functions.
Yes that would be the extreme example and if it was that low you would need to see the trend and lift doing well too. That one data point in isolation isn't enough.
