Netflix Product Designer | Navin Iyengar | Design Like a Scientist 

They found that 46% of people would like to see the catalog before signing up. Their hypothesis that showing the catalog would lead to more sign-ups is quite biased. Based on this finding, it should be something like "showing the catalog upfront will lead people to making a more informed decision of whether to sign-up or not". But of course Netflix is more biased towards formulating hypothesis which tend to result in measuring experiment success by looking only at business metrics and ignoring user needs. They should have then run surveys for each variant, to find out which variant helped people to make a better decision whether or not to sign up for Netflix, and tried to correlate that with behavioral data. But apparently they ignored this and declared the winning variant solely on what benefits the business goals (more sign-ups). This is not what UX Design is about - you should always work hard towards finding a balance between solving user needs AND business goals. Also, not every experiment should have a clear winner. A/B testing is more about learning than it is a tool to iterate on the product. I'm also curious to see what the retention rate for each variant was. It's one thing to increase sign-ups, but if people are signing up just to see the catalog and then churning because they don't find value in the content, did you really even help the business goals? From what was presented, it's not clear what other metrics were considered for the success of the experiment.

Users had indicated that they'd like to know what content they can get before they decide whether or not to get the subscription. So the measure of success here should be that it leads to better decisions, not more subscriptions. So the users were not wrong to mention that in the survey. It is important to differentiate that "Improving user experience in decision making" is different from "maximizing subscription rate" are 2 different hypothesis.

Wow! Simple and mega-informative! Really enjoyed the message : Act based on tested results, not assumptions Definitely bookmarked for sharing and future viewing.

Brilliant talk. I heard before AB testing divides the UX design world, I'm new and so far, from books and my studies, I gathered quanL and quanT research unbiased analysis (as much as is humanly possible ) are key to any successful product design specification. But this talk has gave me insight into how research heavily supports defining a hypothesis for subsequent testing. Thanks for sharing

HOT DAMN! this is one of the best UX - Product Design videos out there, including paid content platforms. Thanks for this internet, Navin, and awwwards

Wow really understood alot about how designers think. By going through so many iterations

This was super simple to understand and a great example to inspire UX Designer like me! thank you! for this content!

Great talk! 💯 I'm sure has an amazing data science team, but in a lot of companies, I think A/B testing causes teams to focus on what's easy to measure, and that doesn't necessarily correlates with what the person using the product is trying to achieve

wow.. this is kinda my first semster in Psychology plus the really basics of UX design ..

like steve jobs said, people dont know what they really want

Awesome, great easy to follow advice.

thank you for sharing your knowledge, Navin!!! :)

Amazing! Thank you!

This is a great video to understand their practical research methodologies and how to synthesize and use the results to understand users. Also very easy to understand the concept of A/B testing. Still lots of designers misuse what a/b testing means.

Thanks for Sharing. Learnt some new approaches to my product.

I believe there are some natural explanations to why not showing specific content worked better. There are two kinds of streaming users: 1) those who are looking for something entertaining but don't have any specific in mind, and 2) those who know which specific movie/series they want to watch. 1) For those who visit Netflix without a specific movie/series in mind and just want to explore if there's anything worth watching there, it's obvious that forcing them to sign up in order to do that, will lead to more sign ups. 2) Human beings judge things by their subjective perception. So if they're allowed to browse all the specific content and few of the titles are interesting to them, they will deem the value of the service as lower (-> decreasing their probability to sign up). On the other hand, by keeping it abstract then each person can imagine that there will be more content to their liking. (in addition to the one specific series/movie they visited Netflix for in the first place).

please stop autoplaying videos with sound dear lord have mercy

Amazing talk.This is inspiring and informative, Thank you!

Excellent talk

love this, so informative.

Awesome

This was really helpful! Thank you.

dope!!

I don't have Netflix, but excellent talk! Keep rocking! 🙂👍

Amazing

Awesome!

So much testing and thinking while I never know the movie library - it seems random, the search works in misterious ways, often My List doesn't save what I want to save, Recommendations are just insane - I never watch Bollywood movies yet I get them like I'm a fan (?), the movies I watched are constantly in front everywhere instead of being at the last of any List - seriously why do I want to keep rewatching the same movie over and over.... and the list goes on. But really helpful the stuff about Galilei, really smart and relevant!

Very WELL explained buddy. It is brilliant presentation ❤

Netflix doesn’t run on my s9 plus. I tried all the possible solution. Still doesn’t work. And I love it.. ✌

Oh can you please Auto mute the previews/Trailers on atv. Its so annoying!!! I think this is a Good example where u actually have to Listen to your Users and Not ab Test.

Wow, this piece of information is really handy. Thank you :-)

Galileo never did this expperiment, because observations would have been insufficient and dangerous. What he did is to throw the same mass through molasses, water and air to extrapolate how it'll behave in a theoretical empty space. Then he made a thought experiment that concluded masses fall at the same velocity, otherwise a string attached to a heavy and a light mass would cause a parachute effect while the system would fall faster as it's more massive than splut masses, therefore it'd be nonsensical to consider masses fall differently.

The best

This is completely misleading. Nerflix' business model is getting as much money as possible and to assure the longevity of the income stream. This is how their app is designed. They want you to keep watching any content they can keep you watching. Example : you talk with your significant other about watching a movie for date night. The conversation prior to using the app is about seeing "something we haven't seen before" and then you both start thinking about what you have seen recently mixed with a discussion about recent popular movies that "should be on Netflix by now". You get to Netflix and you can not answer those questions using the app. It does not allow you to find recently released movies sorted/filtered by popularity in the box office and movies you already have watched on their platform. The user experience categories, favorites type lists or searches you have the option of searching don't let you explore your desires according to your values. Instead you get results that frustrate you to the point you pick something one of you probably already watched. Netflix just got you to effectively pay them for another two hours this month. Combine that with the rest of the time you use their platform this month, and you have a reason to keep the service because you associate all that time with value, never really being delighted by being able to find what you want when you want it, but instead see a lot of "good enough" content. Does Walmart show the latest box office hits mixed in with everything else alphabetically? THAT'S why they need so much experimentation WITH a substandard UI/UX - it's about quantifiable cashflow forecasting.

good one

So basically it's more about getting feedback to improve your product.

An effective A/B test should ask "what can we learn" not "does a user sign up". Navins test proves not that all pre-sign up catalogues should be shrouded in mystery, they imply a need for further testing on the content discovery experience.

Yes I watch Netflix on my Apple TV most of the time and the experience is horrible since day one So many things I hate about its UI/UX on tv

This is a very basic introduction to what product designers at most companies do every day. I hope they teach this to new designers, as what's shared in this video is useful to know -- but NOT at all a new concept. It's been quite a standard approach for 2 decades now

This guy's understanding of p values would make good stats teachers upset.

Wow what a revelation. Great video. Can someone tell me which font's used in that slideshow? It looked like Helvetica Now, but I may be wrong

how can i find subtitle Please Turkey and Russian language

Is it by design that movies don’t have trailers while TV shows do?

Lets designers be designer and scientists be scientists. I wouldn't want either to work like anyone else. Doing A/B test doesn't make a designer scientist.

Navin's null hypothesis was "A/B testing is the best testing for experimentation". He should have designed a between subject comparison using T test to prove his null hypothesis. After he skipped the evidence by just making this statement, now I don't feel like watching the rest of the video.

thank you all very much

How about science like a designer?

This reminds me of Evolution

Thanks for posting this! Just to note: the close up camera shot moves too much, making me dizzy.

Can we end this “edit out the static” editing?

Lol his summary of philosophy at the beginning is ludicrous.

Interested to know how do they evaluate big experiments like high level hypothesis on Personalization, It is difficult to analyse why that experiment is better.

Dude doesn't it bother you that large number of your users never find the titles they are looking for...? I wouldn't try to sell that as an achievement if I were you :-) Just today I tried to watch that movie Netflix ui promoted the day before, I never managed to find it on obvious reasons. You guys really make me laugh!!! (...And curse sometimes)

receipt of information assurance: left at4:00

What I don't get is how all these amazing designers thought that Auto preview was a good idea? did you guys tested it? Please, just do an a/b test with and without the auto preview and let's see which one is better.

I can't stand the user experience of the Netflix app. So should I even listen?

Was his audience in this keynote 5 year olds?

This sounds great on the surface. But, optimizing for subscription? Just make it hard to unsubscribe! Hello, UX dark patterns.

13:03 i don't get it, why would you have this hypothesis?

It's like I always say: Never believe in your users. They're lying to you.

audio could be better

nice shoulder bro

Anyone that knows what A/B testing is can skip this one.

Sorry, this isn't science. And it isn't design. This is user experience.

You know what, make content people love. Your Hindu-phoic content is the worst. You might have some number of customers who appreciate Hinduphobic content. The truth is it doesn't scale and you will never reach the hearts of India by making anti-India content. This is the main reason I never bought Netflix subscription even though I can easily pay. I can't pay for something that harms me and my people. "Go to India and find out their unmet needs"=> Their aspiration to self-loathe. So lets make stories where cultural Hindus are presented as biggest villains on the planet. Even when Laskar Taiba attacks India, show that Hindus masterminded attack on themselves. Netflix really read our people well. EDIT: Added context

Bill Nye isn't even a scientist lol

Somebody design Netflix?

starts off interesting and about customer experience, then ends on a bitter note on how to better sell a product people might actually not even want in the first place. "if we didn't show what we got, more people sign up" is less about customer experience but more about sales tactics.. kindof a bad example for an otherwise interesting talk.

His insta Id@

Netflix recos are useless. I constantly see exact same movies recommended. Thrre is very less customization. And what about search.. Netflix searching sucks. Prime video is no better. Wonder why such a critical feature has been constantly overlooked.

Hate people that talk in a corporate way

His fundamental understanding of science is questionable. There is a difference between physical science and social science. You can't compare experiments with stones and people in the same breath.

Scratching ass haters comments down there.

some wrong information but ok :)

This isn’t product design it’s web design. Why is ui now labelled as product design....

People dont tell the truth? Branding gurus have been saying this for ages. Cool guy - not very valuable information.

Netflix promotes sexuality, so I guess what's the benefits it is giving too our society

Nothing new

Uuughhhh what a COMPLETE WASTE of time. I had to skip many parts and basically watch 1minute.

Netflix sucks. Hate the recommendations. Super irritating content

Omg worst talk waste of time, no he really works with netflix ,such a team

I like taking advise for people wearing business attire as oppose to some sweater or whatever that cliche atire you see in shows like silicon valley So cliche !! I'm done listening ..!!