Тёмный

Mutual Information, Clearly Explained!!! 

StatQuest with Josh Starmer
Подписаться 1,2 млн
Просмотров 85 тыс.
50% 1

Mutual Information is metric that quantifies how similar or different two variables are. This is a lot like R-squared, but R-squared only works for continuous variables. What's cool about Mutual Information is that it works for both continuous and discrete variables. So, in this video, we walk you through how to calculate Mutual Information step-by-step. BAM!
English
This video has been dubbed using an artificial voice via aloud.area120.google.com to increase accessibility. You can change the audio track language in the Settings menu.
Spanish
Este video ha sido doblado al español con voz artificial con aloud.area120.google.com para aumentar la accesibilidad. Puede cambiar el idioma de la pista de audio en el menú Configuración.
Portuguese
Este vídeo foi dublado para o português usando uma voz artificial via aloud.area120.google.com para melhorar sua acessibilidade. Você pode alterar o idioma do áudio no menu Configurações.
If you'd like to support StatQuest, please consider...
Patreon: / statquest
...or...
RU-vid Membership: / @statquest
...buying my book, a study guide, a t-shirt or hoodie, or a song from the StatQuest store...
statquest.org/statquest-store/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
0:00 Awesome song and introduction
2:39 Joint and Marginal Probabilities
6:19 Calculating the Mutual Information for Discrete Variables
13:00 Calculating the Mutual Information for Continuous Variables
14:10 Understanding Mutual Information as a way to relate the Entropy of two variables.
#StatQuest #MutualInformation #DubbedWithAloud

Опубликовано:

 

23 июл 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 158   
@statquest
@statquest Год назад
To learn more about one common way to create histograms of continuous variables, see: journals.plos.org/plosone/article?id=10.1371/journal.pone.0087357 To learn more about Lightning: lightning.ai/ Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
@SelinDrawz
@SelinDrawz Год назад
Thank u daddy stat quest for carrying me through my university course
@statquest
@statquest Год назад
Ha! :)
@faizalrafi
@faizalrafi 9 месяцев назад
I am binge-watching this series. Very clear and concise explanations for every topics given in the most interesting way!
@statquest
@statquest 9 месяцев назад
Glad you like them!
@PunmasterSTP
@PunmasterSTP 4 месяца назад
Same here!
@Fan-vk9gx
@Fan-vk9gx Год назад
Super! I have been struggled between copula, mutual information, etc. for a while, that is exactly what I am looking for! Thank you, Josh! This video is really helpful!
@statquest
@statquest Год назад
Glad it was helpful!
@kenmayer9334
@kenmayer9334 Год назад
Awesome stuff, Josh. Thank you!
@statquest
@statquest Год назад
My pleasure!
@dragoncurveenthusiast
@dragoncurveenthusiast Год назад
Your explanations are awesome!
@statquest
@statquest Год назад
Glad you like them!
@raizen74
@raizen74 Год назад
Superb explanation! Your channel is great!
@statquest
@statquest Год назад
Glad you think so!
@user-rf8jf1ot3t
@user-rf8jf1ot3t Месяц назад
I love this video. Simple and clear.
@statquest
@statquest Месяц назад
Thanks!
@MegaNightdude
@MegaNightdude Год назад
Great stuff. As always.
@statquest
@statquest Год назад
Thank you very much! :)
@Maciek17PL
@Maciek17PL Год назад
Amazing as always!!!
@statquest
@statquest Год назад
Thank you!
@arash2229
@arash2229 Год назад
Thank youuuu. you explain everything clearly
@statquest
@statquest Год назад
Glad it was helpful!
@KatanyaTrader
@KatanyaTrader Год назад
OMG i never see this channel, how many hours would be saveeddd.. new subs here, thanks alottt for ur vids
@statquest
@statquest Год назад
Welcome!
@smilefaxxe2557
@smilefaxxe2557 3 месяца назад
Great explanation, thank you! ❤🔥
@statquest
@statquest 3 месяца назад
Glad it was helpful!
@stepavancouver
@stepavancouver Год назад
An interesting explanation and nice sence of humor 👍
@statquest
@statquest Год назад
Thank you!
@zachchairez4568
@zachchairez4568 Год назад
Great job! Love it!
@zachchairez4568
@zachchairez4568 Год назад
Liking my own comment to double like your video :)
@statquest
@statquest Год назад
Double bam! :)
@sasha297603ha
@sasha297603ha 4 месяца назад
Love it, thanks!
@statquest
@statquest 4 месяца назад
Thank you!
@mohammadeslami7462
@mohammadeslami7462 22 дня назад
Superb!!! I recommend this channel to everyone.
@statquest
@statquest 22 дня назад
Thanks!
@pablovivas5234
@pablovivas5234 Год назад
Keep it up. Great content
@statquest
@statquest Год назад
Thank you!
@PunmasterSTP
@PunmasterSTP 4 месяца назад
Mutual information, clearly explained? More like "Magnificent demonstration, you deserve more fame!" 👍
@statquest
@statquest 4 месяца назад
Thanks! 😃
@bernardtiongingsheng85
@bernardtiongingsheng85 Год назад
Thank you so mcuh! It is really helpful. I really hope you can explain KL divergence in the next video.
@statquest
@statquest Год назад
I'll keep that in mind.
@felipevaldes7679
@felipevaldes7679 Год назад
I love this channel
@statquest
@statquest Год назад
BAM! :)
@felipevaldes7679
@felipevaldes7679 Год назад
@@statquest lol, very on-brand too.
@Geneu97
@Geneu97 5 месяцев назад
Thank you for being a content creator
@statquest
@statquest 5 месяцев назад
Thanks!
@PunmasterSTP
@PunmasterSTP 4 месяца назад
Not just a creator of any content either. A creator of *exceptional* content!
@VaibhaviDeo
@VaibhaviDeo Год назад
you are the best god sent really stay blessed
@statquest
@statquest Год назад
Thank you!
@user-sn4ni3np8h
@user-sn4ni3np8h 5 месяцев назад
Two sigmas are like two for loops, such that, for every index of outer Sigma, the inner sigmaales a complete iteration.
@statquest
@statquest 5 месяцев назад
bam!
@pranabsarmaiitm2487
@pranabsarmaiitm2487 Год назад
awesome!!! Now waiting for a video on Chi2 Test of Independence.
@statquest
@statquest Год назад
I'll keep that in mind.
@ian-haggerty
@ian-haggerty 3 месяца назад
Entropy === The expectation of the surprise!!! I'll never look at this concept the same again
@statquest
@statquest 3 месяца назад
bam! :)
@Lynxdom
@Lynxdom 9 месяцев назад
You got a like just for the musical numbers!
@statquest
@statquest 9 месяцев назад
bam!
@murilopalomosebilla2999
@murilopalomosebilla2999 Год назад
Excellent content as always!
@statquest
@statquest Год назад
Much appreciated!
@isaacfernandez2243
@isaacfernandez2243 Год назад
Dude, you don't even know me, and I don't really know you either, but oh boyy, I fucking love you. Thank you. One day I will teach people just like you do.
@statquest
@statquest Год назад
Thanks! :)
@adityaagrawal2397
@adityaagrawal2397 8 месяцев назад
Just started Learning ML, am assured now that the journey would be smooth with this channel
@statquest
@statquest 8 месяцев назад
Good luck! :)
@buckithed
@buckithed 5 месяцев назад
Fire🔥🔥🔥
@statquest
@statquest 5 месяцев назад
BAM! :)
@user-ol4pp4bk2m
@user-ol4pp4bk2m Месяц назад
you are a genius
@statquest
@statquest Месяц назад
:)
@BorisNVM
@BorisNVM 6 месяцев назад
this is cool
@statquest
@statquest 6 месяцев назад
Thanks!
@666shemhamforash93
@666shemhamforash93 Год назад
Amazing as always! Any update on the transformer video?
@statquest
@statquest Год назад
Still working on it.
@user-yx5rj2jv2d
@user-yx5rj2jv2d 10 месяцев назад
awesome
@statquest
@statquest 10 месяцев назад
Thanks!
@wowZhenek
@wowZhenek Год назад
Josh, thank you for the awesome easily digestible video. One question. Is there any specific guideline about binning the continuous variable? I'm fairly certain that depending on how you split it (how many bins you choose and how spread they are) the result might be different.
@statquest
@statquest Год назад
To learn more about one common way to create histograms of continuous variables, see: journals.plos.org/plosone/article?id=10.1371/journal.pone.0087357
@wowZhenek
@wowZhenek Год назад
@@statquest Josh, thank you for the link, but I guess I formulated my question incorrectly. The question was about not creating the histogram but actually choosing the bins. You split your set in 3 bins. Why 3? Why not 4 or 5? Would the result change drastically if you split in 5 bins? What if the distribution of the variable you are splitting is not normal or uniform? Etc
@statquest
@statquest Год назад
@@wowZhenek When building a histogram, choosing the bins is the hard part, and that is what that article describes - a special way to choose the number and width of bins specifically for Mutual Information. So take a look. Also, because we are using a histogram approach, it doesn't matter what the underlying distribution is. The histogram doesn't make any assumptions.
@wowZhenek
@wowZhenek Год назад
@@statquest oh, yeah, I didn't look inside the URL you gave because your described it as "one common way to create histograms of continuous variables" which seemed very much distant from what I was actually asking about. Now that I checked the link, damn, what a comprehensive abstract. Thank you very much!
@dhanrajm6537
@dhanrajm6537 5 месяцев назад
hi, what will be the base of the logarithm when calculating entropy. I believe it was mentioned in the entropy video that for 2 outputs(yes/no or heads/tails) the base of the logarithm will be two. Is there any generalization to this statement?
@statquest
@statquest 5 месяцев назад
Unless there is a specific reason to use a specific base for the log function, we use log base 'e'.
@marahakermi-nt7lc
@marahakermi-nt7lc Год назад
thankss joshh 😍😍 in 1:30 since the response variable is not continuous and takes on 0 or 1(yes/no) can we model it with logistic regression?
@statquest
@statquest Год назад
Yep!
@archithiwrekar4021
@archithiwrekar4021 11 месяцев назад
Hey, so what if our dependent variable ( here, loves troll 2) is continuous? Can we use Mutual information in that case? by binning aren't we just converting it into a categorical variable?
@statquest
@statquest 11 месяцев назад
You could definitely try that.
@noazamstein5795
@noazamstein5795 7 месяцев назад
is there a good and stable way to calculate mutual information for numeric variables *where the binning is not good*, e.g. highly skewed distributions where the middle bins are very different from the edge bins?
@statquest
@statquest 7 месяцев назад
Hmm... off the top of my head, I don't know, but I wouldn't be surprised if there was someone out there publishing research papers on this topic.
@eltonsantos4724
@eltonsantos4724 Год назад
Que Top. Dublado em português
@statquest
@statquest Год назад
Muito obrigado! :)
@ruiqili1818
@ruiqili1818 3 месяца назад
Your explanations are alway awesome! I wonder how to explain Normalized Mutual Information?
@statquest
@statquest 3 месяца назад
I believe it's just a normalized version of mutual information (so scale it to be a value between 0 and 1).
@Lara-qo5dc
@Lara-qo5dc 2 месяца назад
This is great! Do you know if you can interpret a NMI value in percentages, something like 7% of information overlaps, or 7% of group members overlap?
@viranchivedpathak4231
@viranchivedpathak4231 10 месяцев назад
DOUBLE BAM!!
@statquest
@statquest 10 месяцев назад
Thanks!
@aleksandartta
@aleksandartta Год назад
1) based on what to choose the number of bins? Does larger number of bins gives lesser mutual information? 2) what if the label (output value) is numerical? Thank in advance
@statquest
@statquest Год назад
1) Here's how a lot of people find the best number (and width) of the bins: journals.plos.org/plosone/article?id=10.1371/journal.pone.0087357 2) Then you make a histogram of the label data.
@avnibar
@avnibar Месяц назад
Hi, thank you Josh. I have one question. Does MI score is affected by imbalanced data?
@statquest
@statquest Месяц назад
Presumably - pretty much everything is affected by imbalanced data. This is because you have a much better estimate one class and a much worse estimate for the other.
@RaviPrakash-dz9fm
@RaviPrakash-dz9fm Год назад
Can we have videos about all the gazillion hypothesis tests available!!
@statquest
@statquest Год назад
I'll keep that in mind.
@AI_Financier
@AI_Financier 6 месяцев назад
3 more things: 1- it would have been great if you could make a comparison with correlation too here, 2- discuss the minimum and maximum value of the MI, 3- the intuition of this specific formula
@statquest
@statquest 6 месяцев назад
Thanks! I'm not really sure you can compare Mutual Information to correlation because correlation doesn't work at all with discrete data. I mention this at 1:20.
@ian-haggerty
@ian-haggerty 3 месяца назад
Seriously though, I think the KL divergence is worth a mention here. Mutual information appears to be the KL divergence between the actual (empirically derived) joint probability mass function, and the (empirically derived) probability mass function assuming independence. I know that's a lot of words, but my brain can't help seeing these relationships.
@statquest
@statquest 3 месяца назад
One day I hope to do a video on the KL divergence.
@romeo72899
@romeo72899 Год назад
Can you please make a video on Latent Dirichlet Allocation
@statquest
@statquest Год назад
I'll keep that in mind! :)
@GMD023
@GMD023 Год назад
Off topic question...but will chatgpt replace us as data scientists/analysts/ statisticians. I just discovered it tonight and it blew me away. I basically learned html and css in a day with it. Im worried it will massively reduce jobs in our field. I did a project that would normally take all day in a few minutes...scary stuff.
@insomniacookie2315
@insomniacookie2315 Год назад
Well, if you really want his opinion, watch the AI Buzz #1 Josh uploaded three weeks ago. It’s in this channel. As for my opinion, obviously nobody knows yet, but it will soon be a new ground-level for anybody else. For some that all they can do is basic things ChatGPT does far better, they are in danger; for others that can make more values out of ChatGPT (or any tools to come), they are in far better shape. Which do you think you and fellow data scientists are? And even for the basic stuffs, there should be at least someone to check whether the ChatGPT has done some absurd work or not, right? Maybe at least for a few years or so.
@ayeshavlogsfun
@ayeshavlogsfun Год назад
Just out of curiosity how did you learn HTML and CSS in a day ? And what's specific task that you solved
@toom2141
@toom2141 Год назад
I didnt think ChatGPT is that impressive afterall. Makes so many mistakes is not able to do really complicated stuff. Totally overhyped!
@statquest
@statquest Год назад
See: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-k3b9Mvtt6lU.html
@GMD023
@GMD023 Год назад
@@statquest thank you! This is great. Im also starting my first job today post college as a research data specialist! Your videos always helped me throughout my data science bachelors, so thank you!
@user-il8vc4pc5f
@user-il8vc4pc5f 3 дня назад
In case of continuous variables how to decide the number of bins and the boundaries?
@statquest
@statquest 3 дня назад
It probably depends on the dataset. Usually with things like that I like to plot histograms to make decisions.
@andrewdouglas9559
@andrewdouglas9559 Год назад
It seems information gain (defined via entropy) and mutual information are the same thing?
@statquest
@statquest Год назад
They are related, but not the same thing. For details, see: en.wikipedia.org/wiki/Information_gain_(decision_tree)
@andrewdouglas9559
@andrewdouglas9559 Год назад
@@statquest Thanks, I'll check it out. And also thanks for all the videos. It's an incredible resource you've produced.
@AI_Financier
@AI_Financier 6 месяцев назад
maybe next video on this: KL divergence
@statquest
@statquest 6 месяцев назад
It's on the list.
@harishankarkarthik3570
@harishankarkarthik3570 2 месяца назад
The calculation at 8:27 seems incorrect. I plugged it into a calculator and got 0.32. The log is base 2 right?
@statquest
@statquest 2 месяца назад
At 8:07 I say that we are using log base 'e'.
@6nodder6
@6nodder6 Год назад
Is it weird that my prof. gave me the mutual information equation as one that uses entropy? We were given "I(A; B) = H(B) - sum_b P(B = b) * H(A | B = b)" with no mention of the equation you showed in this video
@statquest
@statquest Год назад
That is odd. Mutual information can be derived from the entropy of two variables. It is the average of how the surprise in one variable is related to the surprise in another. However, this is the standard formula. See: en.wikipedia.org/wiki/Mutual_information
@9erik1
@9erik1 10 месяцев назад
6:18 not small bam, big bam... thank you very much...
@statquest
@statquest 10 месяцев назад
BAM!!! :)
@Tufelkind
@Tufelkind 4 месяца назад
It's like FoodWishes for stats
@statquest
@statquest 4 месяца назад
:)
@ronakbhatt4880
@ronakbhatt4880 28 дней назад
Can't we use correlation factor instead of Mutual information for continuous variable?
@statquest
@statquest 28 дней назад
If you have continuous data, use R^squared.
@AlexanderYap
@AlexanderYap Год назад
If I want to calculate the correlation between Likes Popcorn and Likes Troll 2, can I use something like Chi2? Similarly between Height bins and Likes Troll 2. What's the advantage of calculating the Mutual Information?
@statquest
@statquest Год назад
The advantage is that we have a single metric that works on both continuous, discrete and mixed variables and we don't have to make any assumptions about the underlying distributions.
@usamahussain4461
@usamahussain4461 11 часов назад
this is a nice tutorial and with different useful scenarios. But I didn't completely grasp the intuition of something never changing telling nothing about something that does. I understand it mathematically but hoping for a more intuitive explanation, because even if something does not change, there are some matches between the features.
@statquest
@statquest 9 часов назад
Say like I ask a bunch people what is their favorite color is and how old they are. Some of the people are young, some are middle aged and some are old, but everyone loves the color green. Now, if I told you that someone in that group loved the color green, what would that tell you about that person's age? Nothing. Since everyone loves green (it never changes) it can't differentiate between young, middle aged and old people.
@Chuckmeister3
@Chuckmeister3 Год назад
What does it mean if mutual information is above 0.5? If 0.5 is perfectly shared information...
@statquest
@statquest Год назад
As you can see in the video, perfectly shared information can have MI > 0.5. So 0.5 is not the maximum value.
@Chuckmeister3
@Chuckmeister3 Год назад
@@statquest Is MI then somehow influenced by the size of the data or the number of categories? The video seems to suggest it should be around 0.5 for perfectly shared information (at least in this example). With discrete data using 15 bins I get some values close to 1. Thanks for these great videos.
@statquest
@statquest Год назад
@@Chuckmeister3 Yes, the size of the dataset matters.
@yurigansmith
@yurigansmith Год назад
@@Chuckmeister3 Interpretation from coding theory (natural log replaced by log to base 2): Mutual information I(X;Y) is the amount of bits wasted if X and Y are encoded separately instead of jointly encoded as vector (X,Y). Statement holds on average and only asymptotically, i.e. for optimal entropy coding (e.g. arithmetic encoder) with large alphabets (asymptotically for size -> oo). It's the amount of information shared by X and Y measured in bits. Mutual information can become arbitrarily large, depending on the size of the alphabets of X and Y (and the distribution p(x,y) of course). But it can't be greater than the separate entropies H(X) and H(Y), respectively the minimum of both. You can think of I(X;Y) as the intersection of H(X) and H(Y). ps: I think the case of perfectly shared information is if there's a (bijective) function connecting each symbol of X with each symbol of Y, so that the relation between X and Y becomes deterministic. In that case H(X)=H(Y)=I(X;Y). The other extreme is X and Y being statistically independent: In that case I(X;Y) = 0.
@yourfutureself4327
@yourfutureself4327 Год назад
i'm more of a 'Goblin 3: the frolicking' man myself
@statquest
@statquest Год назад
bam!
@liam_42
@liam_42 2 месяца назад
Hello, that's a great video and it has helped me understand a lot about Mutual Information as well as your other video about entropy. I do have a question. At 11:13 the answer you get after calculation is 0.5004 and it is explained that it is close to 0.5. However when I do the math (( 4 ÷ 5 ) × log ( 5 ÷ 4 ) + ( 1 ÷ 5 ) × log( 5 ) ) the answer I get is 0.217322... Am I missing something? Because from what I understood, the closer you get to 0.5, the better it is but it is not confirmed by my other examples. Is there a maximum to mutual information? Thank you for your video.
@statquest
@statquest 2 месяца назад
The problem is that you are using log base 10 instead of the natural log (log base 'e'). I talk about this at 8:07 and in this other video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-iujLN48gumk.html
@liam_42
@liam_42 2 месяца назад
@@statquest Thank you for your answer. That explains a lot.
@rosss6989
@rosss6989 2 месяца назад
I have same doubt, when both columns are equal it says mutual info is 0.5 then what is maximum value of mutual info and in which scenario ?
@AxDhan
@AxDhan Год назад
small bam = "bamsito"
@statquest
@statquest Год назад
Ha! :)
@TommyMN
@TommyMN Год назад
If I could I'd kiss you on the mouth, wish you did a whole playlist about data compression
@statquest
@statquest Год назад
Ha! I'll keep that topic (data compression) in mind.
@rogerc23
@rogerc23 Год назад
Ummm I know I have a cold right now but did anyone only hear an Italian girl speaking ?
@statquest
@statquest Год назад
?
@FREELEARNING
@FREELEARNING Год назад
Great content. But just don't sing, you're not up to that.
@statquest
@statquest Год назад
Noted! :)
@VaibhaviDeo
@VaibhaviDeo Год назад
i will fite you if you tell daddy stat quest what to do what not to do
@igorg4129
@igorg4129 Год назад
I was always interested how should we think if we want to invent such a technique. Imean ok, lets say I "suspect" that the probabilities here should do the job, and say my goal is to get at the end of a day some "flag" from 0 to 1 which indicates the strenght of a relationship, but how should I think on, to deside like what comes to denominator vs nominator, when use log etc. There should be something like an "thinking algorithm" P.s Understanding this will be very helpfull in understanding the existing fancy formulas
@statquest
@statquest Год назад
I talk more about the reason for the equation in my video on Entropy: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-YtebGVx-Fxw.html
@joshuasirusstara2044
@joshuasirusstara2044 Год назад
that small bam
@statquest
@statquest Год назад
:)
Далее
Entropy (for data science) Clearly Explained!!!
16:35
Просмотров 588 тыс.
Выпускаем трек? #iribaby
00:14
Просмотров 164 тыс.
Teach me STATISTICS in half an hour! Seriously.
42:09
Linear Regression, Clearly Explained!!!
27:27
Просмотров 1,3 млн
Solving Wordle using information theory
30:38
Просмотров 10 млн
Information Theory Basics
16:22
Просмотров 62 тыс.
Naive Bayes, Clearly Explained!!!
15:12
Просмотров 1 млн
Why Information Theory is Important - Computerphile
12:33