Using Linear Models for t-tests and ANOVA, Clearly Explained!!! 

Hi Josh, Love your content. Has helped me to learn a lot & grow. You are doing an awesome work. Please continue to do so. Wanted to support you but unfortunately your Paypal link seems to be dysfunctional. Please update it.

Thank you! :)

You're welcome!!! :)

Wow, thank you!

Thank you very much! :)

Thanks!

Thanks!!! I know, it's a little weird to look at a t-test from this perspective, but it shows how the F-statistic is a generalization of T-statistics. (Here's a cool hint - just like the F-statistic is a generalization of T-statistics, Chi-square statistics are a generalization of normal statistics....)

BAM! Yes, usually t-tests are taught before linear regression, but I like teaching them in this order (regression first) since the extension of a t-test into ANOVA is way more obvious.

@@statquestThat sounds like a good plan.

You're welcome! I'm glad to hear you think the videos are helpful. :)

Thank you very much!!! :)

Thank you very much! :)

Thank you very much! :)

Muchas gracias!!!!

Glad it helped!

HOORAY! I'm glad the videos are helpful.

Hooray! :)

bam! :)

Thank you very much! :)

StatBlessed :D

Hooray!!! I'm so glad you like this video - it's one of my all time favorites. :)

Hooray! :)

I hope you will have the time to answer just in few words please! R sqr tell us how x is useful to predict y, so in the case of a t test or anova how to use it? we just talk about F & p, can we say it explains some % of the variance between treatments or it's useless!? Thank you so much Mr. Starmer

This is a great question. The traditional way to teach and perform t-tests (and ANOVA) only results in 't' or 'F' statistics and a p-value - no R-squared. However, as you see in this video, it's easy to also report R-squared - you just have to want to do it. The case of t-tests and ANOVA are just like regression and R-squared tells you the same thing - it gives you an estimate on the magnitude of the difference. The p-value just tells you that it is significant. If you did a t-test and got a small p-value, but also a small R-squared, then you could easily deduce that there's not a huge difference between the two groups (even if is statistically different). In contrast, if you did a t-test and got a small p-value and a large R-squared, then you would know that there's a big difference between the two groups. So we can see that R-squared is useful for even the t-test. I suspect that one reason presenting R-squared with t-test results is rare, is that often with t-tests, it is easy and very common to plot the data - so people will show you their data and give you the p-value. Seeing the data is sort of like a "visual R-squared" - you can see if the data are very close to each other or far apart.

THANK YOU SO MUCH.... YOU ARE VERY KIND SIR. I summarize if you allow : "significant p-value + R-squared" = how much is the différence Really GREAT! Thanks again & Good luck!

Hooray!!!! :)

Glad you like them!

Yes! :)

Glad you like them!

Bam! :)

Thanks!

Thank you very much!!!! :)

Sure, I think that is a safe thing to say.

ANOVA works fine with unbalanced samples. You just have more rows in your design matrix for one category than another.

Awesome, thank you!

Hooray! :)

Great suggestion! I've added your correction to a pinned comment that will be easy for other people to find.

Hooray! :)

Glad you like the videos! I've added Taguchi, DOE and 2 or more factor ANOVA to my to-do list. I believe that my video on Multiple Regression in R may already satisfy your second request: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-hokALdIst8k.html

StatQuest with Josh Starmer Thanks 😀

Thanks so much! The degrees of freedom StatQuest is high, high on the to-do list. It is never far from my mind. I have it about 1/2 done in my head, but the second half is tricky - some situations are easier to illustrate then others - but it's just a matter of setting aside time just for it and nothing else and it will get done. The good news is that I'm maybe 1 or 2 months away from doing StatQuests on ridge, lasso and elastic-net regression - all examples of linear regression (or, more generally, generalized linear regression since these ideas can be applied to logistic regression) with regularization. So that's sure to happen soon (just as soon as I can!) The last one, log-linear regression, is the logical follow up to logistic regression. I may do a "big picture/main ideas" StatQuest on that as soon as I can. It's on the list!

StatQuest with Josh Starmer thanks for your reply. Can't wait for the next videos

Hooray!!! :)

According to this website [ onlinecourses.science.psu.edu/stat501/node/297/ ], the t-statistic and F-statistic produce equivalent p-values when the F-statistic's degrees of freedom in the numerator is 1. The relationship is t^2(n-p) = F(1,n-p), which apparently means the p-values for each will be identical. Don't know why that is but videos on the relationship between those two distributions may help. Anyway, I assume the relationship applies here in which the df = 1 for the F-statistic numerator when comparing two groups. As a side note, most slopes for p-values in multiple linear regression are calculated with t-tests. However, F-tests comparing the variance between models with and without the slope produce an identical p-value due to the above mentioned relationship. Thinking of slope significance in terms of how much more variance the model explains with vs without the slope seems much more intuitive to me, and I'm glad I found these videos.

:)

I hope to do that one day, however, it will probably be a while since I'm writing a book on neural networks right now.

I'll keep that topic in mind.

Any point at the mean of the data will have the same fit. I use a line to make it easier to see.

Thanks! :)

GLM stands for two things "General Linear Models" and "Generalized Linear Models". Unfortunately, those two things are different - but when most people say "GLM", they most frequently mean "Generalized Linear Models". Generalized Linear Models are, in essence, a way to adapt the concept of a "design matrix" to a variety of problems and models. For example, in this video, we used design matrices to do t-tests and ANOVA. However, these same design matrices can be used with Logistic Regression (see those videos if you're interested) and they can also be used for DE analysis with DESeq2. However, the underlying math is different in all three cases. So the good news is that if you understand design matrices, you can do amazing things in a wide variety of contexts. The bad news is that SS(mean) and SS(fit) in these videos may or may not correspond to something in another system, like with DESeq2 or Logistic Regression. Logistic Regression, for example, doesn't use least squares at all, but instead relies on maximum likelihood to optimize the fit. Does this make sense?

​@@statquest Thanks for the reply! I think I got your point. So the basic idea is to use the generalized linear model (GLM), which is more like a concept, to fit the data, and in the video the linear regression, which is more like a method, is used for the fitting. In programs like DESeq2, they use the negative binomial regression method to fit the RNA-Seq read counts, but the overall idea is still using GLM to describe how experimental factors (e.g. genotype and treatment) determine the expression of a gene (by a design matrix), and the p-value is kind of telling me how well the GLM fits ( or how convincing the result is).

@@drzun You've got it!

@@statquest Hooray! Before watching your videos, I had a really hard time understanding the statistics behind the data analysis of RNA-seq, and I can't express how grateful I am to you & the videos.

@@drzun Hooray!!! That's great. I'm glad my videos were so helpful! :)

I have a series of videos on Logistic Regression if you are interested in that: ru-vid.com/group/PLblh5JKOoLUKxzEP5HA2d-Li7IJkHfXSe

I'll keep those topics in mind.

This is a great question. Unfortunately there are not many good or well known tests to compare standard deviations and other features (other than means). I'm not sure, but it could be that this is due to the lack of a central limit theorem like concept for standard deviations etc. (That's just a guess, so don't quote me on that).

The histogram that I used in the linear regression was intended to illustrate what an F-distribution represents, and it is the same here as well.

We drop the residuals because it doesn't make any sense to include them in the predictions we make with this equation. The residuals only make sense when we are evaluating how well the model fits the data. But with predictions based on new data, we don't know the actual values, so we don't know the residuals.

I'll keep it in mind.

@@statquest that will be awesome. Triple BAM!!!

The null hypothesis is that there is no difference. Thus, the p-value tells us if having parameters (other than just the intercept) are useful for distinguishing between groups. If there is no difference, then we should fail to determine that the estimated parameters values are significantly different from 0.

This is a great question. The t-test is just a specific type of F-test. If you have statistics software, you can compare the results and see that the p-values are the same (however, the F-statistic itself will be the square of the t-statistic. Why the square? Because, as you saw in the first video in this series, the F-statistic can never be negative, but the t-statistic can.) There are multiple ways to calculate a t-test, this using an F-test is my favorite because it is much more flexible. Does that make sense?

@@statquest I knew you would took it to the further level. So basically the two tests are both about model parameters hypothesis significance test, just use different methods, so p-value should refer the same thing. BAM! Thank you so much!

1) This equation simply represents what goes into to the design matrix. The residual is the difference between this equation and what is observed. 2) The equation just illustrates how we create the design matrix and what it represents. 3) You don't need to have equal numbers of samples for each category (they can be different).

Technically, it is all linear regression. However, they give it different names. t-tests are when you have two distinct groups and ANOVA is when you have more than 2 distinct groups.

The first video in this series explains F-statistics and f-values: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html

I'm not sure what your question is. A t-test is a way to compare to categories of things (like "normal diet" vs "special diet") when you measure something continuous (like weight).

SS(fit) is the sum of all of the squared residuals (the difference between the actual observation and the lines we fit to the data.)

yep

ANOVA is really only intended to be used when the dependent variable is continuous.

I have a video that shows how bootstrapping can be used for a t-test here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-isEcgoCmlO0.html

@@statquest Thank you very much I checked it. I understood hypothesis for mean between two groups, bu still I do not understand how it is used for genes. This is complicated I think. I wanted to see a table for t and p values of genes. Am I thinking wrong?

@@akyanus7042 Replace the responses people had to the drugs (feeling better or worse) with the read counts for a gene in different samples. For example, you might have 3 samples that took drug A and 3 samples that took drug b. For Gene "X", bootstrap the read counts for the genes and calculate p-values as described.

@@statquestThank you.

The residual is the difference between our model's prediction and the actual value. Mathematically, it is "Observed value - model = residual", where model is the design matrix times the parameters. If we added the residual to our design matrix, we would get "Observed value - (model + residual) = Observed value - model - residual = (observed value - model) - residual = residual - residual = 0." And that wouldn't be very helpful.

@@statquest thank you for the clarification!

What time point, minutes and seconds, are you asking about? (However, I'm guessing that you are asking about the difference between the equation that perfectly fits the data, because it includes the means + the residuals, and the equation that generates the residuals (because it only includes the means). The equation that does not include the residuals is the one we use to make predictions with future data.

@@statquest Thanks Josh, that is the point I was asking about, I will review the video again once more

If you go back to 6:11, you see that the "design matrix" is formed from the 1's and 0's that turn on/off the values for the control mean and the mutant mean. So when you have y = column1 * 2.2 + column2 * 3.6, to predict a value for a new control sample, you plug in 1 for column1 and 0 for column2 and thus, the prediction is y = 1 * 2.2 + 0 * 3.6 = 2.2.

I'll keep that in mind.

We then have to test each one separately to identify which one is significantly different.

The next video in this series may help you understand how to design experiments: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-CqLGvwi-5Pc.html

This t-test gives the exact same results as the two-tailed t-test used in hypothesis testing. If you want it to represent a one-tailed test, just divide the p-value by 2.

​@@statquest Hi Josh! thanks for the response! What I mean is i am used to seeing t-test in comparing means of sample in hypothesis testing by finding t-statistic with sample mean and sample error to get the p-value . Hence i am a bit confused on the concept of computing a "F" on a "T-test" mentioned in this video.

@@josephgan1262 Most people are taught about the t-test from the perspective of the t-Distribution. This is fine, but the t-test using a t-distribution is not as flexible. In contrast, the F-distribution is a generalization of the t-distribution. The t-distribution only allows us to compare two means. The F-distribution lets us compare 2 or more means. If you want to convert your F-statistic to a t-statistic, just take the square root of it. If you want to convert a t-statistic to an F-statistic, square it.

It depends on the goals of the experiment. However, typically people will do post-hoc tests with a multiple testing correction - however, FDR is way better than Bonferroni, so use FDR if you can.

@@statquest Thanks for your reply. If we use multiple linear regression models to replace ANOVA, whether the t tests on the regression coefficients is like the post-hoc tests in ANOVA without multiple testing correction?

@@Doctor_CCC Pretty much

@@statquest Thank you very much for your explanations. On the premise that the t tests on the regression coefficients is like the post-hoc tests in ANOVA *without* multiple testing correction, I am wondering how to appropriately interpret the p value of t tests on the regression coefficients in multiple linear regression analysis? (To mitigate against multiple comparison problems). By the way, is there any learning resource about using post-hoc tests with FDR after multiple linear regression analysis?

@@Doctor_CCC I should clarify. The t-tests compare the model with and without individual variables. This is different from Post-hoc tests in ANOVA, where we test all possible pair-wise combinations. Testing all possible pair-wise combinations can quickly add up to a lot of tests, necessitating adjusting p-values. In contrast, when we just test the model with and without individual variables, we only do as many tests as we have variables - and usually this means we've only done a few extra tests, which, typically, does not necessitate adjusting the p-values. However, if you have a ton of parameters (variables), then you should adjust them with FDR. In R, this is super easy: stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html

I'll keep that in mind.

The design matrix is just a general way to specify how each measurement fits into the equation.

Do you have multiple columns of means? If so, just use those as normal data. You might also want to watch my StatQuest on Design Matrices: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-CqLGvwi-5Pc.html

@@statquest thank you, i'll do it. and thank you for all your videos

I'm writing my own book right now. I hope it is out next year.

@@statquest Woah! Looking forward to read that.

I describe how p-values are calculated for individual features in these videos: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zITIFTsivN8.html ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-hokALdIst8k.html The concepts apply to ANOVA in the exact same way.

Post-hoc tests with ANOVA are just a matter of defining your "design matrices", which I illustrate in the next video in this series: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-CqLGvwi-5Pc.html

​@@statquest If there are three drugs: drug A, drug B, and drug C, we use drug A as the reference level. We then use dummy coding to compare B vs. A; C vs. A in the linear model. In the linear model, we can determine the difference of B vs. A; C vs. A by calculating the p value of the coefficient. However, it seems that we can not determine the difference of B vs C in the above linear model? Thank you for your reply.

I'm not sure I understand your question, however, the graph in the top right corner at 8:22 shows a horizontal solid black line at the average of the y-axis coordinates.

The residuals squared and added when we solve for the optimal parameters. For details, see: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html

@@statquest thank you josh :)

I'm not sure I understand your question. The residual for each measurement is paired with that measurement, so it is easy to keep track of.

Thanks!

The number of observations or data points.

Did you watch part 1 in this series? If not, it should answer all of your questions: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html

We are predicting the y-axis value, and that is why we are interested in the y-axis more than the x-axis (the stuff on the x-axis is only being used to predict y-axis values.)

You might not realize that this t-test is the same test you would use to do a simple hypothesis test. The equations might look different, but they are equivalent, and the p-values are the exact same.

@@statquest that cleared things up, thanks a lot :)

Sure, a modern computer can handle more than one equation. But back in the day memory was limited and that limited the number of tests a computer could perform. So the the original idea was to unify as much of linear models into a single framework called "General Linear Models", with the idea that one equation could be used in a general setting on a computer without having to check a bunch of different conditions. In the early days, different conditions meant different look-up tables for figuring out the p-values and since computers had very little memory, this limited what they could do.

'n' is the total number of observations (which I call "samples" in this video). Often, the more data we have (the larger 'n' is) the more confidence we can have in the predictions because the p-value becomes smaller.

@@statquest Thank you for clarifying 'n'. my other question ware regarding intuition on F-statistic. For example what is its range and what does the lower end of values imply, compared to higher end etc.

@@janakiramanbalachandran504 I explain the F-statistic, it's range and what lower end values imply in Part 1 in this series: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html

The F-distribution is a generalization of the t-distribution. In other words, the F-distribution can do everything we can do with a t-distribution and more.

I have the same question here. @StatQuest

Because my equation is being multiplied by the design matrix, it is essentially the exact same thing that you have.

​@@statquest Bam!! Thank you for the explanation

ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-nk2CQITm_eo.html

I'll keep that in mind.

The purpose of the t-test is to determine if there is a significant difference between mice with the normal gene and mice with the mutant gene. However, we can also use the model to make predictions with new data. If my test tells me that there is a significant difference between normal and mutant mice, if you tell me you have a mutant mouse, I can tell you that the gene expression should be the mean of the mutant mice. If my test tells me that there is not a significant difference, then I will use the mean of all the mice, normal and mutant, as my prediction.

@@statquest I see, now I undertand better, thank you Mr. Josh.

You can look at the R-squared value.

@@statquest hi Josh I have categorical target variable(three levels) and continuous Independent variable. Can I still use r2 to check the association between two. TIA

@@minakshimathpal8698 yep.

"Correlation" means - you get significantly different mean outputs on different inputs. To prove this we make a null-hypothesis "All means are equal" and try to reject it. If we failed to reject it then our data lacks enough evidence of correlation.