Design Matrices For Linear Models, Clearly Explained!!! 

Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

Statistics never appealed to me since it always seemed boring ... until I started watching your videos a few days ago. Now I'm hooked. Thanks for making statistics so fun and intuitive to learn!

SQ is so addictive. A simple concept clarification youtube search led me down hours and hours of SQ contents. Thank you, thank you, thank you!

WOWW. I have been watching this video for at least 5 times and I always learned something new! I was confused by people saying "regress out the batch effect", but it's that simple!!! Thanks Josh.

"turning something on by letting it be" - some proper life advice there

thank you! I can't believe how clear you are explaining this, seriously thank you!

im so happy when i look for a topic and see that you've covered it.

I never get tired of watching your videos, I have learned a lot. This is my favorite channel :) Thank's!!!!!! Would you consider making a video on assessing the significance of mixed models? Please! this topic is complicated

wow so helpful, this cleared my doubt of combining and interpreting categorical and continuous predictors. Thanks a ton:)

So clear！！！ Thank you for answering my confusion in a such simple way!

BAM!!! crystal clear explanation! Thanks!

Awesome one josh.... Keep up the great work...

6:06 , having flashbacks to week one and two of Andew Ng's ML Coursera course, but now it feels more intuitive!

I love your video, really easy to understand

All the topics of statquest are well explained. Thank you sir for this nice statistics subject based channel Statquest.Good wishes and happy journey for this successful statquest youtube channel.

Great. Really clearly explained. Thanks.

Awesome video as usual. Thank you

Statquest is getting bigger, watch out! hahaha i can't stop humming this

Great video (as usual)! You're definitely one of my favorite "thing-explainers" I've come across :D I was left with a question near the end though, with respect to "correcting for batch effects". After a quick online search, I see this is usually an issue and many packages to attempt to correct it. I could imagine two explanations that lead to different explanations for the difference: i) "We ran the exact same protocol in two different labs. However, the sensors were differently calibrated, so there is a bias in readouts." -> This suggests the batch-effect correction. ii) "We ran the exact same protocol in two different labs. We ensured the sensors were equally calibrated, but there's *still* a bias in readouts." -> This could just be due to inherent variability in the sample, right? (It is probably not *too* likely for the data to be the same, just shifted down a bit. But it's possible! Questions: 1) This correction *assumes* that the difference in batches is *not* due to inherent variability in the features we're measuring (but is instead due to e.g. technician error), right? There would be no way to *prove* it one way or the other, would there? 2) If it's (ii), wouldn't "correcting for batch effects" throw out useful information about the response variable's distribution? 3) Ideally, hopefully both labs calibrated their sensors via e.g. blanks, so (1) shouldn't be immediately the reason. How would you suggest teasing out sensor bias (1) vs sample variability (2)? Would we have to assume a model for the data and compare whether Lab A's two group's parameters significantly differ from Lab B's? (Or maybe the "ideal" situation happens infrequently enough that going for (1) is usually not unreasonable?) Thanks again! Will continue to Quest On :D

Thanks a lot!

In the last part where you combine the linear regression and the t-test, you have a regression line for each category, but the slopes of the lines are identical. Isn't this rare? How would the equation change if you had two lines with different slopes?

do a statquest for wald's test , chi- squared test and fisher's exact test please!!

I love u, its out now, but i need a longer intro. :)

Hi Josh, I really like your videos that clearly explain lots of things! In the last example of this video, is it similar with using a mixed model? Intuitively, it seems the lab was treated as a random variable and mutant was treated as a fixed variable, and here we are interested in the difference between mutant and control after removing the impact of lab A/B.

It would be nice to see a video on how matrices and matrix multiplication in neural networks transform data, and edit the dimensionality of the inputs. Transforming data makes a lot of sense spatially, but what does changing dimensions do? What are the different ways in which you can interpret matrix notation? Talk about the special relationships once you organize the data in that particular format

Design matrices? More like "Dang good videos are these!"

it would be great one video about interaction terms! and how to use for deconvolution of cell types. :-)

Do a statsquest on the F distribution, please

Though I tried a lot but unable to digest all the concepts. Thanks for all detailed explanation. Bdw can we expect another video on design matrix and it's real use with a little simpler explanation!!! if possible

7:46 Compare mean model vs type-only model, p>0.05 12:17 Compare size-weight-type model vs type-only model: p=0.0025

pls do some videos on recommendation systems (like collaborative filtering) and distributed learning (like mapreduce)!!

I'd love to see you do a central composite design matrix for response surface modelling at some point! I seem to understand the topic enough to implement it in Python, but I'm still struggling to put it into a more general context. The final ANOVA output people usually show is the most confusing. I get that you do an ANOVA for each single term in the model (so each term is a "group"), but then there's always an extra "residual" ANOVA and I can't figure out what that is calculated on.

Thank you for your wonderful videos which I became addicted to recently :D I found this one particularly useful for me to understand the concept of design matrices and how one can use them not only to turn on/off certain terms in equations for categorical variables, but also scale terms for continuous variables. Now, I have a question regarding the batch effect example you kindly provided. You assume that the difference between mutant and control mice in lab A is the same as in lab B and one can represent this as the average difference of the 2 labs. You also mentioned in the comments that if we had more measurements from one lab than the other, we can use a weighted average of the differences in the 2 labs. What if the mutant-control difference in lab A really differs from that in lab B. You can already see that in the dot plot. Would it make sense to add an offset term for the difference too as you did for the lab A control mean? In this case the equation should be Y = lab A control mean + lab B offset for the mean + lab A difference + lab B offset for the difference ?

Thank you very much for your video! Can you please tell me, what you explained (when combined t-test and regression) is also called mixed modelling or multilevel modelling?

Thank you very much! I really learned a lot from your channel. I have a question, at 13:25, the second term Lab B offset. Is the Lab B offset = lab B control mean - lab A control mean?

First of all props for your excellent series of videos. First rate introduction to some really hard stuff. An answer to your question at 2:46 is that the problem in general implies two distinct linear equations but algorithms for linear models (eg lm() function in R) only allow for one general linear equation. So yes you can solve it by hand using two separate linear equations but the algorithm won't let you enter the problem in that format. So how do you get around this problem of having only one linear equation to work with when you have two linear equations in reality? Ans: Break up the linear model by using dummy variables in a thoughtful manner or let the algorithm do it for you but check that it's not messing with you. Either way you got to know what's going on and here's one explanation.... For the mutant/control example we have the following linear model: (Note i should be read as a subscript for the ith term and e is the error term. So ei is NOT some madness in the complex plane but simply the ith error term. If ei throws you just treat it as a symbol related to irreducible error (the noise that is always around). That's all it is.) yi = B0 + B1 xi + ei (eq 1) (pretty much y = b + mx with e as some reality thrown in) where B1 is the slope of the line with xi as its associated input values (eg the labels mutant and control but as values in this example) and B0 is the y-intercept. As you can see there are no input values associated with B0 so we can not directly associate input values to B0 through the first column of the design matrix. This explains why the first column of the design matrix is fixed to all ones. This is essentially saying that B0 exists for all i and it's up to the gods of regression to determine what B0 becomes. All is not lost though. Nothing says we can't monkey with the linear model (eq 1) through its variable xi in a creative way that ends up associating B0 with a label. And that's what we're going to do. But first we need to deal with the issue that our labels are not numbers and this creates an opening for some linear equation monkey business without defying the gods. Since our equation won't work on labels we need to assign numerical values (dummy variables) and by selecting the appropriate dummy variables for our labels, we can separate the general equation into two separate equations each of which corresponds uniquely to each label. Word of caution though, how we select our dummy variables determines how our labels get assigned to the separate equations, so it's not an arbitrary choice. So let's try the following: xi = 1 if i is a mutant xi = 0 if i is a control (0 and 1 are the dummy variables and here we are assigning actual numerical values to xi. These are the values assigned in the second column of the design matrix) in which case yi = B0 + B1 xi + ei (eq 1) becomes yi = B0 + B1 + ei if i is a mutant (eq 2) yi = B0 + ei if i is a control (eq 3) (Notice that there is no longer any separate xi term in eqs 2 & 3 since xi has been assigned dummy variable values) and this allows us to interpret our controls relative to B0 whereas our mutants correspond to B0 + B1. Pretty slick and no lightning bolts from above. In this case B0 is the mean for the controls (intercept in the summary report), whereas B0 + B1 is the mean for the mutants. It's important to note that B1 (what is returned second in the summary report) is the mean difference between mutants and controls (ie mean of mutants-mean of controls). If the p-value for B1 is significant that means adding the the difference of mutants-controls to our model is significant with respect to the control alone (ie mutants are different relative to the controls and what we're interested in). Now if we switched our dummy variables, xi=1 for controls and xi=0 for mutants, then B0 would be the mean for mutants; B0 + B1 is the mean for the controls; and B1 is the mean of controls-mean of mutants (ie got reversed). If the p-value for B1 is significant here that means adding the difference of controls-mutants to our model is significant with respect to MUTANTS alone (ie controls are different relative to mutants so equivalent to what we want but is kinda upside down and weird). To get totally weird we could assign xi=1 for mutants and xi=-1 for controls then B0 would be the overall average for the combination of mutants and controls. Bottom line: How we set our dummy variables determines how we can interpret B0 (as well as B0+B1 and B1) and is a slick trick that allows us to separate out from our linear model, two linear equations that uniquely correspond to our labels. An Introduction to Statistical Learning by James Gareth gives some nice examples of this on pg 84 at this level of math. Available for free on-line and also provides details on how to assess the quality of your model which is critical. And if you've gotten this far....Hey Josh, how about some banjo??? Some Ola Belle Reed would fit nicely here.....I've endured, I've endured, how long can one endure!!!!

in the last example for batch effect, Did you suppose that difference(mutant-control) is same for both the labs?

2:40 It might be because, the standard needs only one bit to represent both values, since only change is 2nd bit. We can just ignore the 1st bit while storing thus reducing the size. Just a speculation.

Awesome, wondering how you deal with missing values in such cases?

Hi Josh, could I ask which distribution table u looked at or based on to get the pvalue for the F I calculated?

Please do you have any video on how to build the A matrix from a stochastic model?

hi! can you explain briefly how to obtain p-values from F? i'm currently preparing "advanced statystics for business" in management engineering course in the university of Palermo and i'm really enjoying your videos! thanks a lot!

Thank you for this clear explanation. I just have a question in8:26 if the lines had different slopes, how will be the design matrix in that case?

13:32 About the term for difference(mutant - control), is that an average of Lab A's difference(mutant - control) and Lab B's difference(mutant - control)?

I am here for the singing intro

Could you also kindly explain dirichlet regression and also the segmented regression

Awesome video! In the Control vs Mutant scatterplot @6:16, is each individual data point meant to represent the expression of a single gene within a control or mutant replicate (4 reps per group)? I'm just trying to make sense out of a design matrix that I got from my RNA-seq data with thousands of genes. I wonder if the SSmean for each sample is being calculated using the mean expression of all 18,000+ genes....I'm wondering the same for the SSfit...sigh.

Hey Josh! Thanks for this video. For the first example, I was curious about the equation y = control intercept + mutant offset + slope. Suppose our slopes for our two lines were different. Would the exchange become: y = control intercept + mutant intercept offset + control slope + mutant slope offset? Thanks!

You actually need to watch these two parts backwards as the purpose of the t-test becomes clear at the end of this video. I wish you showed us first why we do that and then explained how.

Excellent video. However, I had a few questions. How to design a design matrix for regression+control group when the slopes are not the same for the two groups. Similarly for comparing Lab A and Lab B measurements, which difference in the mean values (mutant-control) should be taken, since there are two values from the two labs.

TRIPLE BAMMMMM!

Hi, thx for all the hard work. Could you explain what the difference(mutant-control) at 13:36 is?

What if the slopes are different? Do we consider two separate terms for each slope? (and similarly for more terms when there are more variables)

How do you interpret r-square if you have those 2 regression lines? about what line does r-square speak?

Hi josh, M highly indebted to you for all of your awesome videos. I have one doubt which is making me restless. please if you could help. suppose i have only one independent Variable which is categorical (control(0)/mutant(1) ) and one dependent varible(DV), which is continous, so i will get the graph as shown in 0:53 in this video. There is no other independent variable(IV) to make the graph look like as in 8:29. How to check "linearity" between IV and DV, as it is an important assumption of linear regression? We cannot draw a line with some slope and intercept in this case (0:53) or this " linearity" assumptions will not be required to check? and other Linear regression assumptions such as "normality of residuals" and "Homoscedasticity" are they also not required to be ensured in my example? please help. Thanks

Ahhhhh the relationship between design matrices and dummy encoding just clicked for me. The less common case where the the design matrix is all 0s in the first vector in the matrix is the same as one-hot encoding right?

For the mouse weight/mouse size/mutant example, if the two slopes are different, does that mean my new equation can be: y= control intercept+ intercept offset+ control slope+ slope offset And now my new Pfancy= 4, is that true?

can we use pythogorus theorem to find cooedinates on the line

Hi Josh,Actually I wanted to ask you how to decide a null hypothesis for any case .As I understood the concept of p value but I didn't understand how to decide a null hypothesis for any given case.

how do you count the p-value from that F-value? I was a bit lost in there

I'd like to know more about the number F he keeps calculating. Is there a Wikipedia article about it, as a starting point? Or a name commonly used for it?

1. The idea I get from this video is we can use any design matrix we want to create a test between any complex vs simpler model and interpret the significance of their difference in equation terms from the p-value right? 2. I have trouble relating the conclusion at 11:08 (p-value small--> fancy better than simple mean model) with linear regression and what the p-value here means. Why in linear regression there is a p-value for every coefficient (so a whole linear regression has multiple p-values) but here there is only a single p-value?

Hi Josh, the 2 lab example is a bit confusing. 1. First of all when we say difference between control and mutant means, do we mean (lab A control mean - lab A mutant mean) and (lab B control mean - lab B mutant mean). 2. Secondly, what is lab B offset? Is it (lab A control mean - lab B Control mean). 3. How lab B mutant equals (lab A Control Mean + lab B offset + difference). As per the figure, the lab B mutant should be equal to lab A Control Mean + Difference between lab A Control Mean and lab B mutant mean. Why do we have lab B offset in this equation.

10:33 why the degrees of freedom of the fancy model = n-3 , since we have two lines, we have n-2*2 = n-4 degrees of freedom cuz each line can pass through any two points?

How you calculated the p-value?

oh how i would love to be a mutant mice : tall and skinny

Is there any other application of design matrix instead of regression analysis ?

Hi josh, I was able to understand everything prior to the decision matrices topic. Could you please suggest me on how i could improve my understanding? Also, if it's plausible, can you please make a "Decison matrices in python" video cause that would really help.

Sorry! But how you get p-value = 0.003 (11:06 minutes)? From table?

13:31 y = labA control mean + labB offset + difference Shouldn't there be two "difference" i.e. y = labA control mean + labB offset + difference.labA + difference.labB ?

hi guys! hope you are having a beautiful day! hey, I have a question, in the last example where there are 2 different labs, the difference between mutant and control is calculated in this way: Lab A mutant mean - Lab A control mean? or (Lab A mutant mean + Lab B mutant mean) - (Lab A control mean + Lab B control mean). Hope anyone can help me. Thanks!!

te quiero

your voice is different from previous

Hello Josh! I am a big fan with a colleague :-) Could you please do the same vid for models with interactions? Kr. BAmmm!! :-)

I didn't get the last example.

Is this kind of 2 way ANOVA?