The Beauty of Linear Regression (How to Fit a Line to your Data) 

You are not only teaching math stuff but also teaching how to think, thank you very much for the great video. Really inspiring, glad i discovered this channel, waiting for the videos about jacobian , translation , rotation, quaternions

the constant animation loop gets a bit annoying. reversing, stopping and changing the animation from time to time would be a solution (and your newer videos are even better anyway!)

I agree. Honestly I look back on this video and cringe at a few of the details, like how the animation loop goes on and on and is a bit nauseating, and music is too loud. But you live and learn! 😅 When I first started making these videos I really had no idea what I was doing.

@RichBehiel 18:19 on the left hand side you used Laplace Symbol instead of Nabla Symbol. But except that => great video! 👍

With regards to pacing, I want to say that I really enjoy your general presentation style. You're not simply reading a script and getting the perfect take, you're actually doing a "live" presentation and I really appreciate the way you ad lib or go off on little tangents. I burst out laughing in your buoyancy video when you read the integral "zndS" phonetically.

I’m very glad to hear that! :)

And he repeats the animation so we can assimilate what's going on instead of quickly switching to the next thing. Very relaxed explanation which is nice.

Glad to hear that! :)

Спасибо! :)

Thanks! :)

Thanks for the kind comment, John! :) I touch on Fourier analysis in my upcoming video on relativistic QM, the Klein-Gordon equation. Hoping to upload it within a week.

Thanks for the kind comment, and I’m glad you enjoyed the video! :)

Thanks, glad you enjoyed the video! :)

Thanks, I’m glad you liked the video! It’s one of my favorite mathematical concepts, so it’s great to see others enjoying it too :)

Thanks! :)

Sorry 😂

That’s really cool! I’ve read about that kind of thing in an intro to differential geometry book, but hadn’t connected the dots in the context of this video. Thanks for a very interesting comment :)

That’s awesome, I love to hear that! Challenge for you: can you solve it for a general N-degree polynomial? Like with some kind of recursive algorithm. I actually don’t know if this is possible but it seems like a fun puzzle!

@@RichBehiel that would be a fun problem to solve! And even if it can't be solved, I'm sure proving or disproving the possibility of a solution would make a great paper!

Great idea! I’d love to do a video on that someday.

Thanks, glad you enjoyed the video! :)

Yeah that’s a typo, sorry! 😅 Thanks for pointing that out.

Thanks William! :) For this video I used Python, specifically matplotlib. You can use that by downloading Anaconda, which will install Python and some scientific modules, then call “from matplotlib import pyplot as plt”. After calling that line, you can use things like plt.figure() and plt.plot() to make a figure and plot things. In this case the parameter space and real space are two subplots in a figure. They’re refreshing at 60 frames per second in a loop which sets the dot’s position in the parameter space while making the line in the real space, based on the current a and b values. To turn on the error landscape, I also added some code to evaluate the error metric (objective function) at all points in the parameter space for each a and b. Then for the error force I calculated and plotted the negative gradient of that. For the part where the dot descends down the gradient, I used F = ma - kv with mass parameter m and friction-ish parameter k to make the dot roll down the hill and then stop at the optimal point. I’ll be more careful in future videos to post the source code of the animations too. Well, at least for videos after the one I’m going to post this week; for that one, and the previous videos, I was very sloppy with the code and it wouldn’t be too helpful to see them. But there have been a few comments now about how these animations were made, so I figure the best answer is the code itself. In the future I’ll be better about writing cleaner animation codes and sharing them.

​@@RichBehiel wow, you're awesome for such an in-depth reply to this. Thank you, I might try this on my own

Thanks! :) I used matplotlib in Python.

Great question, and I’m actually not sure. Anyone know the answer?

Mastering linear algebra is a great and enduring source of spiritual fulfillment 🙏

The same concept of minimizing a least squares objective function by setting the gradient to zero applies to nonlinear least squares, but there are also extra steps involved.

Great question! It’s invertible as long as its determinant isn’t zero. Since it has the form [A,B,B,N] where A and B are real numbers and N is a positive integer, so its determinant is AN - B^2. For this to be zero would require that AN = B^2, in other words for the sum of x_i^2 times N to equal (the sum of x_i)^2. I’m not sure if this can happen, feels like it can be proven one way or the other without a ton of work, but I’ve gotta go. So I leave that as an exercise to the reader! :)

@@RichBehiel I think one of the cases where the matrix is not invertible is if all the points are on a vertical line. Kind of makes sense since then the form y = ax + b doesn't really work.

I did a mesh grid for a and b from -5 to 5 100 points X,Y. Then I calculate the modulus as Z= the sum of the sqrt of the ^2 of each eq and did a contourf(X,Y,Z), but no luck :/

I think the quiver plot is ok as quiver(X,Y,eq1,eq2)

I did a contourf and a quiver. If the contourf isn’t working, it’s possible the color limits are off? Oh, actually come to think of it, I might have taken the log or sqrt of the error, to flatten out the landscape so it’s easier to see. Basically applying a nonlinear colormap.

@@RichBehiel Thanks a lot! Keep up the great work!

Still no good, I'm sorry. So in contourf is V (or log(V) or sqrt(V)) and in quiver is Fa,Fb, with spanned a and b, right? Sorry to bother but I feel I understand, but not having the same results make doubt what I'm doing wrong :/ Is it too much if you share the code for visualizing the parameter space?

I forget what I did for that, I think I just had some sines and cosines of different frequency in x and y.

I code DNNs too. Um. I understood your words but not your point. Genuinely curious here. So we can calculate the inv matrix. Take the reciprocal of the determinant and multiply it by the matrix with the diagonal swapped and the upper/lower negated. This spits out a new matrix with the property that if you multiply that by the original you get the identity (assuming linear independence). Ok fine, all very useful. But what's that got to do with the price of fish?

Not for linear regression, but for fits with more parameters yes. Gradient descent can sometimes get stuck in a local minimum, a valley other than the best one. If there’s an analytic solution, it might involve the roots of a polynomial or something, so you can have multiple values which are locally optimal. In that situation, the height of the objective function at each optimum can be quickly compared, since the list should be pretty short.

I’d like to someday! The procedure is very similar, but ax^2 + bx + c instead of ax + b. It’s a 3D parameter space, but the same techniques work.

Thanks! :)

Thanks for the kind comment! :)

Thanks! :) Grant is a role model for sure. The aesthetics of his videos are much better than mine though 😅 But I’ll get better over time.

Good point. Frankly I think it’s because OLS is easier, and gets the job done in most situations. But I agree that there are times when Deming regression is better. Although someone who uses Deming would presumably have learned OLS first. OLS is also conceptually ideal for explaining how calculus can be used to minimize fit error, so it’s a good go-to image to have in mind when solving fancier optimization problems.

@@RichBehiel I completely understand, in fact, this subject is making me think about applied mathematics, because if we go deeper, it's not like linear regression in any form is the the best way to actually model most data, so I'm thinking about dividing a function into splines to create a good fit, you can go too far and smoothly fit every point into a function, but then your function is skewed towards the data set, losing the ability for good projections. It's an interesting puzzle (and I hated applied mathematics in college)

Well, there are quite a few advantages of OLS compared to total least squares fit. For one, every measurement where x is tightly controlled and y is the thing you want to learn about, OLS is the right tool. Because there are no or only negligible errors in x, the distance of datapoints to the prediction, dx, doesn't matter and must not be included in the fit. And also it works much better with arbitrary functions than total least squares. For an arbitrary function I don't think there even is _any_ way to calculate the total least squares error. Only well behaved functions work, and even then you have to define the derivative to perform a total least squares fit.

True! For more complicated fits, the parameter space becomes more textured and you’ll often have multiple local minima. But with an analytic solution, these minima can be quickly calculated, for example as roots of a polynomial. Then there’s just a small list of points at which the objective function can be evaluated and compared, and the minimum can be chosen from the list.

@@RichBehiel Thanks! So the analytic solution gives us all the minima and we then can just check which one is the lowest. Cool

Yup. There may be some maxima and saddle points in there too, since those also have zero gradient, but those can either be filtered out analytically by solving some second derivatives for additional constraints, or just kept in the list and they won’t be the minimum so it doesn’t matter. In practice, people almost always do the latter. The only exception would be if the data rate is very high and there’s some benefit to solving those equations in exchange for a marginally faster routine. So in super high performance scenarios, the second derivatives are worth looking at.

I believe so, but I’m not 100% sure actually. As a good exercise in math, you can explore if it might be noninvertible under some conditions, just set the determinant to zero and see what a dataset would have to be like in order for that to happen. I’ve done millions, maybe billions, of linear regressions (on data streams) and have never run into this problem though.

@@RichBehiel Doing just the quickest amount of working out with a dataset of 3 values, I think the sum of outer product would only be singular if all the x values are the same, which obviously isn't going to happen. It's fairly easy to show that if we have a dataset like this, the matrix is singular (the 1st row of the matrix is just the second multiplied by x_i), though I'm not sure how you'd prove it the other way around (i.e. the matrix is non-singular in all other cases).

That makes sense! Btw, these equations are equivalent to a force and torque balance, if the residuals are imagined as elastic springs, so physically it makes sense that it would only be singular if the x values are all the same, or something like that.

Hmm… I’m not sure, tbh. Do you have all the modules installed? I’d recommend installing Anaconda, then running the code in Spyder (it comes with Anaconda). That way you’ll have a lot of mathematical and scientific modules already installed. Plus, Spyder looks cool.

Not sure why you get the window in debugging mode, but for normal python scripts you usually have to call plt.show() manually while notebooks would trigger them inside the corresponding cell. So you may also change your .py to .ipynb and run that in vscode

More than just rambling, you’ve hit on something deep there. It’s kind of weird that the distance between two points is the Pythagorean theorem, and not just dx + dy, right? Sort of related to that, the derivative of y^2 is 2y everywhere, which is nice and easy to work with. But |y| is sharp and not differentiable at y = 0. So it’s easier to work with y^2. You could do it with |y| but it’s more tricky.

Thank you! :)