Least squares approximation | Linear Algebra | Khan Academy 

"Some of you might already know where this is going.." Me: Nope

When I get a real job, I will donate my bonus to Khan Academy. This has saved me so much time and you are so awesome.

This lesson is fantastic! I understood the problem in only 15 minutes! You're absolutely better than my numerical analysis teacher at university, that can't properly teach an argument in two hours! Thank you!

Comes in handy while studying machine learning.

very helpful! Thanks a lot! you are doing great things! I also listened to your other videos, all very wonderful!

This channel is a blessing. I've had some really bad professors and I've had some really good professors. But even the really good professor never made the concepts click with me as well as these videos do. Like, not only do I understand the math better, but just the little diagram you drew showing A's column space, and visibly showing how b is outside of A's column space yet could still be approximated using a vector v in A's column space, like idk how else to describe it but that just made it click for me. Edit: I guess _one_ way to describe it and how it clicked for me: So we use a line to approximate a bunch of data points on a graph, or plane. If these data points were in a straight line, the "approximation" would have no error. However, this is often not the case. Now, think about the equation y=mx+b. Let's use c instead of b to avoid confusion in the next step. So we have y=mx+c. This is the equation used to represent our line. Suppose b=y-c. Then we have mx=b, which looks a lot like Ax=b. And it is! A is just a 1x1 matrix. So the line is bounded by the column space of A, or m, and our variable(s) (in this case, just x) can be changed to get b. Just basic algebra: if m=3 and b=6, then x=2. But say b is a 2D vector, e.g. b=(1, 2)^T. Well now, no matter what x you use, you can't get b (unless b just happens to lie on the line). You can only get as close to b as the column space of A will allow you. In the diagram drawn in the video, the column space of A is a plane, so the span of A is 2. For simplicity, let's suppose A is a 3x2 matrix (a geometrical interpretation of this is that A is a 2D plane "floating" in a 3D space). b appears to be a 3D vector (so while A is only a 2D slice of the 3D space, b is a point that could be anywhere in the 3D space). So, just like before, we try to use a line bounded by the column space of A to get as close to b as possible by changing our variables (in this case, x1 and x2). Correct me if my understanding is wrong :)

Your videos are just great !!! The concepts with geometrical examples make very good sense !!! Thanks a lot

This was incredible, I started this video off being so confused about the least squares, and I just get it entirely now! Thank you so much :)

Very useful man you are doing an amazing job this literally saved me hours of searching and reading can't thank you enough :)

Thanks a lot, very comprehensive ! great job!

I was doing an online machine learning course and got lost when the lecturer introduced the normal equation (which this is, with a different name). Needless to say, I'm finna binge-watch your linear algebra lectures now because I get insecure about using equations I don't understand. Thanks for the playlist, I really wanna put ML in my toolset so we're doing this!

Thanks so much Khan...wonderful explanation in two videos that explains everything...great. You are wonderful

Excellent explanation of a valuable technique.

You are like a billion times better than my professor... and my professor isn't even bad. On the contrary he's my favorite! You're just even better at explaining things. Plus it's impossible for me to lose focus with the pretty colors and your beautiful handwriting. lol I have my Linear Algebra final tomorrow (technically today) and I owe the A that I'm sure to get to you and all your helpful videos!

Best approach to the problem. No gradient, no multivariable calculus. you're master!

This is super useful in solving assignments.THanks khan academy.

It would be great having links when says "I explained (whatever) in a different video" to access that explanation. In this case I wanted to know why C(A)transpose=N(Atranspose). Thanks¡

Helpful exploration of least square properties

Best linear algebra playlist.

Thank you so much. You just simplified long boring hours of confusing lecture

Thank you Salman Khan. I appreciate the opportunity to relearn the method here. You can never hear this stuff enough times.

really helpful

Super clarity......

Good video!!!! And nice work! Good luck with the KhanAcademy :)

can you teach me cubic expressions and cubic equations :) eg. solve the equation x(3X3X3) - 2x(2X2) - x + 2 = 0 by using the factor theorem formula :)

god dang it I knew I should have chosen other bachelor thesis..

Nice derivation of the normal equation

This is surprisingly easy

can we please get a video for the maximum likelihood estimation

Excelent video. Thanks much :)))))))) Vahag

what happens when AT*A is singular. How do we solve for the least square solution?

This guy is good...........

It seems I have seen the best video!

I wish to know how to solve this: x has values of : -2 0 1 2 3 and y : 17 5 2 1 2 and i'm asked to use the least squares method, but i've been absent and i don't know exactly what my teacher ment by that or what that method consists of. Can anyone help me solve this ?

thank you very much sir

Should have used n instead of k its usually mxn in R^n

great geometric intuition of linear regression

Very useful! In my lecture slides I had this term Hx=z for the same problem and I couldn't make sense of how we could get to this as the best solution: x = (Ht*H)^-1 * Ht * z. Now I understand:-)

I tried using this trick for the problem I'm facing, but it turns out that when I multiply AT by A, I get a matrix which isn't invertible, so I still can't solve it. LOL This _still_ seems odd to me, because even if some element in the input matrix A was contributing 0 to the result b, it should _still_ be possible to get a point as close as possible to the result.

thaks

I have one question, whether the LSS always consistent? if yes, how can I prove it? please answer

nice vid, but why did you take the length squared? i understand that the length of the vector would be sqrt(b1^2 + b2^2...bn^2) but why did you square even that?

how did you know that it was a projection to the Col(A) and not anything else like the Range(A)?

thanks

I have a question.. does least sequare approximation has always solution..

love this guy

I am the 60th guy liking it !! :P :D Great vid, thank you. :)

accha hai

당신은 나의 구원자입니다. 정말 명쾌한 강의입니다. 감사합니다!! 👍👍👍

This is the first Khan Academy video I watch and don't understand...

2018? Im alone :(

Big brajn

❤

Sometimes I can't see what he's writing.

bro just do an example lol

n1

ICAM ! ICAM ! .... .. ...... !

🤩

gorgeous

Respond to this video...

thank you sir