Linear Systems of Equations, Least Squares Regression, Pseudoinverse 

This is so cool! When I started this lecture series in order to understand PCA better, I had no idea it would also relate to least squares regression! This blew my mind! Thank you so much for making these. They must be a lot of work, but they are so appreciated!

Pseudoinverse? More like "Super videos for us!" Thank you so much for making all of them.

please keep going with the numerical linear algebra/numerical analysis/scientific computation/applied math stuff thanks :)

Thank you for explaining hard to grasp concepts in a filtered simple manner for us to understand. Your lectures are a great complement to prof strangs both high quality content.

Exceptionally clear explanation, crisp hand-written notes, wonderful!

Excellent Lecture ! So clear to understand! Thank You !

Am I right, that you write on real glass in front of camera and the image is just mirrored by editing? If so its brilliant.

Thank you very much for the clear explanation of pseudo-inverse.

I'm always impressed by how clean the board is, so it looks like there's nothing at all.

Absolutely Great Content

can't thank you enough for sharing your knowledge with entire world

Thank you Professor for this valuable lecture

Great explanation, sir!

My god, you just explained what my professor is trying to explain for 5 lectures

Great content !

great video

Thank you Steve for video. We make the assumption that it is an economy SVD at time 6:28. Then, how can we guarantee that V multiplies with V* will become identity matrix, especially for the under-determined system?

Since A has fewer rows than columns, then A is an m×n matrix with m

Wonderful lecture

Yeah. This is a very good and useful lecture.

Man. It was really awesome..

having second thoughts about doing master cuz your videos are just too helpful

Excellent

I'm amazed. This is so clearly explained!!

knowing that the pseudoinverse exists makes me feel really powerful

This is really nice lecture I've ever seen. I want to recommend this lecture for engineering graduate student basic class!! :-) Thank you so much~ I'll buy the book! ^-^

Thanks for the video was really nice! There are 2 points which seem to be important to me. The invers of Σ is actually not always computable (if there exist a single value =0) so the more nice Expression would also be Σ+ . Where Σ+ is the matrix where every non zero single value is inverted but the zeroes are left as they are. And why not follow the convention of naming matrizes? Normally a matrix is called a mxn matrix. It seems you use a nxm matrix here which i think is a bit confusing at first glance.

Thanks for the video Professor. Could you help me with something please? I am trying to fit a sinusoidal surface the depends on (x,y,t) but in my case b is not a vector but a matrix, what can I do in this case? Thank you

I loves your lectures! it is so clear and save us from mist of information. I noticed that you wrote Sigma matrix is a invertible, if A is not invertible, shouldn't the Sigma matrix also is not invertible, hence Pseudo inverse of Sigma matrix? Thank you for clarification

Thank you for the awesome materials and everything is well explained! One question I have is how we calculate the inverse of the singular matrix which is a non-square matrix? Isn't that back to the problem, i.e. inverse the non-square A matrix, we had at the first place?

In the case of over-determined matrix X, why VVT is equal to identity since we are using economy matrices?

Thanks a lot

Damn, thank you, I finally got it thanks to you after 3h research xD

In an underdetermined case, if you use the economy SVD, is V*V' equal to an identity?

Great!!

Thanks for the great video! One question: 7:50, if you have zero singular value, how do you compute sigma inverse? Thank you!

am I just missing something, why is n

i know this is just about the notation, but i think the majority of linear algebra text use m by n, rather than n by m. It's sometimes a little confusing here...

LOL So moor-penrose pseudoinverse is just inverting one SVD term at a time, and since they U and V are orthogonal they are just transposed. Well that saved me a lot of effort looking into where it comes from.

waw!! Merci beaucoup

Wonder if Steve Brunton can cover for under/over determined systems the nontrivial solutions of Ax = 0, using SVD.

I am confused with the dimension of A matrix. in overperformed, shouldn't overdetermined have m>n

I FOUND GOLD.

How does this board work? Are you writing in opposite direction?

Hi Sir, I am quite curious about the name of the 'transparent' blackboard, I want to buy one, where I can get it? Thank you.

Thanks for your video. However, I'm confused about the proof of the theorem(why the norm is minimized), can you give a simple proof? Thanks.

Hello, thank you for this nice video series. This is so helpful and I use it with your book for my masters thesis. But while going through the equations, one question popped to my head: at 7:32 you use the inverse of Sigma, but for the SVD Sigma is not a quadratic matrix, rather than a nxm - Matrix and as such not invertable (in the "classic" sense of invertable matrices). So while I understand that if I use the economic SVD, this Matrix would be a mxm - Matrix, but I don't understand it for the case of nxm. Is there a video or a page in the book, where this case is discussed? Other than that, thank you very much for saving my masters degree :D

Why in definition of A-dagger you put Sigma-inverse rather than Sigma-dagger? Sigma is non-square and has only pseudo-inverse rather than inverse.

Extremely nice lectures Steve Brunton, thank you very much for all the effort of creating and sharing them! do any of you - or anyone that reads this :) - know of any reference that explores the math of why you get min |x|2 in the underdetermined case? thank you in advance!

I have a set of 3D positions and vectors. I try to find their intersection, so I linearized the equations like written in academic papers, but I dont get the results. I dont have a clue what could be wrong. I have Ax=b I tried: 1) (AA')^-1*A'*b 2) pinv(A)*b 3) A\b 4) I''ve just tried the SVD method, I get the same results with all. I checked my values 100 times. The vectors and directions are correct, I manually calculated them and confirmed with the matbal code.

If we have x and b vectors how to find A matrix ?

I get a bit lost when he constructs the left pseudo-inverse. I cannot grasp why the single value matrix is guaranteed to have an inverse.

Sir, by solving for x, I understand that A dagger cannot be a true inverse of A, but since the SVD is an exact egality of A, and since multiplying by U transpose S-1 and V is again an exact operation, I don't understand where the approximation step was introduced in the calculus.

what if we had a singular value of 0 ? ...cant that happen ? when A has dependent columns? ....in that case we wouldnt have Sigma inverse correct ? ....do we first idk,... drop some columns of A so they are all independent /.?

Wouldn’t the error be ||X-x dagger||_2 not just ||x dagger||_2 in itself?

How can sigma be invertible when it's a rectangular matrix?

It is not Sigma^-1 but Sigma^-T so the traspose of the inverse ?????

Are you using the economy version of the SVD? otherwise you are not able to take the inverse of Sigma, which is a nxm matrix. EDIT YES >> see next video in this list

Hi sir, Can you solve one linear regression problem using svd and upload in RU-vid.? Please it helps me lot and others as well.!

Forget about everything what visual do you use to write and record....

This is cool but computational cost to determine svd is nightmare

I don’t know why such was never explained when I took econometrics.

when you say thank you i emphasize thank you!!!! (not me) :)))

why U trans U cancel to identity. Don't they result in a square matrix???

does this guy actually writes backward?

Under is dual to over, left is dual to right, up is dual to down, in is dual to out. Thesis is dual to anti-thesis -- The Generalized or time independent Hegelian dialectic. Alive is dual to not alive -- Schrodinger's/Hegel's cat. Duality creates reality.