Sparsity and the L1 Norm 

Currently losing my mind over how effective that simple diamond shape is in finding the 1-norm, I suddenly want to research topology. Amazing stuff.

It's amazing how you can work for years with a function and still gain new insights about it and improve your intuition. Thanks you so much for these videos.

Finally got a nibble of insight into L1, L2 norms they bring up in Ng's coursera classes. Seems like it would be good to use when you want to prune the size of a network after training. I love the chunks of insights these videos give and the very understandable way they're explained.

mind blown. this guy is a rare breed. Loving the content

These are all very visual and well made lectures, thanks a lot for the high quality videos and openly available materials!

Amazing videos. The quality and clarity are above and beyond. Thanks!

Best 11 mins spent! Thanks a lot for distinguishing your solution than the rest of the internet!

This was a perfect supplement to a lecture I just watched for class

Sparsity? More like “Super explanation that’s gotta be”…one of the best I’ve ever heard on the topic. Thank you so much for making and sharing all of these incredibly high-quality videos!

Really best videos...I have been reading the stuff for dictionaries creation from last two week and now I understood the concept

Excellent explanation! Thank you!

I don't know why this came up on my suggestions list but was a good explanation! Thank you Steve :)

You are simply amazing. Keep inspiring.😎

You've convinced me to buy your book based on these videos.

Hey prof Steve, could you please make videos about Lyapunove stability and especially nonlinear systems

It is quite simple to use smoothing to solve compressed sensing. Though I found quite slow. You repeatedly correct with the compressed sensing data, smooth, correct, smooth..... The binomial filter is quite good. These days I would prefer to solve with a neural net in one step. Likely to be much faster.

You can train neural networks with sparse mutations. Continuous Gray Code Optimization mutations work well. Each GPU gets the full neural model and part of the training set. Not much data needs to move around, just a sparse mutation list, costs back and accept or reject. A fixed random pattern of sign flips is a simple random projection. If you follow it by a Walsh Hadamard transform you get a much better random projection. The WHT acts as a system of fixed orthogonal dot products. The variance equation for linear combinations of random variables applies to dot products. The CLT applies not just to sums but any combination of adding and subtracting (cf WHT) You can have inside_out neural networks with fixed dot products and parametric (adjustable) activation functions. Ie Fast Transform fixed-filter-bank neural nets. You do a simple random projection before the net to stop the first layer taking a spectrum and do a final fast transform as a pseudo_readout layer. f(x)=x.ai x=0 , i=0 to n-1 is a suitable parametric activation function.

Visually, l_{2/3} looks like an attractive choice as well, as it approximates l_0 even better than l_1 does. Whats the advantage of l_1 over l_{2/3}?

Sir do also have a similar detailed lecture on sparsity introduced by l-infinity norm?

Great video. Also..your setup is nuts! Writing backward? Mirror image? Does not compute!

I love how Y variable sounds like "why"

Sir when can we expect continuation videos for Chapter 3?

Does someone have a simple way of describing why the L0 sparse solution is NP complete?

Are we gonna get more videos on chapter 3?

Hey, steve I have question like in lasso we have regularisation term |w| if this is. Change to w^2/1+w^2 ... What change it make to coefficient?

Hi, will you also be covering the mathematical background of the L1 sparsity property? I understand it intuitively, but I haven't found a good intuitive and rigorous mathematical proof/explanation of it.

Is L_infinity norm the most uniform solution?

Any formal proof that l1 norm minimisation gives the sparsest solution in higher dimensions? I was trying to find a reference for it but unable to get one .

What's the difference between a sparse solution coming up on S1 and one on S2 ?

Sort of a random question, but what would happen if, when using the L1 norm, the s vector was perfectly parallel to one of the sides of the L1 square? Would it return a dense solution, instead of sparse? Would that mean you are in an inappropriate basis for a sparse solution? Thanks!

Does L1 norm regularization for the neural network layer also promote sparsest weights in the layer?

Hi, I still don’t understand why finding the L0 norm of S is NP hard. Could you please explain a bit? I’d appreciate your help. Thanks!

What if the line and one-norm diamond becomes parallel?

Isn't taxi-cab norm L1? As in the distance a car will have to drive in a grid system

I keep picturing scenes from Cheers where Norm walks into the bar and everyone shouts "Norm!" Then Sam asks "What'll it be today, Norm? L1 or L2?"

Steve, I love how your abbreviation of "solution" is "sol" to nth power 😏

How do you prove a L0 if you can not compute it? Maths is weird, yet the L1 is handy for a project I'm doing, so I should really be asking, how did you read my mind...👻 ps is it useful for imbalance classes?

Shoot, this guy's from UW? I'm doing the master's for EEs there right now lol

"there are infinitely many asses" 😂😂😂. You said it two times and I concur

neat

He writes backwards. how impressive!

Now I have to go searching for the formula for l_{2/3} norm. I can guess what it is, but it seems oddly out of place compared to all the integer bases, so I'm as curious where it originally came up.

This comment is literally literal !

He tells a nice story but his explanation requires a lot of background knowledge so in the end I do not feel very satisfied. It would be useful to add also more detailed tutorial links for further study