Who's Adam and What's He Optimizing? | Deep Dive into Optimizers for Machine Learning! 

The Adam Optimizer is a very complex topic that you introduced and explained in a very well manner and in a surprisingly short video! I'm impressed Sourish! Definetly one of my favorite videos from you!

I remembered when my teacher gave me assignment on optimizers I have gone through blogs, papers and videos but everywhere I see different formulas I was so confused but you explained everything at one place very easily.

love the simplified explanation and animation! videos with this quality and educational value are worth of millions of likes and subscribers in other channels... this is so underrated..

Absolutely loved the graphics and intensive paper based proof of working of different optimizers , all in the same video. You just earned a loyal viewer.

Very Clear Explanation! Thank you. I especially appreciate the fact that you included the equations.

Sir your exposition is excellent, the presentation, the cadence , the simplicity.

Nice animations, nice explanations of the math mehind them, i was curious about how different optimizers work but didnt want to spend an hour going through documentations, this video answered most of my questions! One that remains is about the AdamW optimizer, i read that it is practically just a better version of Adam, but didnt really find any intuitive explanations of how it affects training (ideally with graphics like these hahaha). There are not many videos on youtube about it

Wow! Great video, more of these deep dives into basic components of ML please

Incredible video. I especially love the math and intuition behind it that you explain. Keep it up!

This video is amazing! You covered most important topic in ML, with all major optimization algorithms. I literally had no idea about Momentum, NAG, RMSprop, AdaGrad, Adam. Now, I have a good overview of all, will deep dive in all of them. Thanks for the video! ❤

Excellent video, please keep it up! Subscribed and will share with my colleagues too :)

Thanks for the great explanations! The graphics and benchmark were particularly useful.

I am coding backpropagation right now and this helped me so much.

Just found out your channel. Instant follow 🙏🏼 Hope we can see more Computer Science content like this. Thank you ;)

Thank you for such easy, simple, and great explanation. I searched quick overwiev how Adam is working and found your video. Actually I am training DRL Reinforce Policy Gradient algorithm with theta parameters as weights and viases from CNN, where exactly Adam is involved. Thanks again, very informative.

This was a really interesting video! I feel like this helped me understand the intuitions behind optimizers, thank you!

Thank you for a very clear explanation. Liked and subscribed

Nice video :) I appreciate the visual examples of the various optimizers.

Woah what a great video! And how you are helping people on the comments kind of has me amazed. Thank you for your work!

I never comment on anything, but wanted to let you know that this video was really well done. Looking forward to more!

what a good video, I watched it and bookmarked so I can come back to it when I understand more about the topic

So cool, just subscribed! I literally just started researching more about how optimizers work this week as part of my bachelor's thesis. Once I'm finished with my thesis I would love to see if I can create my own optimizer algorithm. Thanks for sharing! Do you happen to have the manimgl code you used to create the animations for visualizing the gradient path of the optimizers?

Great video, glad the algorithm brought me. The visualizations helped a lot

That is some quality work sir!

Nice vid, I'd mention MAS too, to explicity say that Adam at the start is weaker and could fit local minima(until it gets enough data) and SGD peforms well with its stochasity, and then slower, so both methods (peformed nearly like I mentioned in MAS Paper)

Very nicely explained. Wish you brought up the relationship between these optimizers and numerical procedures though. Like how vanilla gradient descent is just Euler's method applied to a gradient rather than one derivative.

best video on optimizers thanks

Very well explained and awesome animations. Hope to see more content in the future!

You are incredibly intelligent to explain such a complex topic formed of tens of research papers of knowledge in a single 20 minutes video... what the heck!

I love it! You are also nice to hear and see! :D

very clearly explained - thanks

The animations are amazing - what did you use to make them??

I'm not even a data scientist or machine learning expert but I enjoyed this video!

I used to have networks where the loss was fluctuating in a very periodic manner every 30 or so steps and I never knew why that happened. Now it makes sense! It just takes a number of steps for the direction of Adam weight updates to change. I really should have looked this up earlier.

Good work man. Which tool do you use for making the animations?

Great explanations

Adam - A dynamic adjustment mechanism

Great video!

This video is super helpful my god thank you

Great video dude!

bhalo video TY

dude love the video title. came just to comment that. i think i searched something like "who is adam w" when i started my ai journey

Hey what do you think, could we use reinforcement learning to train the perfect optimizer?

Thank you So much sir. But I will like you to create videos on upconvolutions or transposed convolutions. Thank you for understanding

well done!

share about batch sgd on pre-training of llm. What were results.

Thank you

What is the mathematical expression for the boss cost function at the end?

I tried using momentum for a 3SAT optimizer token i worked on in 2010. Doesn't help with 3SAT since all variables are binary. It's cool that it works with NNs though!

I am the only one to not understand the RMS propagation math formula? What is the gradient squared is it per component or is the Hessian? How do you divide a vector by another vector? Could someane explain me please.

This is sick

Fantastic

Best ML video title I think I've ever seen haha

Great, great, great!!!

When you write square root of V_t in a denominator, do you mean this component-wise? V is a high dimensional vector I assume. Also what if it has negative values? Don’t you mean the norm of V?

I wonder if we could use the same training loop NVIDIA used in the DrEureka paper to find even better optimizers.

The notation for the gradient is a bit weird but nice video!

What about the Adabelief optimizer? I use it most of the time and it is a bit faster and needs less tuning than the Adam optimizer.

Yooo Sourish this is heat do you remember hs speech

The “problem” the Adam algorithm in this case is presented to solve (the one with local and global minima) is simply wrong - in small amounts of dimensions this is infact a problem, but the condition for the existence of a local minima grows more and more strongly with the amount of dimensions. So in practice, when you have millions of parameters and therefore dimensions, local minima that aren’t the global minima will simply not even exist, the probability for such existence is simply unfathomably small.

What is the square of the gradient?

Sorry did I misunderstand something or did you say SGD when it was only GD you talked about? When was stochastic elements discussed?

Make one about the Sophia optimizer please!

1.54k subs it's crazy low for this quality remember me when you make it my boy

Hmm while rmsprop speeds up the demonstrated example, it slows down the first example.

The intuition behind why the methods help with convergence is a bit misleading imo. The problem is not in general with slow convergence close to optimum point because of a small gradient, that can easily be fixed with letting step size depend on gradient size. The problem that it solves is when the iterations zig-zag because of large components in some directions and small components in the direction you actually want to move. By averaging (or similar use of past gradients) you effectively cancel out the components causing the zig-zag.

Have you seen KAN?

A lot of times in academia, people are just using SGD with momentum but playing around with learning rate scheduling a lot. You don't always want to get the deepest minimum since it can actually give you poor generalizability. That's why Adam isn't that popular when researchers are trying to push to SOTA.

Like for the title. :)

love that title haha

Please explain one thing to me. Why do we negate the gradient vector to get the downhill direction? What if directly opposite to the gradient vector there's a bump instead of a smooth descent? Shouldn't we instead negate the parameter field itself, transforming holes into bumps, and then calculating the gradient?

why not using a metaheuristic approach?

is the code available?

Who is Adam is what sold me hahahahahaha

10:19 What a weird formula for NAG! It's much easier to remember a formulation where you always take antigradient. You want to *add* velocity and take gradient with *minus* . The formula just changes to V_t+1 = b V_t - a grad(Wt + b V_t) W_t+1 = W_t + V_t+1 It's more intuitive and more similar to standard GD. Why would anyone want to change these signs? How often do you subtract velocity to update the position? Do you want to *add* gradient to update V right after you explained we want to subtract gradient in general to minimize the loss function? It makes everything twice as hard and just... wtf...

Gemini 1.5 Pro: This video is about optimizers in machine learning. Optimizers are algorithms that are used to adjust the weights of a machine learning model during training. The goal is to find the optimal set of weights that will minimize the loss function. The video discusses four different optimizers: Stochastic Gradient Descent (SGD), SGD with Momentum, RMSprop, and Adam. * Stochastic Gradient Descent (SGD) is the simplest optimizer. It takes a step in the direction of the negative gradient of the loss function. The size of the step is determined by the learning rate. * SGD with Momentum is a variant of SGD that takes into account the history of the gradients. This can help the optimizer to converge more quickly. * RMSprop is another variant of SGD that adapts the learning rate for each parameter of the model. This can help to prevent the optimizer from getting stuck in local minima. * Adam is an optimizer that combines the ideas of momentum and adaptive learning rates. It is often considered to be a very effective optimizer. The video also discusses the fact that different optimizers can be better suited for different tasks. For example, Adam is often a good choice for training deep neural networks. Here are some of the key points from the video: * Optimizers are algorithms that are used to adjust the weights of a machine learning model during training. * The goal of an optimizer is to find the optimal set of weights that will minimize the loss function. * There are many different optimizers available, each with its own strengths and weaknesses. * The choice of optimizer can have a significant impact on the performance of a machine learning model.

what a title 😂

the title made me laugh i had to click this

Adam has parkingson

Are you real? I have a feeling you are AI video generation hooked up to a LLM that varies a script that the MIC uses to steer humanity to building it's AI god.

Consider watching at 0.75 speed.

It really annoys me when people claim using less electricity has anything to do with environmentalism. Most large ML models are trained in the Pacific North West (Seattle and Vancouver area) where most power comes from hydroelectric. Using more electricity has no meaningful environmental impact there since it's just diverting the energy that rivers are dumping in the ocean anyway. If you're worried about the environmental impact of energy, focus on the generation methods, not the consumption methods.