What is backpropagation really doing? | Chapter 3, Deep learning 

You might think that your videos are fodder for university students boning up on a subject, or mathematicians/engineers in the early stages of their careers - basically that you cater to a younger audience. I'll have to prove you wrong. I'm in my early 60's and have been involved with information technology in some form or fashion my entire career. I enjoy learning; always have. I've viewed many of your videos only because they interest me and have you subscribed on my RU-vid account so as to get notifications of updates. I find the topics about which you speak fascinating and am a bit jealous of those university grads today who now have access to this material at their fingertips. I wish with all my heart that I was able access these videos back when I was in university. It would have made life SOOO much easier for me back then. Your pedagogic skills are astounding, demonstrated by your ability to communicate difficult subjects precisely, concisely and simply. The animation format is integral with the presentation, adding to the delivery of the material. I salute you!! Please keep these videos coming.

I can't claim to have understood everything from the first watch-through of this series, and I will watch these videos again with pen and paper in hand, but even this first viewing has made neural networks go from pure witchcraft and wizardry to something that actually makes sense in my head. I can't possibly thank you enough for posting these videos.

Anyone else smiling through all of his videos because you're understanding so much so well like never before?

This series is totally brilliant. I am 73 years old and used to teach mathematics. I am still learning stuff and with the help of sites like yours it makes it so much easier. Have you thought of doing any videos on the really complex subject of real analysis/. Keep up the good work. Kevin Connolly

I disable adblock for this

Great explanation, your team is awesome. "A drunk man stumbling aimlessly downhill,but taking quick steps" is the best analogy ever for Stochastic gradient descent. :-)

These visualizations are spot-on. Only a few people in the entire world need to make a great explanations backed by powerful visualizations about a topic - the rest of the world just needs to discover these. So much time wasted by learners trying to locate easily-digestible information, among all the inferior presentation methods out there. Glad to have found one of the best for this topic.

Note to myself: Aditya, if you're having trouble understanding, read this. Scroll to 06:17. Listen to what he's saying, "In a sense, the neurons that are firing while seeing a 2, get more strongly linked to those firing when thinking about a two." Now, pause the video and listen. All this is is just a fancy way of saying: when we show our model a picture of a handwritten 2 and we tell it, "hey listen up this thing is a 2" (like in our training set we have labels for our input pictures) and then we find out the activation units that have a say in influencing the hypothesis value of the 2 label, ie, the activation units which can heavily increase or decrease the value of the output unit for the label 2, we tell those activation units "hey guys when you see something that resembles this thing, fire up the 2 label, ie increase the hypothesis value of the label for 2. Basically, we assign those activation units greater weights (or parameters) which influence the hypothesis value of the 2 label more (so that they can actually give us the right answer and say "oh look this is probably a two". I hope this helped and didn't complicate it further. If you don't get it, go over the video a few more times and review Andrew Ng's notes on this in the ML course on the Backpropagation lecture in week 5. Cheers, bro. Love you.

Your animations are amazing!

3blue1brown > Indian guy on youtube >>> my CS teacher

I've watched these videos 3 times and everytime I watch them, I feel a bit smarter. It starts with understanding little to progressively understanding more and more and finally seeing the big picture. Can I say I've understood everything? No. I am getting there and I'll be coming back for these again. For anyone who's feeling discouraged, I can assure you that you'll get there. Thank you so much for creating quality content that brings the driest, most theoretical concepts to life! You're a hero.

I must say it. I am 100% seriously learnt English just to be able watch your videos. This is the content that will help everyone wandering grow stronger in their favourite subjects. I thank you for your work from the bottom of my heart ❤️❤️

I find it astonishing how well you convey some of the intricacies here, way better than most of ML practitioners who teach the public, whether as youtubers, online instructors or public speakers who end up on video on the internet. I very much resonate with the way you frame things, the metaphors you choose, your visualizations and of course your evident love for understanding and sharing thereof. Your work is a great gift to all of us - students, engineers, researchers, philosophers, random viewers from all walks of life. Thank you.

Grant, you're a diamond.

First time I caught myself having a moment of awe while watching educational content. The production value is incredibly high... the way the connections twinkle and move to represent adjusting the weights, the small animations, the descent into various shapes, how the little arrows indicating the desired change move and change size. Beautifully put together! Thanks a lot!

your videos have made it incredibly easier to grasp fundamental concepts of ML. Great work!!

Kurzgesagt and 3Blue1Brown in the single hour! I am in heaven.

I just want to say that I love that you're moving into the mathematics of ML. The visualizations convey the concepts so well!

No form of words can express enough what magic you're creating! I don't know how much you actually think you impact us.. but let me tell you Grant, your effect on my life is immeasurable! And the fact that you learnt it the hard way, and made it so simple for us, so that we don't have to go through the same, makes me respect you even more and more every single day.. Thank you so much.. 3B1B is undoubtedly the best channel on RU-vid..

All of the videos shown on this channel have been written so well. Even I can understand, mathematics was a complete mystery for me in school. I have a sixth grade education and I feel really smart after watching one of these programs. Thanks so much. I'm really grateful that you have taken the time to educate the people that have a hard time understanding but still want to learn. What you're doing is just as important as any other volunteer or charity work. I'm excited I found this .

I am a “seasoned” systems engineer, inventor, and IT professional. IOW, I’m old (40+), lol. Like @Robert MacKinnon has said, you may think that your wonderful videos are only for those new in the field or younger folks in general. This couldn’t be further from the true. Your ability to explain things with clear diction, amazing graphics, and compelling story telling is phenomenal. They help me to not only refresh to get spun up quickly, but also serve as a point of reference for me to direct those newer in my teams; as your ability to run through this at just the right level and pacing far exceeds mine. I would go so far as to suggest that you freelance as a guest speaker or instructor for hire for larger firms having folks wanting to get spun up quickly. Apologies if this is old news and you’re already pursuing this. I simply feel compelled to tell you as we often don’t understand how awesome we are and need to be reminded by others. Consider this your reminder ;) Thanks again for your wonderful videos where you ask for nothing. I would pay for a Patron page if you get one to be able to ask for more content or explanation on topics such as the vanishing gradient problem prone to recurrent neural networks and how LSTM addresses it. I’ve been looking for a good one that does this graphically with intuitions. I haven’t found a good one yet, but I did come here first to check ;) All the best, CCM.

You are a wizard when it comes to animations and understanding

Thank you for what you are doing. I have just started learning ML and I find your video really helpful. Thank you again from bottom of my heart for everything you are doing. I just wanted to let you know that you are really making changes and inspiring youngsters like me.

I have just found your videos. I have enjoyed watching science educators on youtube for many years, but these videos are the best examples of complex ideas being explained clearly whilst still being entertaining. I can't believe I had never heard of this channel before.

I've been reading the book for quite some time but your explanations using animations have pushed my understanding to new levels. So many thanks to you. PS: Keep the background music. Holds our concentration for a long time.

Man, the clarity and the animations make your videos masterpieces

I love that Fermat is labeled as ,,Tease"

Thank you so much for all the time and effort that you put into making a visual understanding of Neural Networks. I've been trying (admittedly on and off) to get a good intuition on Neural Networks for at least a year now. This is by far the best fundamental video I have come across to get an understanding of what the theory behind the scenes is for. I know that this will increase my (and many others') learning curve by a large amount. Thanks again!

I love how you enunciate the words as if you are truly interested and into the topic. The art direction and the animations are also on point. Kudos to the animator!

All I can say is thank you! Every 3b1b video that I have watched is simply incredible. The chronology of ideas introduced throughout a video makes complex topics easy to follow, and the animations are BEAUTIFUL. I have never been able to visualize math so well before. You are an amazing teacher and have contributed invaluable knowledge to society! I'm still in shock that educational resources this profound are available for free. I truly appreciate the work you are doing!

I had to stop seeing your video for a moment because I needed to write down my admiration for your work. It is outstanding the amount of intelligence, communication skills, careful graphic design and passion in very single frame of your videos. Congratulations and thank you for your contribution to maths education. See you in patreon!

Thank you so much for the simplicity in your explanations. You try to explain everything not by their definition, but with an understandable, simple way. Thank you!

* eagerly waiting for a video about convolutional neural networks *

So far, I think this is the best intuitive intro to backprop I've seen. This channel means a lot to me. Because of its teaching style, I managed to get back to learning and grok maths while I was studying machine learning last year, 13 years since doing my undergrad in CS. It's almost the end of 2017 and I still keep hearing from the people I talked to that they fear/hate maths because they think it's a tough subject to tame. May be this example tell us why our education system (internationally) is still broken? As an aside, David Perkins, in his book "Making Learning Whole" also touch about this widespread diseases of the educational system, namely "elementitis". I think we can do better. Grant is doing great work to lower the barrier and making math more accessible to everyone. This is not an easy feat. I think we need some sort of concerted effort for encouraging more people to teaching maths or any subject through intuition. Visualization is one way to improve the teaching methodology. We can also distill intuition from stories, feelings, situations, etc. More examples, see: distill.pub/ colah.github.io/

I am a neuroscientist, and all your comparisons of actual neural networks to artificial neural networks seem pretty spot on to me. This is a great video series.

All of the other commenters beat me to it, but I can't express enough how insanely high the quality of these videos is, and how much they help teach people new things. Thank you so much for making all of them and sharing some of your vast amount of knowledge!

You are amazing!! Best explanation I've seen on the internet yet!!

This series is better than my in total 6-hour lecture from school.

Normally, I never comment on youtube videos, but I'm making an exception for this one. Even though you might not see this, I want to thank you for your contribution to everyone's education. And for FREE access to it too. Amazing videos with amazing presentation, structure, and explanations. I'm 3 years late, but please do keep up the great work!

This series is the best I've seen on neural networks. Thank you for making it. ❤❤❤

I never understood neural nets like this before... Thanks!

Came for the knowledge, stayed for the Animations. Here I'm thinking how it can be done in After Effects, and turned out it's custom engine!

Im taking a Neural networks course at my university right now and I'm finally starting to understand this stuff after watching this video series! Thank you so much!

Ngl man one of my Goals, once I get a good enough job. Will be to be able to live contribute to your patreon. Your videos have been so entertaining, inspiring, and helpful that I hope the move that I give allows you to continue with this channel.

Oh dear! Absolutely brilliant. This is one heck of video. How would you consider creating an exhaustive ML and Neural Network video series? I'm sure there are a lot of curious folk out there, waiting for it!

I love the statement "the most confusing part is the notation". This is so true 😂 It would be really, really nice if you did a last video on how to program this network in a comprehensive way 😄 IMO the inventors of DNN chose all those activation functions and matrice multiplications really carefully to make computation feasible. But all this beautiful simplicity is gone once these unnecessarily complicated math formulas come into place. Honestly, you did an amazing job with your videos on DNNs and backpropagation. The first time I seemed to understand it.

I changed my major from cybersecurity to philosophy and mathematics because of these videos. This is much better than any college course I’ve taken.

Implementing a neural network from scratch for a class project and this series helped me so much to get started. Thanks!!!

This should be art. I don't normally comment on videos but I feel I must express my gratitude and appreciation for your amazing explanations and animations! Visualisation is so important and helpful and you have nailed it beautifully!

You have made life so much easier.

This is the best video on machine learning I have seen so far. Thank you for making this complexity as simple as possible.

your videos are simply amazing.. awesome visualisation of what actually happens...i imagine a huge effort went into making those.... kudos - i almost never comment in that manner... greatest thanks and respect for that!

Now to watch all three parts at once!! Great video 3b1b, keep up the great work man! δ.δ

This man is the eternal fountain of knowledge straight from the heavens

This is definitely one of the best educational video series I have seen. Explanation is concise, abstract when needed and visualizations are phenomenal

I am writing my internal assesment for school about neural networking and thank god your channel exists cuz everything about this subject is now clear to me. You explanations are perfect!

Thank you soooo much! This channel is pure *GOLD*!

9:00 Eureka moment. I cried.

I have watched first 3 videos of this series and i have gained a lot of insight about what basically happens within the black box. Thanks a lot 3Blue 1Brown for these wonderful video illustrations. It took me more than 3.5 hours along with note making. I have still not completely understood NN, but its ok. I am looking forward to video number 4 of these wonderful series. Thanks a ton once again :))))))

Half the time I'm in awe of the quality of the content. The other half I'm learning. Thank you so much man.

I wish I had the option to pay my college tuition to you instead of my university. Well done mate!

This video is literal ART. Grant is my favorite artist.

Am constantly in awe of how intuitive this is.

Thankyou so much for such a visually descriptive and intuitive video regarding Backpropagation in deep learning.

I've never clicked on a video this fast

This is a lifesaver for completing week 5 of Andrew Ng's Machine Learning course!! Been stuck at backpropagation for hours and finally found some clarity in this video. I love your channel! ❤️

You are great at teaching! So many gets lost in the mathematics of it, not caring if their audience understand it or not. You make it incredibly simple! Great job!!

This is by far the best educational content i've ever seen on youtube. Thank you and go on with it.

I might misunderstand but maybe not. At 7:38, we want to reduce the activation of the neuron responsible for 3 right? So we should decrease the activation of the neurons that have a positive weight to 3, no? In the video, it's actually the opposite. For example, the first neuron has a positive weight with the neuron responsible for 3, so we should decrease its activation right? Can someone help me on that please?

AAAAAAAAA YEEEESSSS I'VE BEEN WAITING FOR THIS VID

Really well done. I'll probably have to go through the entire series once again, but that's to be expected. There is so much content in so little time. I'll probably actually get into designing a neural network from the ground up.

You're saving my presentation! I didn't know how to explain neural networks to my fellow students but now I know how to do it! Thanks mate!

You guys just gotta stop blowing my mind ....💥

"This is where we were in the 80s/90s" - I did my degree in the 90s, so this is the level I learned to. The more up to date stuff would be interesting.

It is stunning how much better and clearer you can explain all this than my professor at university, and you are even a lot faster! Thank you so much for producing all those brilliant videos. Some of some help me understand what my professors are trying to tell me, and some of them are just fascinating. Keep on going!

wow, I am surprise anyone knows this sophisticated concepts to explain them clearly, and yet you also accompany them by amazing visuals which I believe not easier to do. Thank you!

Hey! Just want to shout you are my HERO! I'm an AI and ML researcher and I LITERALLY cry watching those awesome explanations! YOU ARE MY HERO!

I think it'd be extremely beneficial if you were a complete lecture series for deep learning. Your way of teach is billion times better than this somewhat popular Andrew Ng lecture series, and also a lot more intuitive. Please consider this:) Best wishes

I cannot overstate how much I wish I had these videos during my machine learning course a few years back

your videos are insanely amazing, and perhaps THE best source to learn any mathematical concept. We are extremely grateful that you take so much time and effort to create such a fantastic content.

I would expect the training data to also include a bunch of random images that the network could classify as "not a number" (not hotdog 😉) That way, instead of just taking its wildest guess at which number it's detecting, it could also intelligently say "this doesn't look like a number to me"

Not drunk, just stochastic

Amazing use of visuals to explain concepts. Well done man ...

Hello, I'd like to start by saying your videos are amazing and beyond intersting. One thing I'm still confused about is how to update the biases through backpropagation? Thanks! and keep it up!

At 12:25, he labels Fermat as a tease 😂

You have no idea how useful this video is to me. I learnt deep learning at uni but honestly this has just helped me through some hard time!

Amazing series; the best I've been able to find on explaining such a complex topic. Thanks!

I feel like I'm missing something with the functionality of backpropogation. So, I get that you adjust the weights between the last two rows to adjust how the second last row feeds into the last row in the optimum way, and then you change the second last way, and then you backpropogate the second row of weights away from the end, etc. But, what I don't understand is that if you change the second last weights, that's going to change the second last row of inputs, which means that the tweaked weights between the last hidden layer and the output will be wrong again. I'm trying to work out what I'm missing. Is it that all the layers of weights are tweaked simultaneously in order to be accurate? If so, I'm still not sure I understand how that works (I'll probably try and watch the video again in the morning when my brains properly awake.)

For me the most difficult part about maths is ALWAYS notation, as they expect me to know what exactly that greek letter stands for in every situation. Or maybe this is what's it's all about

This series is really well done and so important - thank you so much!

This channel is absolutely my favourite!

At 7:37, shouldn't the activations associated with the digit 0 neuron (whose activation should be decreased) change the opposite direction? For example, if the neuron at the very top of the second-to-last layer increases activation, with a positive weight, the digit 0 neuron's activation would increase. Is that correct?

OMG just noticed that Markus Persson (Notch - Creator of Minecraft) sponsored this video.

extremely well made high quality material, every animation, every transition perfectly timed with the explanation. priceless

I love these moments in maths or other sciences, where you have this one singular moment of pure enlightenment. You got me at 7:37 ....I got goose bumps when the first column of "+" appeared. This is just brilliant.

I'm just starting out with Deep learning this quarantine and man...it's a bit difficult to get my head around these concepts.

Why don't you have ads on these. I have no money for patreon and I want to help you out

Your animations are so good!! And when you do recaps and reference to what you have showed us..this is to awesome!

Amazing work !! Such a pleasure to understand those underlying notions ! Thank you so much

Dude how are you making these animations