They explained the topic in a very easy manner even a guy with no background in ML can understand it. I wish they also had the tutorial for the Practical Sessions.
I am the first graduate of my family. It will be near impossible to listen MIT lecture unless there is a initiative like this. Now education/knowledge is open-sourced
While stuck at home over the summer, I decided to code up the infrastructure for a bunch of different kinds of AI. I started with perceptron layers, then added recurrent perceptron layers, then LSTMs (that was particularly hard - I had to intensely study at least a dozen research papers to piece it together before all the unstated pieces gelled) , then convolutional nets, then a transformer (encoder/decoder) infrastructure. What I discovered in this process, is that the amount of "intro" level information, as well as "how to AI in python" sort of stuff, completely drowns out the nuts and bolts. In the end, you're stuck reading research papers, which are targeted at an audience that is already a subject matter expert. I suppose universities are supposed to fill that gap, but honestly, this stuff is just not that hard once you decode the language of the field. It's just differential calculus and a bit of optimization theory. Good 3rd year engineering students have the math background, and combine that with some coding skills and you're golden. The other thing I learned in this process, is that convolutional nets and encoder/decoders are just amazing. Even though I wrote every line of code, and understand how and why they work, and train them myself, it feels like magic to watch them work.
everything about this course is phenomenal. It's so good that sometimes I get distracted by thinking about how amazing the course is. The language and the pace used are perfect. The slides are perfect; there's just the right amount of information on the slides so that I don't get overwhelmed by having too much to read while listening, (something I struggle with in my regular classes). It's just so fascinating. Teachers all around the world should take notes. Thank you so much for sharing this with us.
@@DarkLordAli95 Sure, the slides have been published since last year on the 2020 course site: introtodeeplearning.com/2020/. The most recent course iteration contains the 2021 slides (which are also published, but slightly different from this talk).
I would like to say thank you, Alexander and Ava for making it happen! I would make my personal note for Convulational neural network and RNN in 2am and learning from MIT, thank you
Amazing delivery! I have never thought that I would be able to understand any lecture from MIT 100% because of the lack of pre-requisites knowledge. Looking forward to see more videos on GNN, GCN etc..
Thanks! You may also want to check out next week's lecture which will also be on Deep Sequence Modeling but contain a lot of cool new material on Transformers and Attention. The link will be here but it is not published yet: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-QvkQ1B3FBqA.html
This one is nice. Nicely fit with my class's slides. The best part for me is that it clearly reveals the concept of timestep computational graph, which I have not seen in previous introduction videos.
This series of MIT lectures were very effective and productive, especially in this lockdown and WFH situation. I learned a lot. Thank You !!! Alexander Amini Sir to make it possible.
Hi Ava, Thank you for sharing the knowledge. I have couple of questions: 1. First of all how does back propagation in RNN leads to vanishing gradient but not in case of deep neural network since Mr Amini did not bring it up in the introductory lecture. 2. Secondly you mentioned that having weights as identity matrix will solve the problem of vanishing gradients. So how is that possible. What is the math behind it ? 3. Thirdly, I am confused as to how you are showing the matrix as n*m matrix in the first place. Since the example you took had a sentence which seem to be like a n*1 matrix (where n is the number of words, isn't it?)then how can the weights be a n*m matrix(focusing on the dimensions here). If what is shown is correct then how will you multiple the weight matrix with the example that you have described. 4. And lastly, I could not correlate how relu results in derivative greater than 1. I mean when you use the relu function, then combination of product of weights *x is forced to be a constant value 1 isn't it. So no matter what the value of the dot product of weights and x (and dot product of the state and another weight matrix )is going to be as long as it is greater than zero it will result in a y value of 1, if I understand correctly. However, the value of the function is constant no matter what, then wouldn't that yield the derivative of 0 since the derivative of constant is 0. Maybe I have misunderstood something but you further clarifying this would help certainly. Thanks in Advance!
Please, correct me if I'm wrong. At 36:18, "uninterupted gradient flow" is essentially due to the fact that operations along the route c0,c1,c2,c3,... has no weights to be updated at all. From what I can see, there are kinds 3 operations along that route: [1] point-wise multiplication (type C x C -> C) [2] point-wise addition (type C x C -> C) [3] copying (type C -> C x C) where C is whatever data type c0,c1,c2,c3 is.
I struggled with this when I came across and ended up resorting to delving into the maths to grasp why LSTMs solve the vanishing/exploding gradient problem. "Uninterrupted gradient flow" does not explain sufficiently. I found this article to be of use (if you can wade through the maths it really explains why LSTMs are better than RNNs for gradient problems) medium.com/datadriveninvestor/how-do-lstm-networks-solve-the-problem-of-vanishing-gradients-a6784971a577
Absolutely excellent explanation! To further motivate and inspire your great work, where the ball goes next is the holy grail of all models of everything: Quantum Mechanics. You can easily infer the quantum "genome" sequence of all reality with this technology. Again, great presentation and content
Great Lecture Series! I just tried to download the slides from your site, but they are unavailable; the site just states that slides and videos are upcoming for 2021 lectures. Do you know where to find the 2020 lecture slides?
Thanks Ava....for sharing us informative lectures....Learning a lot from your videos...Can I ask any questions from you people either in slack or somewhere else?
I am getting an error while playing the songs ('C:\Users ame\PycharmProjects\tensors\venv\lib\site-packages\mitdeeplearning\bin\abc2wav' is not recognized as an internal or external command, operable program or batch file.)
Hi all I was just confused about something. The point of RNNs is to preserve the order in a sequence. Let us say our sequence is "I took my cat fora walk". If we use a really large fixed window and make it into a fixed vector to put into a Vanilla feedforward network is the order preserved? I guess what I am really asking is can a feedforward network preserve order?
How does the vanishing gradients problem is solved by the LSTM? If the LSTM includes sigmoid operations, doesn't that contribute to make gradients smaller?
I am 30 now and I wish to have these sort of contents on my school/college days. Still trying to learn as much as possible. Thank you for your contribution to democratize world class education
Thank you for sharing knowledge and making it available for all. I am feeling difficulty in understanding the lecture videos, I had never before learnt anything about ML or DL but I really enjoyed the first video. Is there any perquisite information required for this series of lecture?
it's a single RNN cell with more information (and basically the addition of a cell state and also the gates). One unit in the layer, not quite a single neuron.
A question related to lab session 2: Why does the last code block show that there are no songs in the text when clearly I generated it using the generate_text function in the previous block?
If a neural net generates music, after training using a copyrighted music, does the rights of generated music still belong to the copyrighted music creator?
I noticed that you cut some of the student questions out of the videos. I think you should include all of them in, because it would make us understand better in my opinion. Of course if a student does not want you to publish it, that's fine.
Its vague in the diagrams... in reality the input to each gate is calculated as: weight * ht-1 + weight * xt + bias, where each gate has their own weight and bias and ht-1 is the previous 'hidden state' and xt is the new input.
@@mattrowlands5751 I thought about it and most probably they represent flow of information with the tanh box representing tanh(Whh*ht-1 + Wxh*xt) i.e the entire function/expression has been abstracted to that box.
