Attention for RNN Seq2Seq Models (1.25x speed recommended) 

Shusen Wang, this was extremely beneficial, absolute masterpiece.

Thanks man, I've done my last minute prep for the exam through this video.

Thank you for the fruitful lecture! Instead of α_i, using α_{i,j}=align(h_i, s_j) makes the equation easier to see for me. But it's super helpful for beginners like me, thanks again!

The best on RU-vid, thank you very much

Your explanation was very clear and useful. I strongly recommend this video if you want to understand the concept of the Attention mechanism in RNNs.

very beautifullly and simply explained ,GGs

Best explanation I've ever seen

Amazing explanation 👏 just one question what are A and A prime in this video? h correspond to hidden states of encoder at different time steps

I have a question: what is the output of attention and how do you measure the loss?

Very good explanations thank you very much

Explained nicely. Thank you.

Beautifully explained

This is really good ! Thanks !

In the decoder part of Seq2Seq with attention model, decoder uses three inputs. At first it uses c0, s0, and x'1 to predict s'1, here s0 is the latent representation of encoder and x'1 is the start sign, s0 and x'1 is different. In the next step uses c1, s1, and x'2 to predict s'2. Aren't s1 and x'2 same here? Because s1 is the previous hidden state and the x'2 is the predicted word, which is like a result of probability distribution based on s1. If I am not wrong, it supposed to use only one of them, or always use the s0. Can someone clarify this?

你讲的很清楚！谢谢！

Hi @ShusenWangEng, which template you have use to create this slide? I search but cannot found any slide in overleaf like this. Thanks

What is x' ? And where do you get x'1 from?

Best attention video!

You mention at 11:19 that x1prime is the start sign, later (15:22) you mention x2prime as obtained in the previous step, but how? You show clearly how to obtain s1 and c1 but not x2prime.

Very clear lecture. Thank you!

It is not clear to me what the vector V, used for inner product with tanh of W and hiS0, corresponds.

How it comes at about 7:27 that s0 is suddenly a vector? In the previous slide you state that s0 = hm. Useless video...

the video image is too poor, you need to fix it more

Amazing content bro.lots of hard work . thank you so much .please make more AI playlists like NLP, RL , Deep RL ans Meta Learning with these amazing animations.

at 19:26 the number of weights should be m*t+1 or am i getting it wrong ? because we have c0 as well

It could be better if you can address QKV with your notation. I'm new to attention mechanism and I'm getting confused with some of your notations. But the explanation itself is very clear.

So excited. Great supporting material to Goodfellow textbook. I am building my knowledge for the vision Transformer model.

You're my hero. Marry me! Hahaha this is just a comment to let you know that your explanation can easily be the clearest one on RU-vid to understand attention. Keep up the good work! Thanks a mil!

I was looking for way to implement the encoder decoder , with attention model , with out using the for loop at decoder stage, is it possible???

very well explained ... thank you very much

Astonishing pedagogic effort Shusen! That’s a lot of work involved to share knowledge. Kudos !

Excellent video

So good! Thanks

Very well explained!

7:25 I think concatenation before the linear layer is from the paper of Luong et al. 2015. In Bahdanau et al., the authors performed matrix multiplication with linear layers first (on both h and s), then the concatenation.