Rasa Algorithm Whiteboard - Transformers & Attention 1: Self Attention 

I have watched tons of videos and finally an original video that actually teaches these concepts. There are so many RU-vidrs that simply make a video regurgitating something they read somewhere but they don’t really teach anything because they themselves don’t really understand the idea. Bravo. Well done. I actually learned something. Thank you!

After dozens of papers and videos studied, that‘s the first one really make me understand the context. Many thanks fornthat!!! It also highlighted for me one fact: Self attention is a smart idea, but the real magic souce is the way word embeddings are created. That will decide on if the contexts created by self attention make sense or do not.

When the student understands its teacher, it means the teacher understood what he explains. I understood everything contrary to the majority of other youtubers on the same topic. Thanks !

mind blowing how simple self-attention is explained - thank you

Only 4 minutes in and I can tell this series will be a banger. Don't stop teaching, this is pure quality content, much appreciated!

the most intuitive explaination I have ever seem!!! excellent drawing and accent

The guy in the video not just understands the concept, but also understands what it is that understanding of others might lack so he can fill in the gaps.

Best ever explanation of "self-attention". Awesome job.

Can't believe I've only stumbled up this now. Fantastic original explanation.

The clearest explanation of attention mechanisms I've ever seen. Thank you

I love how you use different colors to represent different dynamics between relationships. Such a simple idea, yet so good at conveying meaning

This was a absolutely eye-opening. Congratulations sir! You win the Internet for a while.

This is incredible after like 10 attention explanations i finally get it here

I had so many doubts about the actual operation that happens in self attention. This video just cleared it. Excellent delivery in such a short time.

Best Attention explanation, by far.

Thank you so much for your explanation! When you said: "This is known as SELF ATTENTION". I just thought: BAM! Awesome job Rasa!

This is the best video I have seen so far as to explain things so clearly. Well done.

This is the best explanation of attention mechanism so far for a regular person like me! I came here after going through Coursera NLP spec and several papers, but only now I am actually able to see how that works. Seems like embeddings themselves are the secret sauce indeed. Thank you.

I've known self attention for some time, this is by far the most intuitive video I've ever see, thank you.

Yes sir, this is it. You have nailed it: not only you know the subject; also know he art of creating the condition for everyone else to go into it gradually and logically. Great.

This is the best explanation of Self-Attention mechanism I've encountered after combing through the internet! Thank you!

best video i have found in self-attention.

best video hands down on the self attention mechanism. Thank you!!!

Finally somebody explicitly saying that the distributional hypothesis makes no linguistic sense

Still the best explanation 3 years later

Since there isn't a like/dislike ratio anymore, for those wondering, this video is great

I've been looking for a clear explanation for so so so long. First one I've found. I think all the people that made explanatory videos so far, have 0 understanding of the subject. Thank you.

I've read dozens of papers and blog posts about this topic, but all they do were mostly walking through the math without showing any intuition. This video is the best I've ever seen, thank you very much!

This is the best explaination for the topic I have come across! Thanks!

OMG, this helped me immeasurably. Thanks so much. I just couldn't quite get it from the other explanations I've seen. Now I can go back and probably understand them better. Yay!

The best video on self-attention. No debates

Best explaination I have seen so far .... Thanks!

Best Self Attention Intuition i have ever seen. Andrew Ng Level stuff!

on of the best example on the web that explains attention mechanism , after reading many blogs i still had my doubts , the way attention is explained between time series and text data is brilliant and helped me understand better.

A very simple explanation .. the best one!

Thank you sooo much for this video! Finally, someone explained what the self-attention is!

This attention series is the best clear and intuitive explanation of self-attention out there! Great work!

Never have I ever seen such a great concept explanation. You just made my life easier,@Vincent!!

Thanks for sharing I have watched based on recommendations from a friend on Facebook I will watch the whole playlist. Thanks for sharing, good bless you 🙏 😊

I have read a lot about this: paper, medium, video, this one make me the best understanding. Very nice!

This is really an amazing explanation! Liked how you build up from time-series and go to text. Keep up the good work :)

Have gone through some 10-12 videos on Self Attention. This Attention Series 1,2 &3 are by FAR THE BEST EVER. Many Thanks for these Videos. [came back and updated this comment ;-) ]

This is one of the best videos on the topic, if not the best!

Great series about attention!

best explanation of attention mechanism out there!! thank you a lot!

Love the style. The more talent takes the time to teach new talent, the better. Very appealing style! Subscribed 🦾

please link this video to the tf doc. I tried a whole day to get behind the concept of attention and this explanation is just beautiful!

This is an exceptionally good explanation! Thank you so much. It is easy to see why the thumbs-up count is so high wrt views.

At 11:29, the presenter says "cross product", but I think it's the dot product, so that each of the weights (W_11, etc.) are numbers (otherwise using cross product they would be vectors). Thus we can build a new vector from W_11, W_12, ... Great videos, exactly what I was looking for.

Crazy useful video!

Loved it. Very clear explanation.

Beautiful explanation sir. I'm not sure if i got it all but i could tell you that I've got better idea about self-attention from your video compared to the other ones i watched. Thanks a lot 🙏

This video is the KEY for my QUERY! Pun intended. Thank you so much!

Amazing explanation!! The best I have come across till date. Thank you so much!

This was so interesting! Thank you for this amazing video.

Excellent introduction

SOTA explanation. that's what i was exactly looking for. [tr] okuyan varsa, bu herifi takibe alın, efsane anlatıyor.

Great stuff in here. Super clear and efficient for begginers ! Thanks

The best explanations you can get in the world. Thanks! BTW, were you aware at the time of making these videos that transformers will be so revolutionary?

love the format

By far, the most intuitive explanation of self-attention. DISAPPOINTED However, with the number of Likes:(

Awesome explanation! It takes great skills to explain such concepts. Looking forward!

Thanks a lot! Really amazing , awesome and very clear explanation.

Brilliant explanation

Wow this video is so well done

Amazing explanation ! , it did not find too much sense to the comparison between time series, but the second part was really good :)

Solid gold, thank you.

You are a great teacher. Thanks for this content.

Thank you for sharing, great tutorial, looking forward to watching more and more great ones.

Very informative and simple explanation of a complicated topic. 👍🏻

amazing stuff . Thanks so much Rasa Team :)

This was extremely helpful !!

Thank you for sharing, great tutorial!

Best explanation, that you so much.

Fantastic explanations‌‌ ^_^ Already subscribed!

Well explained! Thank you!

amazing video, thank you very much

Perfectly explained

To filter (in signals (Low Frequency) and programming (filter predicate vector)) means to hold, not to throw away.

this was an excellent explanation. Thank you

very well explained: thank you

Best Explanation

Thank you for posting this educational and useful video. though I can not undetstand everything yet, I'll keep seeing the rest of series and trying to understand :) I mean it.

Great video! thanks

Your whiteboarding is beautiful. How are you doing it? I'd love to be able to present in this manner for my students.

thanks alot, that was very simple and useful. hoping all the best for you

Beauty!

I love your tutorial 😙😙 Best explanation

Just Awesome... thank you

You are the best ....

I got the idea, thanks for the explaination.

Incredible video! Did you mean to say dot product instead of cross product at 11:30?

Why did you use cross product at 11:31? Wouldn't that be making the weights small when the word embedding vectors are similar, which would then mean the related words in the sequence would be unable to influence the current state?

Clearly explained!

this is so genious i love you

Thank you !

Hi. This is a really great visualisation of weightings - thank you! I have a question though: at 11:30 you say you're going to calculate the cross product between the first token's vector and all the other vectors. Should this instead be the dot product, given that you are looking for similarity?

the content for the word vectors are all the OTHER words seen to statistically co-occur in corpora, weighted by their frequencies....so stopwords [the, a, is] cuz they are so frequent, hence dont contribute much topic/semantics, while contentwords are less frequent so contribute more. the word vector ('meaning') for is just all the N words most frequently observed nearby CAT in corpora, discounting for frequency.... works great for cases like [king, queen] cuz they occur in similar contexts in copora... but not for [Noah, cat] cuz that's peculiar/local to this instance..... and also not for co-references [cat, she] which are harder to resolve....you gotta keep a STORY context....where presumably you mighta already seen some reference to ...... and for the co-reference.....well, they're just harder to resolve, tho in this example *HAS* to resolve to either Noa or cat, cuz those are the ONLY choices, and by chance (we assume) all three co-refer..... ==> after all, there's a legit chance that isnt the cat in the example, but the cat's MOM, who can be an ANNOYING MOM, yet nevertheless Noa is still a great cat.....

Thanks for the easy and thorough explanation. I just have one question. How is "Y" now is more representative or useful (more context) than "V"? can you give an example ?

what is the reweigh method that you used in the beginning? i would like to look that up and get more details on it.