LSTM is dead. Long Live Transformers! 

Good to see Adam Driver working on transformers 😁

Thank you for this concise and well-rounded talk! The pseudocode example was awesome!

That's one of the best deep learning related presentations I've seen in a while! Not only introduced transformers but also gave an overview of other NLP strategies, activation functions and also best practices when using optimizers. Thank you!!

The best presentation/explanation to the topic I have seen so far. Thanks a lot :)

I love this presentation Doesn't assume that the audience knows far more than is necessary, goes through explanations of relevant parts of Transformers, notes shortcomings, etc; Best slideshow I've seen this year, and it's from over 3 years ago

Thanks for this! It gets to the heart of the matter quickly and in an easy to grasp way. Excellent.

World deserve more lectures like this one. I don't need examples on how to tune U-net, but the overview of this huge research space and ideas underneath each group.

Great talk. It's always thrilling to see someone who actually knows what they're supposedly presenting.

This is like 90% of what I remember from my NLP course with all the uncertainty cleared up, thanks!

Leo is an excellent professor. He explains difficult concepts in an easy-to-understand way.

This finally made it clear for me why RNNs have been introduced! thanks for sharing

I was trying to use similar super-low frequency sine trick for audio sample classification (to give network more clues about attack/sustain/release positioning). Never did I know, that one can use several of those in different phases. Such a simple and beautiful trick The presentation is awesome

This is beautiful. Clear and concise!

Best transformer presentation I’ve seen hands down. Nice job!

Thank you for sharing it. Really helpful!!

Wonderful and educational, value to those who need it!

Excellent presentation! Perfect!

Such a useful talk! TYSM 🤗

Its hard to overstate just how much this topic has(is) transformed the industry. As others have said, understanding it is not easy because there are a bunch of components that don't seem to align with one another and overall the architecture is such a departure from the most traditional things you are taught. I myself have wrangled with it for a while and its still difficult to fully grasp. Like any hard problem, you have to bang your head against it for a while before it clicks.

More vids please this was really informative on what actual SOTA is

This helped me a ton to understand the basics. Thanks!

This was fantastic. really well presented.

12:56 the review of the pseudocode of the attention mechanism was what finally helped me understand it (specifically the meaning of the Q,K,V vectors), what other videos were lacking. In the second outer for loop, I still don't fully understand why it loops over the length of the input sequence. The output can be of different length, no? Maybe this is an error. Also, I think he didn't mention the masking of the remaining output at each step so the model doesn't "cheat".

Excellent talk. Thank you @leopd !

One of the best talks on Deep Learning!...thank you

Thanks! Really good compare/contrasting.

very impressive presentation. thank you.

this is easily the best NLP talk ive heard this year

This is one of the clearest and most informative presentation about nlp models and their comparison. Thank you so much.

Great presentation!

Great Presentation!

What a great explanation, thank you.

interesting looks a lot like my signal class. how to implement various filters on a dsp.

Excellent talk. Kudos!

Clear, precise, fluid thank you!

Wow... that was a quick summarization of all the NN research things in past many decades...

Well done and thank you

This is hands down the best presentation on LSTMs and Transformers I have ever seen. The speaker is really good. He knows his stuff.

Amazing presentation

Great presentation.

This talk is awesome!

Clear and concise👍

All i want is his level of humbleness and knowledge

You folks need to look into asymptotics and Padé approximant methods, or for functions of many variables as ANN's are you'd use the generalize Canterbury Approximants. The is not yet a rigorous development in information theoretic terms, but Padé summations (essentially repeated fraction representations) are known to yield rapid convergence to correct limits for divergent Taylor series in non-converging regions of the complex plane. What this boils down to is that you only need a fairly small number of iterations to get very accurate results if you only require approximations. To my knowledge this sort of method is not being used in deep learning, but has been used by physicists in perturbation theory. I think you will find it extremely powerful in deep learning. Padé (or Canterbury) summation methods when generalized are a way of extracting information from incomplete data. So if you use a neural net to get a few first approximants, and assume they are modelling an analytically continued function, then you have a series (the node activation summation) you can Padé sum and extract more information than you'd be able to otherwise.

Amazing video, I feel like I actually have a more concrete grasp on how transformers work now. The only thing I didn't understand was the Positional Encoding but that's because I'm unfamiliar with signal processing.

This was more than meets the eye

Great video. Thanks

This is by far the best video. Ever.

This is outstanding!

Wonderfully clear and precise presentation. One thing that tripped me up, though, is this formula at 4 minutes in: Hi+1 = A(Hi, xi) Seems this should rather be: Hi+1 = A(Hi,xi+1) which might be more intuitively written as: Hi = A(Hi-1,xi)

RIP LSTM 2019, she/he/it/they would be remembered by....

Amazing talk. It would be of great help if you can post link to the documents.

Great stuff.

This was amazing.

Great presentation. Thank you!

Very good presentation

Simply Wow!

For anyone feeling overwhelmed, it is completely reasonable, as this video is just a 28 minute recap for experienced machine learning practitioners, and lot of them are just spamming the top comments with "This is by far the best video", "Everything is clear with this single video" and all.

He is incredible One of the best presenters

Great summary! Wonder if you have a collection of talks you give on similar topics ?

this is awesome!!! thank you

Best Transformer explanation ever.

Very well explained... Love it.

where can I find the presentation doc of this talk amigos? thanks

Four years ago! Shocking.

This is brilliant.

-sorry for the lack of technical terms- I did not completely get it how transformers work regarding to positional information: Isn't X_in the information of the previous hidden layer? That is not enough for the network, because the input embeddings lack any temporal/positional information, right? But why not just add one new linear temporal value to the embeddings instead of many sinewaves at different scale?

Amazing!!!

Great ! Thanks

A great introduction to transformers.

Very nice recap of Transformers and what sets them apart from RNNs! Just one little remark, you are not doing things in N^2 for the transformer since you fixed your N to be at maximum some sequence length. You can now set this N to be a much bigger number as GPUs have been highly optimized to do the according multiplications. However, for long sequence lengths, the quadratic nature of an all-to-all comparison is going to be an issue nonetheless.

Thank you...

Nice video but forced to watch on 2x speed trying not to fall asleep

Does the multi-headed attention + position encoding work equally well and better than plain vanilla LSTM but on numeric input ( float or integers ) vectors / tensors ? Your input is highly appreciated

Awesome!

great talk!

I've always wondered how standard Relu's can provide non-trivial learning if they are essentially linear for positive values? I know with standard linear activation functions any deep network can be reduced to a since layer transformation. Is it the discontinuity at zero that stops this being the case for Relu?

I have a question: is it possible to use those SoA models shared in the very end od the presentation as document embeddings? Something analog to doc2vec. My intent is to transform documents into vectors that well represents them and would allows me to compare the similarity of different documents

@10:30 - Attention is all you need -- Multi Head Attention Mechanism --

question regarding 26:27 so if i plan on analysing time series sensor data should i stick to LSTM or is the transformers model a good choice for time series data?

Well done--thanks!

Presentation: perfect Explanation: perfect me (every 10 mins): " but that belt tho... ehh PERFECT!"

Thankyou!

good video!

Thank's so much for video . can'i ask some one if he know where i can find a pre-trainded modele to identfiy number in Image that are from 0 to 100. No writied by hand specialy and can be any where position in image ? Thank's for adavance.

When I want to use transformers for time series analysis while the dataset includes individual specific effects. What do I do? In this case the only possibility would be to match the batch size with the length of the individual data length? Right?

impressive!

Dude, can you share your PPT or PDF. Thanks in advance!

20:00 If I multiply a small scaling factor λ₁ (e.g. 0.01) to the output before feeding to activation function, sigmoid will be sensitive to difference between, say, 5 and 50. Similarly, if I multiply another scaling factor λ₂ (e.g. 100) to the sigmoid output, I can get activated output ranging between 0 and 100. Is that a better solution than Relu, which has no cap at all?

I need that belt.

great talk

perfection

Was curious about machine learning and feel like I'm getting a lesson on how to speak in heirogliyphs.

Relevance is just how often a word appears in the input? NM on this. I looked it up. The answer is similarity of tokens in the embedding - ones with higher similarity gets more relevance.

6:41 hahaha this is GODLIKE! The fact that Schmidhuber is on there makes the joke even better!

Great! thx

This aged like fine wine.

ooooh I so want to see a documentary about this ==> @25:20

I had a tutorial few hours ago on how to build an LSTM network using TF only, left me feeling completely stupid. Thank you for showing there is a better way.

Excellent vid, am wondering about a point made around 22:00 about SGD being "slow but gives great results." I was under the impression that SGD was generally considered pretty OK w/r/t speed, especially compared to full gradient descent? Maybe it's slow compared to Adam I guess, or in this specific use-case it's slow? Perhaps I'm wrong. Anyways, thanks for the vid!