Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks, Clearly Explained!!! 

To learn more about Lightning: lightning.ai/ Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

This channel is like the Khan Academy of neural networks, machine learning, and statistics. Truly remarkable explanations

This channel is gold. I remember how, for my first coding job, where I had no programming knowledge (lol) but had no choice than to take it anyways, I quickly had to learn php and mysql. To get myself started, I searched for the simplest php coding books and then got myself two books from the php & mysql for kids series, even though I was already in my mid twenties. Long story short, I quickly learned the basics, and did code for a living. Complex topics don't have to be complex, in fact they are always built on building blocks of simple concepts and can be explained and taught as such IMHO. Thank you so much for explaining it KISS style. Because once again, I have to learn machine learning more or less from scratch, but this time for my own personal projects.

I literally searched everywhere and finally came across your channel. seems like gradient descent worked fine .

I can't thank you enough for these tutorials on NLP. From the first tutorial related to RNNs to this tutorial, you explained so concisely and clearly notions that I have struggled and was scared to tackle for couple of weeks, due to the amount of papers/tutorials someone should read/watch in order to be up to date with the most recent advancement in NLP/ASR. You jump-started my journey and made it much more pleasant! Thank you so much!

An awesome video as always! Super excited for videos on attention, transformers and LLM. In the era of AI and ChatGPT, these are going to go viral, making this knowledge accessible to more people, explained in a much simpler manner.

I just wanted to mention that I really love and appreciate you as well as your content. You have been an incredible inspiration for me and my friends to found our own start up im the realm of AI without any prior knowledge. Through your videos I was capable to get a basic overview about most of the important topics and to do my own research according to those outlines. So without taking into consideration if the start up fails or not, I am still great full for you and I guess the implications that I got out of your videos led to a path that will forever change my life. So thanks❤

Coming from video about LSTMs. Again, the explanation is so smooth. Everything is perfectly discussed. I find it immersively useful to refresh my knowledge base. Respect!

Love your videos Josh! Thanks for sharing all your knowledge in such a concise way.

Wonderful tutorial! Studying on Statquest is really like a recursive process. I first search for transformers, then follow the links below all the way to RNN, and finally study backward all the way to the top! That is a really good learning experience thanks!

Great video! thanks for producing such a high quality, clear and yet simple tutorial

This is amazing. Can't wait for the Transormers tutorial to be released.

Another great explanation! It is so comforting to know that whatever I don't understand in class, I can always find a video in your channel and be confident that I will understand by the end. Thank you!

Been waiting for this for so long. ❤. Thank you Josh.

Incredible, Josh. This is exactly what I needed right now!

Absolutely amazing as always, thank you so much. Can't wait for attention and transformers lessons, it will again help me so much for my current internship !

It took me more than 16 minutes (the length of the video) to get what happens since I have to pause the video to think, but I should say it is very clearly explained! Love your video!!

Been waiting for this from you. Love it.

Omg I’m sooooooo happy that you are making videos on this!!! Have been heard it a lot but never figured it out until today 😂 cannot wait for the ones on attention and transformers ❤ Again thank you for making these awesome videos they really helped me A LOT

I genuinely love you for these videos holy smokes

Thank you Joshhh !!! I really love the way you teach everything

Amazing! Can't wait to check out the Self-Attention and Transformers 'Quests!

See this is the kind of explanation I was waiting for❤

Best ML vids out there, thanks!

awesome vid as always Josh :)

This channel is awesome. Thank you

I got my finals of my final course in my final day tomorrow of my undergraduate journey and you posted this exactly few hours ago.. thats a triple final bam for me

Wow man, triple bam indeed, the concept is crystal clear to me now !

Yaas more on Transformers! Waiting for statquest illustrated book on those topics!

Can't wait to see the stanford parser head structure explained as a step towards attention!

Really well explained! Thnx! :D

you are an excellent teacher

I liked it a lot. thanks ❤

thank you so much, I learn from this vedio a lot about LLM

Best channel ever ❤

Great explanation!

This video is very helpful... BAM!

Thank you, so I now can have intuition of why the name is encoder and decoder, that I've curious for full 1 years.

love your songs so much

This is pretty cool!

Another amazing video and I cannot thank you enough to help us understand neural network in a such friendly way! At 4:48, you mentioned "because the vocabulary contains a mix of words and symbols, we refer to the individual elements in a vocabulary as tokens" . I wonder if this applies to models like GPT when it's about "limits of the context length (e.g., GPT3.5, 4096 tokens) or control the output token size.

yoooo Lets goooooo , Josh posted !

you posted this video when I needed the most Thanks man and really awesome 👍🏻

Love your vids

Wow. Splendid!..

14:34 seems like a painful training, but one that, added to great compassion for other students, led you to produce those marvels of good education materials!

using as the first input in the decoder to start the whole translation does appear to be magical

Thank you, Josh. You are amazing. Would you please teach Graph Neural Networks?

We are getting to Transformers. LEETS GOOO

You are amazing TRIPLEBAAAAAMMMM

You're videos are really amazing... ❤ Can you make a video on boltzmann machines?

Hey, thanks for your awesome work in explaining these complex concepts concisely and clearly! However, I did have some confusion after watching this video for the first time (I cleared them by watching it several times) and wanted to share these notes with you since I think they could potentially make the video even better: 1. The "ir vamos y " tokens in the decoding layer are a bit misleading in two ways: a. I thought "ir" and "y" stood for the "¡" and "!" in "¡Vamos!" Thus, I was expecting the first output from the decoding layer to be "ir" instead of "vamos." b. The position of the "" token is also a bit misleading because I thought it was the end-of-sentence token for "¡Vamos!" and wondered why we would start from the end of the sentence. I think " ir vamos y" would have been easier to follow and would cause less confusion. 2. [6:20] One silly question I had at this part was, "Is each value of the 2-D embedding used as an input for each LSTM cell, or are the two values used twice as inputs for two cells?" Since 2 and 2 are such a great match, lol. 3. One important aspect that is missing, IMO, in several videos is how the training stage is done. Based on my understanding, what's explained in this video is the inference stage. I think training is also very worth explaining (basically how the networks learn the weights and biases in a certain model structure design). 4. Another tip is that I felt as the topic gets more complicated, it's worth making the video longer too. 16 minutes for this topic felt a little short for me. Anyways, this is still one of the best tutorial videos I've watched. Thank you for your effort!!

Damm again awesome stuff.

These videos are doing god's work. Nothing even comes close.

I'm a student who studies in Korea. I love your video and I appreciate that you made these videos. Can I ask you when does the video about 'Transformers' upload? It'll be big help for me to study NLP. Thank you.

Thank you Professor Josh, now I understand the working of Se2Seq models completely. If possible can you make a python based coding video either in Keras or Pytorch so that we can follow it completely through code? Thanks once again Professor Josh !

Hello Sir, I was going through your stats videos (qq plot, distribution etc)and loved your content. I would be really grateful, if you can make something regarding a worm plot. Nothing comes up on youtube when I search it.

this is my homework assignment today. how did youtube know to put this in my feed? maybe the next statquest will explain. 😂

Please do a series on time series forecasting with fourier components (short-time fourier transform) and how to combine multiple frame-length stft outputs into a single inversion call (wavelets?)

谢谢！

Hi Josh, Thanks for the much-needed content on encoder-decoder! :) However, I had a few questions/clarifications in mind: 1) Do the number of cells between each layer within the Encoder or Decoder be the same? 2) From the illustration of the model, the information from the second layer of the encoder will only flow to the second layer of the decoder. Is this understanding correct? 3) Building off from 2), does the number of cells from each layer of the Encoder have to be equal to the number of cells from each corresponding layer of the Decoder? 4) Do the number of layers between the decoder & encoder have to be the same? I think my main problem is trying to visualise the model architecture and how the information flows if there are different numbers of cells/layers. Like how would an encoder with 3 layers and 2 cells per layer connect to the decoder that perhaps have only 1 layer but 3 cells.

Hello, thank you for the wonderful tutorial once again. Just a question about word2vec output of embedding values, I'm a bit confused as to how we can input multiple embedding values from one word input into LSTM input. Unrolling it doesn't seem to make sense since its based on one word, if so, do we sum up all these embedding values into another layer of y=x and with weights associated them in order to get a single value for a single word input?

Awesome!

Thank You Sir...................

This channel is great. I have loved the series so far, thank you very much! I have a question: Why do we need a second layer for the encoder and decoder? Could I have achieved the same result using only 1 layer?

Hi Josh - this one didn't really click for me. There's no 'aha' moment that I get with almost all your videos. I think we need to walk through the maths - or have a a follow up - even if it takes an hour. Perhaps a guest lecturer or willing student (happy to offer my time) ... alas I guess as the algorithms become more complex the less reasonable this becomes, however you did a masterful job simplifying CNN's that I've never seen elsewhere so I'm sure if anyone can do it, you can! Thanks regardless - there's a lot of joy in this community thanks to your teaching.

cant wait for the tranformers video

Thanks!

Once again, we can't appreciate you enough for the fantastic videos! I'd love a clarification if you don't mind. At 8:44 - 8:48, you mentioned that the decoder has LSTMs which have 2 layers and each layer has 2 cells. But, in the image on the screen, I can only see 1 cell per layer. Is there something I'm missing? Meanwhile, thanks a lot for replying on your videos. I was honored when you replied promptly to comments on your previous video. Looking forward to your response on this one.

perfect as usual🦾

Dr. Starmer thanks for the video and I had a doubt about this one. While I could understand the training cycle of the model I ain't quite sure about how inference testing is done, because during inference there wont be any tokens to be fed into the decoder side of the model, then how would it come up with a response? If I have to keep it crisp I couldnt understand how the architecture distinguishes training from inference? Is there some signal passed into the decoder side of the model.

Awesome Video. Please make a video on GAN and BPTT. Request.....

Great explanation, love it! PS do you have a suggestion for where I can learn to work with seq2seq with tensorflow?

What is the activation function used in the output fully connected layer (between the final short-term memories and the inputs to the Softmax)? Is it an identity activation gate? I see in various documentations "linear", "affine", etc.

hi, 9:00 does the deco connects to the encoder 1 on 1? or do we have to connect each deco output to each encoder input all to all fully connected fashion?

Love it

300 million bams! ❤

Great lecture Josh!!! What is the significance of using multiple LSTM cells since we already have multiple embeddings for each word? TIA

😀😀😀Love it

Great video! Finally some good explanation! I have a question regarding SOS and EOS tokens, sometimes it is mentioned that the decoder start the process of decoding by taking the SOS token, how does the whole picture differ then, for the both input sentences we always have then SOS and EOS tokens?

Hey... Hope u r doing good..... So u are about to reach MEGA BAMMMMM

Hello! Awesome video as everything from this channel, but I have a question: how do you calculate the amount of weights and biases of both your network and the original one? If you could break down how you did it, it would be very useful! Thanks!

Hi Josh, I have a question at time stamp 11:54. Why are we feeding the token to the decoder, shouldn't we feed the (start of sequence) token to initiate the translation? Thank you for sharing these world-class tutorials for free :) Cheers!

In this full network where does we are telling to convert English word to Spanish word? for example in LSTM OR in Neural network before SoftMax function?

This is what the internet is made for, world class education at home for free.

When we connect the outputs from layer 1 to layer 2, do we connect both long/short memories or only the short term memory?

Hi great video! Just a question, to give the input to lstm, the input length will be constrained by lstm length right? For example 'let's' in first one and 'go' in second one.

can someone explain to me more thoroughly what is the purpose of the multiple layers with multiple LSTM cells of the encoder-decoder model for seq2seq problems because i didn't understand it too well from the video as the explanation was too vague. but still it's a great video 👍

So, if I understand correctly, the context vector in this example has 8 dimensions? 2 dimensions to the word embedding, times 2 since each layer outputs long and short term states, times two because there are two layers. So the context vector can be represented by 8 scalars...?

Thanks for the video Josh, it’s very clearly explained. I have a technical question about the Decoder, that I might have missed during the video. How can you dynamically change the sequence lenght fed to the Decoder? In other words, how can you unroll the decoder’s lstms? For instance, when you feed the token to the (let’s say, already trained) Decoder, and then you get and feed it together with the token, the length of the input sequence to the decoder dynamically grows from 1 () to 2 (+). The architecture of the NN cannot change, so I’m unsure on how to implement this. Cheers! 👍🏻👍🏻

Awesome videos! I was wondering how do people training larger models, know "im ready to press train" on the big version? Because if some of their assumptions were wrong they wasted all that time training. Is there some smaller version they can create to verify theyre getting good results, and theyre ready to train the big one?

Hello Josh !!😊

Hi all, with this seq2seq, can we apply for embedding a sentence, than using the output vector for semantic similarity ?

Oups 🙊 What is « *Seq2Seq* » I must go watch *Long Short Term-Memory* I think I will have to check out the quest also *Word Embedding and Word2Vec…* and then I will be happy to come back to learn with Josh 😅 I am impatient to learn *Attention for Neural Networks* _Clearly Explained_

Do you need the same number of lstm cells as there are embedding values?

hi love your videos can make a video about "Association Rules"

First of all, thank you so much for the clear explanation! I was confused when you said in the decoder during training that the next word we will give to the LSTM is not the predicted word, but we will use the word in training data. How will you let the network know whether the predicted token is correct?

Thank you so much sir for your clear explanation! But I have a question is that if you do word embedding for all tokens in d (let's say >2) dimensions, is that mean we can use the number of LSTM cells as d rather than just 2 cells for each layer? Or even more deep layers not just 2? Thank you!

Hi sir, one question can there be a case where number of layers in decoder could be different than the encoder. Or it can never happen due to size of context vector? will adding a new layer in decoder give any advantage?

Is the matching of number of embeddings to number of LSTM cells per layer a coincidence or does each LSTM cell read/receive one of the embedding dimensions? (simple example had 2 -> 2, Seq2Seq paper had 1000 -> 1000)