Illustrated Guide to Transformers Neural Network: A step by step explanation 

@The A.I. Hacker - Michael Phi please do !

The video actually led me to expect this example as well! It would be highly beneficial.

I agree. I can't seem to find a good explanation on the BERT model

Sorry to be offtopic but does anyone know of a tool to log back into an instagram account?? I stupidly forgot the password. I appreciate any tips you can offer me.

@Matias Santino I really appreciate your reply. I found the site through google and Im trying it out atm. Takes a while so I will reply here later with my results.

@Matias Santino It worked and I actually got access to my account again. I'm so happy:D Thank you so much you saved my account!

@Ronnie Adam you are welcome :)

What about the architecture made you struggle if I may ask?

Strongly agree!

this is critical! I was pretty confused on this for awhile until I realized he swapped the Query and Values on accident.

ah someone else realised it. this comment should be pinned ^^

you are correct

@@mrexnx Correct me if I'm wrong but the only reason you put the mask so it doesn't attend to "future" words in the sentence is cause of the nature of the Ebglish language..since English is written left tor ought unlike other languages. Otherwise you shouldn't have thst mask because you would need to attend to words on right or maybe left also?

@@leif1075 I also thought something like that, that means, in case of Arabic, this direction of masking should not work !

Ik that's true but still it doesn't make sense, What's the use of the masked attentions score that we just computed in that Multi-Head attention layer since it's not carried forward

yes you are right

I asked chatgpt (GPT4) for the same and look at the response: "Let's break down the process: Encoders in Transformers: They take the input sequence and process it through multiple layers of self-attention and feed-forward networks. The final output from the last encoder is used as the "Keys" (K) and "Values" (V) for the subsequent decoder. Decoders in Transformers: Decoders also have multiple layers. In the first self-attention mechanism of the decoder, the target sequence (for example, the partially generated translation) attends to itself. This is analogous to the self-attention in the encoder but has a slight difference. In the decoder's self-attention, future tokens are masked, so they can't be attended to. This ensures that the prediction for a particular word doesn’t depend on future words in the sequence. In the subsequent multi-head attention mechanism, the "Queries" (Q) come from the decoder's previous layer, and they attend to the "Keys" (K) and "Values" (V) from the encoder's output. This essentially means that the decoder is using the information from the encoder to help generate the next token in the sequence. So, your statement is correct: "Value and Key from Encoder with Query from Decoder are combined." In the Transformer's decoder, for every step in its layers, the Queries (Q) from the decoder attend to the Keys (K) and Values (V) from the encoder output."

Unless it’s a decoder only transformer

indeed it is like this as the dot product of the keys and querries construct the relation between the input and the already generated output if noy K and Q where from the encoder it wouldn't capture the relation between the input and already generated output

thanx man . i was gonna ask have you written a papper for it like you did for LSTMs

Also the positional embedding in the illustrated guide is incorrect. Its alternating between even and odd

Your video is the best explanation of Transformers I have ever seen

How do we actually split the queries and keys into multiple values for multiheaded attention?

haha thanks micheal, I was like wtf eveything i learned is wrong at 4:18 until i scrolled down

Thanks! Your content is also super helpful as well and has helped me before

Happy to help!

Thank you! 😄

Same here

Thank you 😃

Thanks, will do!

No, and it's not even understandable how you got to such conclusion.