Transformer Neural Networks Derived from Scratch 

Thank you so much!

Was coming to comment this, thanks

Please make video

Please!

Damn FNN's and CNN's are basic stuff we were taught in our 4semester of our undergrad. Adam and RNNs were in the "additional resources" section for an Introdcutory course for Deep Learning I took in the same semester. Encountered LSTMs through personal projects lol Still haven't used GANs and Autoencoders, but it they were talk of the town back then due to the diffusion models.

@@newbie8051 yea I did FNN from scratch in high school, I was really hopeful for getting into Ai Research and then the transformers arrived in my college year…

amen. the power of out of the box teachers is infinite.

Would you share the source code for the animations?

Thanks for the tip.

"they were using a convolutional neural network in the transformer" No no, Transformers do not have any convolutional layers, the author of the video just chose CNN as a starting point in the process "Let's start with the solution that doesn't work well, understand why it doesn't work well and try to improve it, changing the solution completely along the way". The main architecture in natural language processing before transformers was RNN, recurrent neural network. Then in 2014 researchers improved it with attention mechanism. However, RNNs do not scale well, because they are inherently sequential, and scale is very important for accuracy. So, researchers tried to get rid of RNNs and succeded in 2017. CNNs were also tried, but, to my not-very-deep knowledge, were less succesful. Interesting that the author of the video chose CNN as a starting point.

@@Hexanitrobenzene, I suppose I’ll have to watch this video again. I’ll look for what you mentioned.

@@terjeoseberg990 A little off topic, but... Not long ago I noticed that RU-vid deletes comments with links. Ok, automatic spam protection. (Still, the thing that it does this silently frustrates a lot...) But, does it also delete comments where links are separated into words with "dot" between them ? I tried to give you a resource I learned this from, but my comment got dropped two times...

...Silly me, I figured I could just give you the title you can search for: "Dive into deep learning". It's an open textbook with code included.

@@Hexanitrobenzene, The best thing to do when RU-vid deletes comments is to provide a title or something so I can find it. A lot of words are banned too.

Thanks for the tip! I'm always happy to answer questions.

Thanks so much!

My one-sentence summary of why transformers matter would be "they are standard CNNs, except the words are re-ordered in a way that makes the CNN's job easier first before being fed ". Also, a single NN layer IS a linear layer; I'm not sure what you mean by saying you don't know how important the distinction between the two is.

thanks@@maxkho00

the first so far this year

Thanks, any title in particular you'd recommend?

The entire model is trained end-to-end to solve the training task. What this means is you have some training dataset consisting of a bunch of input/label pairs. For each input, you run the model on that input, then you change the parameters in the model a bit, evaluate it again and check if the new output is closer to the training label, if it is you keep the changes. You do this process for every parameter in all layers and in all value and score networks, at the same time. By doing this process, the importance score generating networks will change over time so that they produce scores which cause the model's outputs to be closer to the training dataset labels. For standard training tasks, such as predicting the next word in a piece of text, it turns out that the best way for the score generating networks to influence the model's output is by generating 'correct' scores which roughly correspond to how related 2 words are, so this is what they end up learning to do.

this is a good question and it's also a GOFAI type approach where we make the mistake thinking we can inject some human semantic idea to improve a network. but the reality is it will do this automatically without our help. For example papers back in 1986 show tiny networks automatically grouping words into nouns or verbs, it's amazing. let me know if you want more details

Yes that's correct, the transformer just adds x without passing it through an additional linear layer. Including the additional linear layer doesn't actually change the model at all, because when the result of self attention is run through the MLP in the next layer, the first thing the MLP does is apply a linear transform to the input. Composition of 2 linear transforms is a linear transform, so we may as well save computation and just let the MLP's linear transform handle it.

The entire model is trained end-to-end to solve the training task. What this means is you have some training dataset consisting of a bunch of input/label pairs. For each input, you run the model on that input, then you change the parameters in the model a bit, evaluate it again and check if the new output is closer to the training label, if it is you keep the changes. By doing this process, the score generating networks will change over time so that they produce scores which cause the model's outputs to be closer to the training dataset labels. It turns out that the best way for the score generating networks to influence the model's output is by generating 'correct' scores which roughly correspond to how related 2 words are, so this is what they end up learning.

What ever you train it to. People have trained transformers to categorize text, predict the sentiment of sentences, all sorts of things. ChatGPT is specifically trained to predict the next word that comes after a partial piece of text. It turns out that you can use this to generate new text from scratch by repeatedly applying it to its own output. This technique is known as 'auto-regression' and I explain it in more detail in this video: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zc5NTeJbk-k.html

Thank you for your support!

It's almost impossible for that to happen in practice because we compare words against themselves. So if one word has no relationship with any other word in the sentence, it will still have a large score for itself: so the normalized weight will be 1 for itself and 0 for all other words. Which means that its vector won't include information from any other words, but that's kind of what you want if it really doesn't have any relationship to any other words.

@@algorithmicsimplicity Right, of course, that makes sense. I hadn't thought about words having weights for themselves. Thanks! Your channel is really great, I love the level of depth you go into while still keeping the material approachable.

Yep all my videos so far have been done in Manim.

Yeah it's a term I made up so you won't find it in any sources, sorry about that. Usually sources will just talk about self attention in terms of key, query and value lookups, so you can look at those to get a more detailed understanding of the transformer. The value transform is equivalent to the linear representation function I use in the pairwise convolution, the key and query attention scores are equivalent to the bi-linear form scoring function I use (with the bi-linear form weight matrix given by Q^TK). I chose to use this unusual terminology because, personally, I feel the key, query and value terminology comes out of nowhere, and I wanted to connect the transformer more directly to its predecessor (the CNN).

@algorithmicsimplicity, this is a surprising connection. Thanks a lot for the explanation.

@@algorithmicsimplicityit would be great if you could make this connection between terminology in video form. maybe next time?

I"ll be doing this, but in short, the little networks he showed connected to each pair are KQ (word pair representation) and the V is the value network., all of this can be done in the decoder only model as well. and cross attention is the same thing but you are using two separate sequences looking at each other (such as two sentences in a translation network). it's nice to know that GPT for example is decorder only, and so doesn't even need this

The pair vectors are first concatenated together into one vector e.g. [00010100], and this vector is then run through the neural network which produces the output vector. The output is the result of the weights in the neural network. Initially, those weights are completely random (usually sampled from a normal distribution centred at 0), and then they are updated during training. The neural network is trained on a labelled training dataset of input and output pairs. For example, ChatGPT was trained to do next word prediction on billions of passages of text scraped from the internet. In this case, each training example is a random part of a text passage (e.g. "the cat sat on the") and the output is the next word that occurs in the text (e.g. "mat"). For every training example an update step is performed on the neural network to update all of the weights of all of the layers. The update step works as follows: 1) Evaluate the neural network on the input. 2) For every weight in every layer, increase the value of that weight by a small amount (e.g. 0.001) and then re-evaluate the entire neural network on the input. If the new output is closer to the target (e.g. the vector output is closer to the one-hot encoding of "mat") then it was good to change that weights value, so it keeps the new value. If the new output is further away from the target, then it was a bad change, so reverse it. And that's it. Just keep repeating that update step for billions of different inputs and all of the weights in all layers will eventually be set to values which allow the transformer as a whole to map inputs to outputs correctly. Also I should point out that in practice there is a faster way to do the update step which is called backprop. Backprop computes exactly the same result as the update process I described, it is just faster computationally (you only need to evaluate the model twice instead of once for every weight), but it is also more difficult to understand.

The importance-scoring neural network is trained in exactly the same way that the representation neural network is. Roughly speaking, for every weight in the importance-scoring neural network you increase the value of that weight slightly and then re-evaluate the entire transformer on a training example. If the new output is closer to the training label, then that was a good change so the weight stays at its new value. If the new output is further away, then you reverse the change to that weight. Repeat this over and over again on billions of training examples and the importance-scoring neural network weights will end up set to values so that that the produced scores are useful.

The animations were made with the Manim Python library (www.manim.community/ ) and edited with KDenLive.

The "key-query" attention scoring is equivalent to the bi-linear scoring function in my explanation, where the bi-linear form matrix is given by K^TQ. The value transformation V is exactly the linear representation function in my explanation. I still have no idea why they decided to give the scoring function matrix two different names (key and query), it just confuses everyone.

Let's assume X is our input, a sentence containing N words. Each word has embedding dimension of size P. Thus X is an NxP matrix. Then according to the "Attention is All You Need" paper, we have: K = X * W_k Q = X * W_q V = X * W_v A = softmax(QK^T) Output = AV Where: W_k, W_q and W_v are PxP matrices K, Q and V are NxP matrices. A is an NxN matrix Output is an NxP matrix. I am confused about how the V matrix connects to the "pair-wise representations". In the video, you show operations being done on pairs of words (such as 13:40). However, there doesn't seem to be any pair-wise operations occurring when computing V? If there was a pair-wise operation, wouldn't the dimension of W_v need to be NxN instead of PxP? I agree that we are computing an a single "attention" scalar value for each word pair. This is why A has dimension NxN. However, it seems like V contains individual representation of the words that are "smooshed" together when we multiply by A, rather than V containing (or operating on) pair-wise representations? Again, great video! And I greatly appreciate your help!!@@algorithmicsimplicity

@@dmlqdk When you apply the linear transform V to the pair [x1, x2] the result is V1x1 + V2x2. Basically we are applying a linear transform to each input and summing them. Because, in a given column, x2 is the same for every pair, you are effectively just adding a constant value to each V1x_i. You can factor this constant value outside of the attention weights, at which point it just becomes part of the residual term. I explained this process in more detail here: www.reddit.com/r/MachineLearning/comments/17cmzcz/comment/k5t7g70/?context=3 At this point, you no longer have 'pair' representations, since each value vector is now just a linear transform applied to one word. Each column of the [NxN] grid of value vectors contains V1x_i for i in {1,...n}, i.e. all of the columns are identical. Since all of the columns are identical, instead of elementwise multiplying the matrix of attention values by the matrix of value vectors and then summing, you can instead rewrite this operation as a single matrix-vector product, which is what the AV operation is in the standard self attention. V is that column of value vectors, where each entry is just V1x_i.

This makes so much more sense now. Thank you!!@@algorithmicsimplicity