The Forward-Forward Algorithm 

Thank you! As soon as I have time, I'll also try to add a video about the FF Alg PyTorch code.

Thank you for the tip! I'll try that with the next video.

I think just a linear classifier, of which there are many. It does feel like a bit of a cop-out to do all this work to learn a representation only to have to do some ML on top of it all to actually make predictions, but at least that model is a simple one that can fit without back propagation of gradients through a deep network.

That's a great question! I've started implementing Hinton's one, but then looking on other GitHub implementations, I found that one which showed faster convergence performance. However, if you think about, it's only a slight modification of Hinton's sum of the squared activities. I'll highlight better the difference in the upcoming video.

Hi! The idea under the loss proposed by Hinton is simple: make neurons of a layer react to positive sample and stay neutral to positive ones, looking only at their inputs. If you think at a binary classification, this is simpler to grasp. Indeed, if you've only two classes, a positive one and a negative, and you can get the last layer of the DNN to "activate" with positive ones, you can easily classify your inputs. Is it clearer?

​@@escvm In other words, the loss was configured in a way that only positives were activated. thank you It's more clear.

Hi! Yes sure! What can I do for you?

Hi! This's a good question! I'll try to cover it in the coding session. Anyway, the threshold is a hyperparameter of the algorithm. So, it needs to be tuned with each problem addressed. The threshold affects the loss and how activations' sum of squared should be to minimize the overall loss. You can try to play with it, and you should notice that its most evident effect is on convergence speed.

Hi! I'm curious: why are you suggesting that? :)

@@escvm have you seen the work of that guy, who did an experiment, in which he use false positive of pre neural face detection, then average them over 1million data, in which it revealed an average human face? I have this inside jokes that all neural is really averages, noises and relaxation. Given than trained network are sparse, we aren't optimizing correctly the actual capacity of the network. That experience and the relative efficiency of simple bag of word and ngram model to extract some degree of high level semantic, it prove that there is something more fundamental in the frequency distribution of the data set. Something we can probably extract and bypass guessing hyper parameter. That's for the context of the joke, now I said diffusion because, like with other training, you will set the neural weight to random, then using a training sample you will try to remove the noise by relaxing the weight around the sample activation, given all sample that kinda sound like a convoluted way to do averages. Now Alex J Champagnard, did some very early experiment with, if my memory is exact, it's call extreme neural network, that is using the network without training it, and fir image generation, you could retrieve images with similar features than the original, the conclusion was that the architecture probably had a bigger impact than the training, in some way, and that's my other joke, network weight are really just attention mechanics, we just got better at designing the architecture. The last observation is that, when you look at transformer weights activations, they try really hard to undo some data we put in the embedding, like removing the offset of the positional encoding. Given the network activation are already sparse, it's funny to see most activation, at least in early layer, are all about finding data we already know but obfuscated instead of passing them directly. I feel like there is a solution that would be simpler than training network, closer to Huffman encoding but taking into account the insight we got with neural network, that is there averages but also (hierarchical) differentiations of the statical spread, that hold all the relevant information without training. After all self attention is a kind if ngram, and it's use to select the right average in the FF layer. We wouldn't need blind walk in hyperdimensional noise. The data is all you need. I don't know if that makes sense.

Imagine doing machine learning and be confused by the backprop.

Hi @adsick5014 I'm sorry to hear that. Yes, I agree that sometimes formulas and jargon can make simple concepts hard to grasp. Nevertheless, it is also hard to find the right compromise between being too superficial or too rigorous. Anyway, I think if there's anyone good at this is 3Blue1Brown, and it has a super nice series on backprop. I suggest you to take a look at it: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-aircAruvnKk.html

Andrej Karpathy has some extremely instructive videos on the nuts and bolts of backprop.