This is more about simulating large associative networks and their dynamics, it has nothing to do with the brain.

Interesting, where can one find open implementations?

@31:19 "Reverse direction of a string" would also be a good answer.

Great talk!

If tic-tac-toe has around 4500 positions, along with a value for each position (d = 10), does this mean we can store all (positions, value) pairs in 100 (d^2) floats (i.e. 400 bytes) and retrieve them with exponential hopfield network? (exp 10) => 22026.465 ... as for chess, with some funny usage of equivalence classes, store the values of all legal positions? (exp 66) 4.6071865e28 in (expt 66 2) => 4356 floats (i.e. 17424 bytes)? If that is true, chess is as good as solved.

You have to have all the patterns stored somewhere in order to retrieve them. This is nothing like a Hopfield network that stores the images inside the weights. This idea of exponential storage only applies to the mask that is created that is multiplied by ALL the stored images to get the result. The stored images do not have the exponential quality. This is very misleading. Someone explain why I am wrong. See the graph in 48.02 to see what I am talking about. All the images are stored already as separate files/vectors.

I watched this seminar and it was very useful for me. Thanks.

Extremely interesting. This shed a lot of light on the PointConv-related papers. Thank you for sharing!

Thank you this is very clear and informative! I do care about neurobiology so I appreciate the attention to biological plausibility.

Thanks a lot for this video. For me, the highlights were the explainability aspects and the connection between size-dependent in-context learning capabilities (from the GPT-3 paper) and Modern Hopfield Networks (ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-k3YmWrK6wxo.html).

Is there a tensorflow implementation of your Hopfield layer? It would be awesome to do some experiments with it and read the code to understand the details.

At ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-bsdPZJKOlQs.html, I do not understand the significance of the connection between the Continuous Modern Hopfield update and the attention mechanism: yes, on a very abstract level, both use scaled softmax of a matrix multiplication. But the transformers derive queries, keys, and values from the data vector by learned linear transformations, whereas the Hopfield update does not specify where the queries and keys come from, and always uses the keys as values. So at ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-bsdPZJKOlQs.html, point 1 (transformer attention), the transformer queries are in fact Y * W_Q for self-attention, so you could leave out R as input, leading to the same layer architecture as point 2 (Hopfield Pooling). Thus I find it confusing to make such strong connections (Hopfield update equals attention) on such an abstract level. For instance, if you allow arbitrary learned or fixed arguments in your Hopfield layer, shouldn't you allow the same flexibility for the attention in a transformer block, and thus the attention in a transformer block could just as well perform pooling or k-nearest neighbor!?

Thanks a lot for the talk, that clarifies a lot from your paper. About the visualizations at ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-bsdPZJKOlQs.html: Shouldn't there be local minima for metastable states that are in the middle of pattern clusters, e.g. for the bottom right picture at least one local minimum somewhere in the middle of the picture (close to the middle of a cluster of patterns)?

I have been looking for someone out there in A. I. who is putting some good consideration into Douglas Hofstadters work. Thank you

Deep Learning looks to be a mile deep and an inch wide. AI needs to find the right balance between exploitation and exploration to make progress, and the directions Mitchell points to not only have the pioneers' endorsement, but also make that very elusive common sense. A tour de force of a presentation!

Hi, thank you so much for the great talk. I was wondering if we could be accessible to the slides. Look so great.

Super interesting content, thanks for uploading. Are they slides available somewhere?

Thank you Prof!

It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first. What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing. I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order. My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at arxiv.org/abs/2105.10461

Melanie is asking the questions everyone should be asking. There is a lack of metacognitive abilities in science and the focus on solving metrics without actually making progress has become so prevalent that it's becoming damaging. There are a lot of lessons in the perspective Melanie provides.

Thanks for sharing this wonderful talk!

Thanks for posting this. However, I'm finding it very hard to understand. For example, at 5:52 I hear "if mewk size is equal to oik size". If you don't speak English well, please speak slowly. Also, if you do speak English well, please speak slowly. This information is too complex and subtle for a hurried presentation.

Fantastic talk!

congratulations Francisco!

🙌🙌🙌👏👏👏

14:50 restricted boltzman machines are far better options at this point.

Very interesting

fascinating, gold mine in here. love the Q&A part most

thank you very much! This is very insightful!

Great! Than you for uploading it online! 💙

If you look at the variance equation for linear combinations of random variables as it applies to dot products you see that fully distributed inputs are prefered. Also especially for high storage density non-linear behavior is needed. Thus to use the dot product as associative memory first you do a vector to vector random projection, then apply non-linear behavior, then do the dot product. You can use a random fixed pattern of sign flipping followed by the fast Walsh Hadamard transform as a quick distributing random projection.

The sound quality could be a bit better.