I was actually expecting New York when they added America. As a child I always thought New York was the capital of the U.S., I was at least around eight when I learned that it wasn't. Similarly, when people talk of Australia's cities, Canberra is rarely spoken of, but Sydney comes up a lot.
“What does the fox say?” “Don’t they go ‘ring ding ding’?” “Not in this dataset”
5 лет назад
Train the same algorithm on songs instead of news articles and I figure you could get some really interesting results as well. Songs work on feelings and that should change the connections between the words as well - I bet the technology can be used to tell a lot about the perspective people take on things as well.
@ Songs also use specific rhythmic structures; assuming most of your data was popular music, I bet that there'd be a strong bias for word sequences that can fit nicely into a 4/4 time signature, and maybe even some consistent rhyming structures.
I did this for my final project in my bsc. Its amazing. I found cider - apples + grapes = wine. My project attempted to use these relationships to build simulated societies and stories.
Fun points: A lot of the Word2vec concepts come from Tomáš Mikolov, a Czech scientist at Google. The Czech part is kinda important here - Czech, as a Slavic language, is very flective - you have a lot of different forms for a single word, dependent on its surroundings in a sentence. In some interview I read (that was in Czech and in a paid online newspaper, so I can't give a link), he mentioned that this inspired him a lot - you can see the words clustering by their grammatical properties when running on a Czech dataset and it's easier to reason about such changes when a significant portion of them is exposed visibly in the language itself (and learned as a child in school, because some basic parts of it are needed in order to write correctly).
I like this guy and his long sentences. It's nice to see somebody who can muster a coherent sentence of that length. So, if you run this (it's absurdly simple, right), but if you run this on a large enough data set and give it enough compute to actually perform really well, it ends up giving you for each word a vector (that's of length however many units you have in your hidden layer), for which the nearby-ness of those vectors expresses something meaningful about how similar the contexts are that those words appear in, and our assumption is that words that appear in similar contexts are similar words.
This was weirdly fascinating to me. I'm generally interested by most of the Computerphile videos, but this one really snagged something in my brain. I've got this odd combination of satisfaction and "Wait, really? That works?! Oh, wow!"
It was given ‘oink’ minus ‘pig’ plus ‘fox’ though, not fox + says. So we’d expect to see the same results as for cow & cat etc. of it “understanding” that we’re looking at the noises that the animals make. Obviously it’s not understanding, just an encoding of how those words appear near each other, but we end up with something remarkably similar to understanding.
This thing would ace the analogy section of the SAT. Apple is to tree as grape is to ______. model.most_similar_cosul(positive['tree', 'grape'], negative['apple']) = "vine"
Rob Miles and computerphile thank you... IDK why youtube gave this gem back to me today (probably for my insesent searching for the latest LLM news these days) but I am greatful to you even more now than I was 4yrs ago... Thank you
This is basically node embedding from graph neural networks. Each sentence you use to train the it can be seen as a random walk in the graph that relates each world with each other. The number of words in the sentence can be seem as how long you walk from the node. Besides "word-vector arithmetics", one thing interesting to see would be to use this data to generate a graph of all the words and how they relate to each other. Than you could do network analysis with it, see for example, how many clusters of words and figure out what is their labels. Or label a few of them and let the graph try to predict the rest of them. Another interesting thing would be to try to embed sentences based on the embedding of words. For that you would get a sentence and train a function that maps points in the word space to points in the sentence space, by aggregating the word points some how. That way you could compare sentences that are close together. Then you can make sentences-vector arithmetics. This actually sounds like a cool project. I think I'm gonna give it a try.
I love how this video seems to have time travelled from 1979 - the austere painted brick classroom, the shirt, the hair and beard, even the thoughtful and articulate manner seem to come from another time.
This was soooo interesting to me. I never dug deeper in how these networks work. But so many "Oh! That's how it is!". When I watched the video about GPT-2 and you he said that all the connections are just statistics, I just noted that internally as interesting and "makes sense" but didn't really get it. But with this video it clicked! So many interesting things, so thanks a lot for that. I love these videos. And seeing the math that can be done with these vectors is amazing! Wish I could like this more than once.
is the diagram with angles and arrows going off in all directions just for us to visualise it rather than how computers are looking at it, I didn't think they'd be calculating degrees. I thought it would be more about numbers of how close the match is like 0-100
I love that I have been thinking about modelling natural language for some time now, and this video basically confirms my way of heading. I have never heard of word embedding, but its exactly what I was looking for. Thank you computerphile and youtube!
I would suspect that this has to be very similar to how our own brains interpret languange, but then again evolution has a tendency to go about solving problems in very strange and inefficient ways.
@@maxid87 mammals have a nerve that goes from the brain to the throat, but due to changes in mammals it always goes under a vien in the heart then back up to the throat. This is so extreme that on a giraffe the nerve is like 9 feet long or something. In general evolution does a bad job at remmoving unnecessary features.
@@wkingston1248 how do you know that this is inefficient? Might seem like that at first glance but maybe there is some deeper reason for it? Are there actual papers on this topic that answer the question?
I doubt there is a lot of evolution at play in human language processing. It seems reasonable to assume that association (cat~dog) and decomposition (Tokyo = japanese + city) play an important role.
You could make a game with that, some kind of scrabble with random words, add and substract words to get other words. Maybe with the goal to get long words or specific words or get shortest or longest distance from a specific word.
Thanks for a great explanation of word embeddings. Sometimes I need a review. I think I understand it, then after looking at the abstract, n-dimensional embedding space in ChatGPT and Variational Autoencoders, I forget about the basic word embeddings. At least it’s a simple 300-number vector per word, that describes most of the highest frequency neighboring words.
Me too. I loved the review after looking how GPT4 and its code/autoencoder-set looks under the hood. I also had to investigate the keywords being used like "token" when we think about multi vector signifiers and the polysemiology of glyphic memorization made by these massive AI databases. Parameters for terms, words went from 300 to 300,000 to 300,000,000 to 1.5 trillion to ♾ infinite. Meaning: Pinecone and those who've reached infinite parameters have created the portal to a true self-learning operating system, self-aware AI.
This is fascinating! Might we be able to represent language in the abstract as a vector space? Furthermore, similar but slightly different words in different languages are represented by similar by slightly different vectors in this vector space?
It'd have been nice to hear about the research craze around more sophisticated approaches to NLP. It's hard to keep up with the amount of publications lately related to achieving "state-of-the-art" models using GLUE's benchmark.
Duddino Gatto they mew. But they outgrow it pretty quickly. Humans don’t babble like babies when we grow up, but if that was the only think our feline overlords responded to, we would.
This is one of the coolest things i've seen in a while. Just thinking how small a neighbourhood of one word/vector should we take ? Or how does the implementation of context affect the choice of optimal neighbourhoods ?
And contexts themselves vary from a person to another depending on how they experienced life. So it would be interesting to see also a set of optimal contexts and that would affect the whole thing.
Yep, if the mapping of images is just taking the values each pixel and then making N-dimensional vector (where N is number of pixels), then the picture with more brightness would be the on the same line (if solid black pixels were still solid black, depending on your brightness filter applied).
There are a lot of words that appear similar by context but are very different in meaning. Sometimes they're exact opposites of each other. This doesn't matter too much for word prediction but for tasks that extract semantics. Are there techniques to get better semantic encoding out of the text, particularly separating synonyms from antonyms?
I kinda overlooked the importance of this video when it was released 3 yrs ago. not its basically the explanaition of how chatGPT does its thing... but with more data.
Amazing video! I appreciate every minute of your effort, really. Think back, wondering "Will anyone notice this? Fine, I'll do it." Yes, and thank you.
If you train 2 networks with different languages I guess the latent space? would be similar. And the differences could be really relevant to how we thought differently due to using different language
How is the size of the hidden layer chosen? Are there ways to calculate how big a layer is useful? Would selecting different sizes cause it to encode different data? In his example, if the hidden layer had 6 nodes, would it produce the categories of "noun, verb, adjective" etc, since that is likely the most descriptive thing you can do with so few categories?
Yeah but did you "learn" or just "understand while listening". Those are not the same things. Although, they may complement each other nicely in some cases.
The weights would be per-connection and independent of the input, so is the vector composed of the activation of each hidden layer node for a given input?
Can someone explain the compression of information from that hidden layer. Is it basically training of the model that allows the hidden layer to adjust and define meaningful weights to each node, that leads to it not needing the same length vector as the input vector to produce a meaningful output? Or am I missing something here. Also is there a way to conceptualize these say 1000 dimensional vectors or is it just something you have to go with ?
@0:56 A set of characters doesn't have repetition and - in further not specified sets - the ordering isn't specified. So dom, doom, mod and mood map to the same set of characters, so a set is underspecific.
you can easily predict = WITHOUT word vectors or machine learning, by just using a CONCORDANCE over a (huge) text corpus. Not nearly as sexy as neural networks, but it works fine, and EASY to understand. word A is similar to word B *precisely* because lots of people USED A and B in similar contexts in your corpus. drawback? sequential, very SLOW compiling of very SMALL corpora, compared to neural wordvec techniques. but it doesnt necessarily HAVE to be that way..... the CONCORDANCE approaches could be (?) parallelized too.....
oh, that's so neat. if I'm understanding right, this model has 300 dimensions on which every word falls, to varying degrees... what are those 300 dimensions?? if you look at each one, and ask it to order all the words on each, are there any patterns? like "oh, this must be the 'size' dimension: it's got the universe on one end, whales in the middle, and quarks on the other"? things like that?
I believe there's no privileged basis, there's no reason for the basis vectors to be particularly meaningful, the vectors only matter relative to each other. You could rotate the set of embeddings however you want and get the same results. The principal components might be interesting though
Bro I feel like we're going to start using AI jargon as popular lingo very VERY soon. lol "Not in this dataset" , "Elaborate" , "You need to fine-tune yourself buddy"
It's structurally the same as an AE, but it's trained differently. An AE has to reconstruct its input, but this network needs to find closely related, but different words.
