Hi William, thank you very much for your videos! Its a pleasure to watch and learn from them. I have one question though: When you train your NER model and it 'learns' the relations between the words from your training data and the rest of the text, how does it know which ones it adds to the map connecting the various words? Not sure, if this is clear, so I will try with an example. In a previous video you trained a costume NER model with Harry Potter character names. I assume if I would then look at a visualisation similar to the one you display in this video I may see that spatially the words 'Harry Potter' and 'Gryffindor' are relatively close to each other as they are related in text. Yet, we did not tell the model to grab the word 'Gryffindor', so why would it choose that word over any other word to add to the map? Or does it actually just use all entities (depending on our n-gram definition, e.g. 1, 2, 3, etc.) it finds? Thanks in advance.
Hah! No problem. Harry Potter is one of my favorite books. Would you mind rephrasing your question? I'm not sure I understand. Do you me if we've removed them at this stage of the process? In this video? At all? Sorry for not understanding.
@@rahuldey6369 I cannot recall if I removed stop words for this demonstration. I don't believe I did. I was more focused on the concept of word vectors for this video and not necessarily cleaning the data appropriately. Excellent catch! And the vectors of stop words is certainly altering the way the data is being presented.
Great question. Word vectors do not help the model identify tokens (I think that's what you mean by start/text counterpart), rather they help the NER model generalize on what those tokens (either individual or Multi-word tokens) are in respect to the labels. For example, if I had an NER that could identify people and places, the model was certainly never trained on every personal name or place name in the world. In order for the model to generalize well, it learns context during training. Word Vectors allow for the model to learn context more easily.
