This was the best explanation of HS that I saw! Very clearly explained. In my opinion, the most essential part of this was that even with HS we do have |V|-1 output units..but only log|V| units need to be computed while training since remaining units are "dont-cares" and we can compute loss based on the log|V| output units only. However, while testing, we would have certainly required to compute all |V| softmax probabilities to make any prediction, but we don't really care about testing/predicting since our aim is just to train the embedding.
Thank you Chris. well explained and just perfect for people like me who are interested in understanding the concepts and usage rather than academic maths behind it..
Thank you. Good explanation. Some questions: 1. At 2:40. Why are we interested in getting the sum as 1, which softmax provides? What's wrong with using existing output values, we already have the weight for 8 higher than others, so we have the answer. Why do we need the extra work at all? 2. At 9:49. What is this "word vector"? Is it still one-hot vector for the word from the dictionary or something else? How is this vector represented in this case? 3. At 15:00. That's fine, but if we trained for "cpupacabra", what would be with the weights when we train for the other words? Wouldn't it just blend or "blur" the coefficients making them closer to "white noise"?
Hi Chris, thank you for the video. So, if I understand correctly there are 2 things that are being learned here. 1) The word embedding, 2) the Output Matrix which contains the weights associated with the output layer?
Hello dear Chris, Hope all is well, Thanks for your lecture, that was fabulous. I have some tiny questions: - For negative sampling, it is said that negative samples will selected randomly. So, It means that we just need to update the params for those samples instead of all possible words in softmax? (so, in softmax we need to update the params for both correct and incorrect classes, true?) -How we can calculate the output matrix? How do we have it? -if we want to calculate the prob of all context words, we need to traverse over all the tree, right? Best Wishes.Mahdi
Hi Chris. Great Video. Could you explain the part how the binary tree is trained. I can't seem to understand for the input-context pair(chewpacabra-active) how do we select 0/1 at the root node or the subsequent nodes below.
I think you know the sequence of 0/1s for the context word. So for each node, you have a logistic regression model that takes as input the embedding for the input word and outputs probability of 0/1 for that node. So for the example you describe, we know the true label for root node should be "1", for 4th node is "0", and for 3rd node is "1".
The video is great! Is there any rules in why each blue node corresponds to the given row in the output matrix? Like why first blue node is row 6? How is it determined?
Hi Yuan, It's just a byproduct of the Huffman tree building algorithm. If I recall correctly, I think it does result in the rows being sorted relative to the tree depth (the frequency of the word). This isn't important to the implementation, though.
You can assign each blue node to any row in output-matrix. Order of assigning rows is unimportant since this is not like any RNN output(i.e. it isn't sequential output). Just like input units can be in any order in a vanilla neural net.
how to implement the hierarchical softmax for any model? Do frameworks like pytorch tensorflow have inbuilt implementation for them? If not, how can be build to work it on any model ?
Okay, I missed the part that answers it. So, output vector's particular row would correspond to one of those non-leaf nodes, and the size of the row would be equal to the number of units in the hidden layer? Thank you Chri,s!
