Here is the outline for the video, let me know which ones you think I missed: 0:00 - Introduction 0:21 - 1. Didn't overfit batch 2:45 - 2. Forgot toggle train/eval 4:47 - 3. Forgot .zero_grad() 6:15 - 4. Softmax when using CrossEntropy 8:09 - 5. Bias term with BatchNorm 9:54 - 6. Using view as permute 12:10 - 7. Incorrect Data Augmentation 14:19 - 8. Not Shuffling Data 15:28 - 9. Not Normalizing Data 17:28 - 10. Not Clipping Gradients 18:40 - Which ones did I miss?
I'm getting error : RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same. I have assigned the input and model to 'cuda'. could you throw some light on this.
This channel doesn't provide the basic tutorials which are there in the documentations and that's why it's very awesome. Thanks for your genuine content :D
Some of my favourites are breaking the computational graph (e.g. using numpy functions instead of pytorch ones) or backpropagating somewhere you shouldn't. Or getting dimensionalities wrong and getting screwed over by Numpy''s automatic broadcasting. Or in general not looking for existing Pytorch functions and reinventing the wheel over and over again.
Common mistakes for me: Getting confused with tensor dimensions (as a new guy you can spend plenty of time before harnessing the power of unsqueeze()) Forgetting .cuda() or .to(device) Getting confused with convnet dimensions after conv layer is applied Not attempting to balance or disbalance the dataset on purpose, which can be useful etc. Love your videos man, they've helped me alot.
My fun mistake - added a ReLU in the last layer (before CrossEntropyLoss) - the model trains poorly for a while, then just stops training (once all logits have been driven below zero).
I think you have an error in the check_accruracy function. You need to put the scores(since the are just the logits from a Linear layer) first in a softmax layer and then calculate the argmax .Am i missing something?
Real Computer Scientists already know the answer to life, the universe and everything. That's trivial so we just use start index as 0. Mathematicians are a bit behind so they use 1.
I don't think we need to shuffle the validation or test set right? Cuz there we will only be making the predictions and calculating our metrics like loss and accuracy, which are totally unaffected whether you shuffle or not. Plz do correct me if I'm wrong, thanks.
Great Vid! One more doubts. What's the exactly difference between torch.nn.Conv1d and torch.nn.functional.conv1d? Both seems to be present equally. That confusing me😅
For nn.modules you need to initialize them in the init function, for functional they are "stateless" and you need to manually set the weights. Basically functional has things without parameters/weights (and you would need to set weights manually). You can read more on the forum: discuss.pytorch.org/t/what-is-the-difference-between-torch-nn-and-torch-nn-functional/33597/6
Can you explain sir how to solve "CUDA out of memory" error in fastai package in pytorch. I am a beginner in fastai package and pytorch in general. Thanks for your great content sir.
9:23 I think it is the gradient of the two layers' biases that are equal. If so, isn't having a bias in the conv layer and another one in the batch layer equivalent to having bias only in one of them but multiplying its gradient by 2?
From my understanding if we're first running it through a bias (and let's say every node activation gets raised +1) then running it through BatchNorm is going to remove this regardless and therefore it was completely irrelevant of having the bias. So I guess it's not a big deal but it's just an irrelevant parameter. I follow your point that the gradients are equal for the two layers but I don't follow when you say multiplying the gradient by 2
@@AladdinPersson Oh, you are right! I was not aware of the fact while I wrote the comment that the bias of the convolutional layer will be removed first by the batchnorm layer and after running the BN will we just add the bias of the BN layer. For some reason I thought that we add the BN's bias right after we've added the conv's bias. In that case would be the gradients of the two bias terms be the same. There comes from the factor of 2. But I was completely wrong about how we do batchnorm, so I was completely wrong. Then, the difference of the loss after training with and without the conv's bias term could be because of numerical reasons, couldn't it?
Hi! Love your channel! I have a question, what if the data that you want to normalize is not an image but a vector (a sequence of numbers)? What do you think would be the best type of normalization? I've tought about max-min norm that also set the data into [0,1] range but would it be necessary to use normalize with respect the mean and std? Thanks!!!
Always normalize using mean and std, because if your inputs are always positive then the gradients of the first layer connected to your inputs will always be either positive or negative (depending on the sign of the upcoming gradients) which essentially mean the weights of that layer are all updated in the same direction (either all increase or decrease) and this makes training a bit tricky using zigzag paths
In Tensorflow we often divide only be 255 to normalize. Would that be possible in Pytorch as well? (Would probably save time so we do not have to figure out mean and std) Thanks
Just doing ToTensor() will divide by 255 so it gets in the range [0,1], but it's been shown to be better if you also do the additional step of obtaining zero mean and std 1 so it gets in the range [-1, 1]
Can someone explain to me how I should manage Dropout layers, considering I am using batch state - action - reward? I don't understand how just setting a mode.train() would work out. In my view, the Dropout layers would have to drop the same way when performing backpropagation, on the batch. Am I wrong? Is there something I am missing and how could I synchronize them if required and possible. Or do they just average out somehow?
You are right when you said the dropout (randomly) drops particular neurons in a layer based on the probability defined by an engineer. However, you do not perform backpropagation when validating (testing) your model. Specifically, model.eval(), which turns your model into testing mode, does not backpropagate; consequently, it does not to use dropout.
@@TeachAI-UZ there are training methods which REQUIRE a history in order to organize outputs and rewards and I would assume they need the exact form being used; for example q-learning with epsilon-greedy; if everything changes between the saved input-output-reward state and the moment you do the backpropagation, then I see no way that could work out
What you didn't understand is that by typing in the mean and the variance for your normalization error you introduced a bias and that's why the performance has risen. Read "Learning from data". Awesome video otherwise, thanks 👍
Great work, as always! I used this to normalize images. I want to know is this good? loader = torch.utils.data.DataLoader(train_set, batch_size=16, shuffle=True) def mean_and_std(loader): mean = 0. std = 0. nb_samples = 0. for data,_ in loader: batch_samples = data.size(0) data = data.view(batch_samples, data.size(1), -1) mean += data.mean(2).sum(0) std += data.std(2).sum(0) nb_samples += batch_samples mean /= nb_samples std /= nb_samples print(mean) print(std)
The way you're calculating mean seems good but I think with the standard deviation there's a bit of a mistake. Since standard deviation isn't a linear operation you cannot do std += (batch_std_here). Doing in this way you will not obtain the real standard deviation. I made a video on it to show how you would do it which you can check out. Although your way will probably work just fine even with a minor flaw with the std :)
Great thanks from Russia. Really love your videos. In a very short time a got PyTorch essentials with the help of yours. So many models have been understood and implemented with your help. Keep it going buddy!!!!
I'm not understanding how Normalizing data doesn't hurt accuracy With time series data, if I take in a set of numbers, and then normalize those numbers, don't I get a normalized output instead of an accurate output?
Always normalize inputs using mean and std, because if your inputs are always positive then the gradients of the first layer connected to your inputs will always be either positive or negative (depending on the sign of the upcoming gradients) which essentially mean the weights of that layer are all updated in the same direction (either all increase or decrease) and this makes training a bit tricky using zigzag paths
Can you please elaborate on why they would move in positive or negative directions? Imagine that your inputs are positive, you pass them through a Linear layer, then apply BatchNorm, then Linear again, then CrossEntropy. It is not obvious why the grads would be positive if chain rule would change signs for some direction derivatives.
Do we really have to normalize the data in initial data transformation, if we use BatchNorm2d layer in our model architecture, because both would perform an identical task.
Thank you for this super helpful video! Do you have to do transformations and normalize your data (if it is images), or can you just feed in the pixel array without transformation/normalization?
Nice video bro....😇😇 More pytorch videos....about Layers, activation func,optimizers, etc...... Dont know really where to use which layer and activation func... 1. How to find mean and std of RGB ? 2. Is it possible to use batchnorm1d in linear layer ?
