Jeremy came from some different world. Where people care. About looking under the hood and rewriting ReLU. About making sure nobody is left behind and very high quality education is free. thank you SO much, Jeremy, for being awesome!
Training a fully cnn model, weight scaling by glorotxavier init(1/√n), kaiming he init 1/√(2n), simple batch normalization, still our model is not having std dev of 1 and mean of 0, Soln: leaky relu( relu is incompatible as it threshold all value under 0 to 0 so the mean will never be 0), init methods that dont fiddling around weights what we have done so far these methods basicallly changes the act values (LSUV, batch normalization, layer norm) until they have 0 and 1 mean and std also on the same time they tweak the weights and biases as we have seen in lsuv the weights are divided and biases minused and then again act values mean and stds are checked again do it in a loop, finally improved the accuracy.
