This was exactly what I needed, thank you! after learning Newton's method for finding the x-intercept, I was confused at first on how it was being used for minimization problems
Video makes sense up until the point where the "pictorial" representation of the 2nd order method comes in. That to me makes absolutely no sense whatsoever, the "pictorial" should not be the function itself but you rather the 1st derivative of the function and you apply Newton's method to that.
I think that the visualization makes sense if we think about approximating the function f(x) by its second order Taylor expansion around x_t. Taking the derivative of the second order Taylor expansion and setting it equal to zero leads us to the formula of the Newton's method for optimization. This operation is the same as minimizing the second order approximation of the function at x_t as depicted in the video.
So here is the thing ... My function y = (guess^2 - x ) Now i want to minimize y by approximating guess so i use first order with guess = guess - (guess^2 - x)/2guess .... which is xt+1 = xt - f(x)/f'(x) but if i do on more derivative then xt+1 = xt - f ' (x)/f " (x) which is guess = guess - (2guess)/2 = guess - guess = 0 What to do the function is finding square root by newtons method
Thanks for posting these videos. They are quite helpful. So, to ensure that we minimize and not maximize, is it sufficient to ensure that the newton step has the same sign (goes towards the same direction) as the gradient? Is it ok to just change the sign of the step if that's not the case? (my experiments seem to indicate its not, but what should be done then?)
Hii @Alex N ,as per my knowledge this methods are used for Machine learning, where gradient descent is a classical algorithm to find minimum of a function( not always zero), If you know basics about ML then you will be familiar with loss function , so we have to minimize that function, for that we need its derivative to be zero, for finding that we use gradient as direction where the change in function is maximum.Now we have the direction but the we dont have the magnitude , for that we use a learning rate as a constant which is what 1st order does.In 2nd order we would use the magnitude which gives us the magnitude for which the point where derivative of function is 0 can be reached in less iterations.Thus 3rd order will ultimately result in finding the minimum of dervative of the loss function , but we need to find minimum of the loss function so ,it will be useless. Hope this was helpul
