Wow! Insane and clear explanation. I tried presenting a weekly research presentation for a course a couple years ago and almost had no clue what this method was about. This would have saved my life back then lol Much better understanding now, thanks for making this!
Rewatching the vid, I thought that the method 3 at 5:30 seems familiar and I was right. Method 3 actually is the David-Fletcher-Powell (DFP) method for unconstrained optimization. It's the earliest quasi-newton method for approximating the hessian which is symmetric so the update should be symmetric. I think there may be a typo because in Wikipedia, the middle and last terms in the inverse Jacobian update are negative and positive respectively while here in the video, it's the opposite. I've been dabbling with optimization lately and I discovered that both Broyden's good and bad methods perform very badly in optimization problems. In the Rosenbrock function, the bad method simply blows up with nan results while the good method takes hundreds of iterations just to converge at the minimum at (1,1).
I probably did make a typo there and nice catch! I did not notice this relationship at all even after looking at DFP. Optimization dabbling sounds like a lot of fun. Always a good time when you learn new methods. A new video has been in the works for a long while but real life keeps getting in the way. Hoping I can get a lot done in the next few weeks.
Great video! I've read Broyden's original paper and it says "The functions that require zeroing are real functions of real variables". Are you aware of any generalisations to complex-valued functions? Thanks a lot!
hi, first of all thank you, I have a question I am using broyden 's method for solving indirect shooting method(optimal control), which is basically based on ode's , but using this method is actually diverging rather than converging.....
The normal Broyden method doesn't guarantee convergence. He does make the case for a variation in his original paper that can induce convergence but it isn't generally used and I didn't discuss it in the video. There are other methods that I have lessons for on this channel that do guarantee convergence like the Global Newton Method.
You find the difference in the step size, then compute the L-2 norm (square root of the sum of squares), and square it. Broyden wrote it as the step size vector transposed multiplied against the step size vector. Same result.
