I wonder how much can be done here with stochastic continuous evaluations in the spirit of MCMC or recent "Walk on Stars" style evaluations, where you don't have any discretization error at all, but trade that off with some noise...
I am a graduate student in Physics. This video is clear, easy to follow and highly informative. Many thanks for making this video public! This is very helpful for me
Thank you for a very informative video. I have a very limited mathematics background and was wondering if there are any good resources to better understand the differentiation in ODE. Please let me know if have such resources if you see my comment. Cheers!
Thank you guys! Very solid video, and good tempo. You present the material with a smile in a very user-friendly manner, that's a rare delicacy :) I wish new successes for your trio in the coming year. Separate thank you for the website and the code! I think I will try to apply DEQ to image denoising.
Hi Zico Kolter, great work! ....What about the inference time of DEQs w.r.t DNNs? Are they similar? ...Another question Do you recommend to use JAX instead PyTorch or Tensorflow2?
Hi, I'm just getting started with DILs/DEQs but from what I can tell, their inference time tends to be x2 slower when compared to DNNs. Still, depending on your application it might not be important at all; e.g. in my case we are interested in processing requests on the minute, while a feed-forward DNN takes milliseconds to do inference, so doubling the milliseconds is not going to be a problem. In fact, our hope is that solving the optimization problem directly via this method will save time overall (compared to DNN + optimization algorithm).
Thanks for the tutorial! I have a question about the representations created by DEQs, in normal Deep Networks depth means you can compose features and deeper layers are supposed to have higher level representations, does the same story apply for DEQs or is there a similar way to understand its computation?
This work is amazing! When I saw GPT-3 use 175 billion parameters to build a language model, just feel hopeless. It's more fair to compete state-of-the-art performance based on model complexity.
I learn a lot. Thank you very much. There are 2 questions about DEQ. 1. Why does equilibrium point z* matter? How is z* better representation than any intermediate representation z_t? 2. ALBERT is BERT but share the weight by all transformer layers. How DEQ save memory sounds like ALBERT computes the gradient of only last layer and update the "shared" weight. ALBERT actually computes all gradients of all layers and update the "shared" weight by average of gradients. Why does DEQ work even though it doesn't care of gradient of intermediate layer?
Thanks for the questions. For 1) this is mainly just an empirical issue, but in practice we do see that "deeper" networks (even in the weight-tied setting) do appear to work better, and thus the equilibrium point works best as the final representation (plus allowing efficient differentiation). 2) Yes, ALBERT would store all the intermediate activations, and compute gradients through the whole unrolled network. The idea of the DEQ model is that this is actually unnecessary, though, precisely via the implicit differentiation method we discuss in the tutorial.
Really cool. One question though? What is the fuss about neural ODEs? Honestly, I think I am missing something. They look just as taking a fireing rate model as an RNN... What is the difference?