Hi, I'm just getting started with DILs/DEQs but from what I can tell, their inference time tends to be x2 slower when compared to DNNs. Still, depending on your application it might not be important at all; e.g. in my case we are interested in processing requests on the minute, while a feed-forward DNN takes milliseconds to do inference, so doubling the milliseconds is not going to be a problem. In fact, our hope is that solving the optimization problem directly via this method will save time overall (compared to DNN + optimization algorithm).

Thanks for the questions. For 1) this is mainly just an empirical issue, but in practice we do see that "deeper" networks (even in the weight-tied setting) do appear to work better, and thus the equilibrium point works best as the final representation (plus allowing efficient differentiation). 2) Yes, ALBERT would store all the intermediate activations, and compute gradients through the whole unrolled network. The idea of the DEQ model is that this is actually unnecessary, though, precisely via the implicit differentiation method we discuss in the tutorial.

Thanks for pointing this out! We've re-uploaded them to properly sync. They should work correctly now.