What a great deconstruction of the paper. I understood the general idea much better than when I watched the presentation of one of the authors at ICML.
I think the covariance term prevents the network to encode similar information in "different" dimensions in the embedding space, not in the "same" dimensions.
Thank you for the correction. Your suggestion sounds more clear. I probably used the word "same" to refer to the single embedding space, but it should have been "similar information across different dimensions in the embedding space.
