Harnessing the Universal Geometry of Embeddings (Jha et al.)

May 23, 2025

-Platonic Representation Hypothesis: all image models of sufficient size have the same latent representation

-additions:

1.universal latent structure of text representations exists

2.can be learned

3.harnessed to translate representations from one space to another without paired data/encoder

-method: cycle consistency + adversarial loss

1.translate vector from domain A to domain B

2.use adversarial loss to make the translated vector look like a "real" B vector

3.translate the vector from domain B back to domain A, use cycle consistency loss to ensure it returns close to the original

The Strong Platonic Representation Hypothesis:

-neural networks trained with the same objective and modality, but with different data and model architectures, converge to a universal latent space such that a translation between their respective representations can learned without any pairwise correspondence

Translation enables information extraction

-Solving unsupervised translation will allow us to use information extraction tools designed to operate on vectors produced by known encoders

-mappings for the same text from different encoders (vector spaces) should have the same latent space (underlying representation)