L’un des visages les plus connus du monde de l’IA, Yann LeCun, va quitter son poste chez Meta. Un départ dû à des divergences de vues avec la direction.
Meta se précipite pour rattraper son retard sur les LLMs, quand LeCun est plus intéressé par la recherche à long terme pour enfin arriver à une IA.
https://x.com/iScienceLuvr/status/1988560605133812119
Tanishq Mathew Abraham, Ph.D. @iScienceLuvr 12 nov
A NEW PAPER FROM YANN LECUN: LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
This could be one of LeCun’s last papers at Meta (lol), but it’s a really interesting one I think.Quick summary: Yann LeCun’s big idea is JEPA, a self-supervised learning method. However, there are various failure modes of this approach, so training strong JEPA models is very brittle, unstable, and quite difficult. So overall JEPA has seen little adoption in practice.
This paper tries to directly address this, making specific design decisions that improve training stability.
The authors identify the isotropic Gaussian as the optimal distribution that JEPA models’ embeddings should follow and design the Sketched Isotropic Gaussian Regularization (SICReg) to constrain embeddings to reach that ideal distribution. This forms the LeJEPA framework, which can be implemented in ~50 lines of code.
On empirical tests, the authors demonstrate stability of training across hyperparameters, architectures, and datasets. A result particularly interesting to me however is that training a LeJEPA model from scratch directly on the downstream dataset outperforms finetuning a DINOv2/v3 model on the dataset!
https://x.com/teortaxesTex/status/1988792111093936269
Teortaxes! @teortaxesTex
LeCun, LeJEPA, Le Swan Song
I don’t think this is another joke, he might have proven something very general here, something that will influence our thinking about objectives well outside the JEPA franchise