About this video
A text model that never saw a single picture agrees with a vision model more than a 72-billion-parameter language model does, and it is 660 times smaller. A new paper by Randall Balestriero and Yann LeCun says the lever isn't size: it's what a model is taught to treat as the same thing.
Which matters more for AI: scale, or what you train on?
We read "Proving and Improving the Platonic Representation Hypothesis with JEPAs" (Brown University, NYU, AMI Labs) page by page: what a model's map (embedding space) is and how two maps are compared (mutual nearest neighbours); the Platonic Representation Hypothesis (Huh et al. 2024); JEPAs and positive pairs; the Platonic Representation Theorem (good views make separately trained models learn the same map, up to one rotation) and the bound when pairs differ; bad views, nuisance, and why next-token prediction has the form of a bad-view JEPA; the Flickr tests (4.4% true caption first, 44x chance; the scene explains 80% vs 65% vs 61% of the map; trimming is the knob); web scale (42x fewer samples; a 109M text JEPA at 18.4% vs Qwen2.5-72B 15.4% and Llama-3-70B 15.1%) and the honest catches; a scaling law that predicts runs before training; a captioner whose only paired part is one rotation (CIDEr 0.84 on COCO); alignment with no pairs at all (64% vs 56% top-5 on a 100-class ImageNet test); what the paper does not show; and four things to do if you build with AI.
Chapters
0:00 Two strangers, one world
0:54 Every model draws a map
2:13 Idea 1: Good views give one shared map
3:26 Idea 2: Next-word prediction is a bad view
4:44 Idea 3: The test on Flickr
6:21 Idea 4: A small model beats 72 billion parameters
8:22 Idea 5: One rotation becomes a captioner
10:19 What this paper does not show
11:33 What to do about it
12:44 Recap
Paper: Proving and Improving the Platonic Representation Hypothesis with JEPAs. Randall Balestriero (Brown University, AMI Labs), Yann LeCun (New York University, AMI Labs). arXiv:2610.39302, October 2026 (preprint, not yet peer reviewed). https://arxiv.org/abs/2610.39302
Figures, tables and excerpts from the paper are shown under its CC BY 4.0 licence (https://creativecommons.org/licenses/by/4.0/) with credit; the highlights, redrawn bars, animations and characters are ours.
Also mentioned: Huh, Cheung, Wang, Isola, "Position: The Platonic Representation Hypothesis", ICML 2024.
Music: Voice Over Under by Kevin MacLeod, CC BY 3.0, via Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Voice_Over_Under_(ISRC_USUAN1600001).mp3)
Pranav Innovates: AI research, explained. Subscribe to stay up to date.
YouTube: https://www.youtube.com/@PranavInnovates
Instagram: https://www.instagram.com/pranavinnovates
What the video says
A text model that has never seen a single picture sees the world more like a vision model does than a seventy two billion parameter language model. And it's six hundred and sixty times smaller. Picture two people in Plato's cave. One only ever sees photos. The other only ever reads descriptions. They never meet, and never compare notes. Do they end up with the same picture of the world? For AI models, the answer seemed to be yes, as they got bigger. But nobody could prove it, or say how to get there without just scaling up. Now Randall Balestriero and Yann LeCun say they've turned that hunch into a theorem. And the theorem says the lever isn't size. It's what the model is taught to treat as the same thing. Stay to the end, where one simple rotation turns two models that never met into an image captioner. The paper is called Proving and Improving the Platonic Representation Hypothesis with JEPAs. Let's read it, page by page. First, what does it mean for a model to see the world? A model turns each input into a list of numbers, a point in a big space. Similar things land close together. Think of it as the model's map. An image model has a map of photos. A language model has a map of sentences. They're trained separately, on different data. So how do you compare two maps? Take a photo and its caption. Find the photo's ten nearest neighbours on the image map, and the caption's ten nearest on the text map. If the maps agree, the neighbours match. That overlap is the score used here. Pure chance gives about one percent. In twenty twenty four, Minyoung Huh and colleagues measured this across many models. As language models got bigger, they agreed more with vision models. They called it the Platonic Representation Hypothesis: different models drifting toward one shared picture of reality. But it stayed an observation. No proof of when it holds, and no recipe. The only lever was, make the model bigger. This paper uses a different kind of model: a JEPA, a joint embedding predictive architecture, the family LeCun has long argued for. A JEPA learns from pairs it's told are the same thing, called positive pairs. It pulls each pair to the same spot on its map, with one rule that stops everything collapsing into a single point. Five ideas follow. Idea one. The theorem. Say an image JEPA learns from two different photos of the same scene. And a text JEPA learns from two different captions of that scene. The paper calls these good views: two separate observations of one moment in the world. The Platonic Representation Theorem says: if the models' positive pairs connect the same scenes, they all learn the same map. Not a similar map. The same one, up to a single rotation. Turn one map, and it lands on the other. Why? Because their training problems are really one problem, written in different coordinates. Pixels and words are just two ways of describing the same scenes. Solve the same problem, and you get the same answer. It holds in an idealised world: a fixed set of scenes, each description pinning down its scene, and training that reaches its best solution. When the pairs don't match exactly, a second theorem limits how far apart the maps can drift. They checked that limit on three hundred random worlds. The error stayed under it every time. For builders: if two models agree up to a rotation, one small matrix connects them. In theory, you only need as many matching pairs as the map has dimensions to pin it down. Idea two. What goes wrong with bad views. A bad view is two damaged copies of one single observation. Two crops of one photo. Two masked versions of one sentence. A lot of self-supervised training works exactly like this. The problem: two copies of one caption share more than the scene. They share its exact wording. The paper calls that nuisance: whatever is particular to one description, like the words a writer happened to choose, or the camera angle of a photo. A bad-view model learns to keep that nuisance. And every direction of its map spent on nuisance is a direction no rotation can line up with another kind of model. Good views share only the scene, so they spend nothing on it. Now the twist. Today's chatbots are trained by predicting the next word. Is that a good view, or a bad one? What do you think? Bad. The paper shows next-word prediction has the form of a JEPA. Its pairs are a piece of a document and what comes next in that same document. Same author, same style, same nuisance. That's their explanation for why language models line up with vision only as they get huge. They get there the slow way, through scale. The authors call this a prediction to test, not a proof. So they tested it. Idea three. Does it actually happen? First test: Flickr. Twenty nine thousand photos, five human captions each. One image model, trained from scratch, and three text models of the same size, trained on the same captions. The only difference was the pairs: good views, bad views, or next-word prediction. No model ever saw the other side. Then they fitted one rotation and asked: for each test photo, is its true caption the closest of a thousand? With good views, it came first four point four percent of the time. That's forty four times chance. Bad views and next-word prediction stayed well below, almost on top of each other. Swap in a strong pretrained vision model, DINO version two, and the good-view text model found the right caption first for about one photo in four. That's roughly twice the bad views, next-word prediction, and a small pretrained language model. Could good views just be getting extra information, knowing which captions belong to one photo? They checked. Next-word prediction trained on two captions of the same photo barely moved. Same information, almost none of the gain. It's the way of learning, not the data. They also measured the nuisance directly. With good views, the scene explained about eighty percent of the text model's map. With bad views, sixty five percent. Next-word prediction, sixty one. The order the theory predicts. Which setting matters? They changed one thing at a time. Cutting each caption down to a random piece mattered most. Without it, alignment dropped by a quarter. Almost everything else barely moved, and gradient clipping prevented most training collapses. Idea four. If good views are the lever, how far can they go against scale? They moved to web data: the texts of up to nine hundred and forty million web images. Each image has its alt text and two captions written by an AI captioner. Those three texts are its good views. Same corpus, same model size. Next-word prediction needed nine hundred million training samples to reach a certain agreement with the vision model. Good views got there with twenty one point six million. Forty two times fewer. Then the headline test. A good-view model of a hundred and nine million parameters, against big language models, on the original Platonic paper's own benchmark. Who agrees more with the vision model? Take a guess. The small one. Here are the scores. Eighteen point four percent of neighbours shared. Qwen two point five, at seventy two billion parameters, fifteen point four. Llama three, seventy billion, fifteen point one. And next-word prediction on the very same texts, eleven point one. The small model has six hundred and sixty times fewer parameters, and read over a hundred times less text. Two honest catches. First, the AI captioner that wrote most of those texts sees images through CLIP, a model trained on image and text pairs. So some picture knowledge was already in the words. In a smaller run without that captioner, good views reached only a little over half the score. Second, one of four training runs came out lower. Averaged over all four, it's still above every language model they tested. They even found a scaling law for it. Fitted on forty five runs, it predicted the caption retrieval of eighty five new runs before they were trained, about as accurately as rerunning a run with a different random seed. For builders: the biggest language model is not automatically the best bridge to images. A small encoder trained on the right pairs can beat it, for a fraction of the compute. Idea five. If two maps match up to a rotation, you can plug them together. They trained a caption writer on text alone. It learns to turn points on the text model's map back into captions, and it never sees an image. Then, for a photo: embed it with a frozen vision model, apply the rotation, and decode. The rotation is the only part ever fitted on photos with their captions. When the two maps have different sizes, it's a close cousin of a rotation, one that also drops the extra dimensions, a case the theorem doesn't cover. Here's what it writes. A little kid eating a chocolate donut. The human caption: a little girl is eating a chocolate doughnut. Those are selected examples. On five thousand COCO test photos, its best version scored zero point eight four on CIDEr, a standard caption score, here on a zero to one scale. Captioners built on CLIP, trained on four hundred million image and text pairs, report between zero point three five and zero point nine two. Why a rotation, and not a more flexible map? They tried. A free linear map fitted the Flickr pairs a little better, but lost a fifth of its score on COCO's new photos. A small neural network lost on both. A rotation can only turn one map onto the other, so it carries over. Then the boldest test: no pairs at all. An existing method found the rotation from the shape of two clouds alone, images from one half of COCO and captions from the other. On a hundred-class ImageNet test, this paper's small model then named the object within its top five guesses sixty four percent of the time. The best any language model managed in that benchmark, up to eight billion parameters, was fifty six. Next-word prediction on the same texts failed in half its runs. For builders: to join two separately trained encoders, try a rotation first, fitted on pairs you trust. It's tiny, it's fast, and it's the map least likely to overfit. Now, what this paper does not show. The theorems live in an idealised world: a finite set of scenes, every description fully pins down its scene, and training reaches its best solution. Real captions only describe part of a photo, and no experiment meets the first theorem's conditions exactly. Good views need someone to know which texts describe the same scene. The authors call that the main caveat. At web scale, those groups came from an AI captioner that sees through CLIP, and the comparisons with language models rely on those texts. The comparisons between kinds of views, on identical texts, don't. The image side mostly used ordinary crops, which are bad views by the paper's own definition. Settings were tuned only for the good-view model. The scaling law misses further out. And this measures agreement with vision models, not how well a model writes, reasons or codes. It's a brand new preprint, not yet peer reviewed. The authors' conclusion: a provable condition for shared representations, and evidence that next-word prediction, the objective behind today's largest models, is an inefficient way to get there. In their words, positive views, and not scale alone, are the lever. So what do you do if you build with AI? Four things from the paper, then my take. One. Choose your positive pairs on purpose. Two different descriptions of the same thing teach a model the thing. Two damaged copies of one description teach it the wording. Two. Look for natural good views in your own data. My examples, not the paper's: several reviews of one product, several photos of one listing, two support tickets about the same bug. Three. To connect two existing encoders, fit a rotation on trusted pairs before you train anything bigger. In this paper, it carried over to new data better than freer maps. Four. Don't assume the biggest language model gives the best embeddings for search across images and text. Measure agreement on your own data. A small encoder trained on good pairs may win. And my take. For years, Yann LeCun has argued that predicting the next word isn't the road to understanding the world. This paper is concrete evidence for that, but on one narrow measure: agreement with vision models. Chatbots still write and reason far better than a small encoder. My reading is simpler: what a model compares matters as much as how big it is. Let's recap. The Platonic hypothesis said models drift toward one picture of the world as they scale. This paper proves when: if the pairs are two real observations of the same scene, separately trained models learn the same map, up to one rotation. Next-word prediction is a bad view. On the same texts, good views needed forty two times fewer samples, and a small model beat language models up to seventy two billion parameters. And one rotation turned two strangers into a captioner. Back to the cave. Two people who never meet can end up with the same picture of the world. Not because they look harder. Because they compare the right things. Remember the text model that never saw a picture? It agreed with a vision model more than a language model six hundred and sixty times its size. Not because it was bigger. Because it learned from two descriptions of the same moment, not one sentence and its next word.