Embeddings have a ceiling: three changes we shipped after ICML 2026

The real lesson isn't “embeddings are bad.” It's that corpus size is not the thing that breaks them — query shape is. These failures are compositional queries: ones where relevance depends on a combination of criteria rather than one blob of topical similarity. “The bit where Priya talks about the Q3 budget” is that kind of query. So is “the drone shot over the harbour, not the one over the city.” Single-vector search is being asked to encode, in one point, every combination someone might ask for — and past a point it simply cannot.

What this meant for us

MediaFind was already partly hedged. Search here isn't one embedding index; it's sixteen channels fused together — transcript, on-screen text, visual, named entities, file details and more — and channels that agree corroborate each other into a higher rank. Keyword-exact and facet channels don't share the dense ceiling at all, which is precisely why they earn their place in the default mix. The paper is a strong argument for keeping them, not trimming them.

But reading it turned up three gaps. Here they are, each with what we measured.

Change 1 — put the cross-encoder where the ranking is actually decided

Look at what does survive that table. The architectures that hold up are the ones that don't compress everything into a single point ahead of time: keyword matching, which compares actual terms, and multi-vector models, which keep several. A cross-encoder takes that furthest — it scores the query and the document together, so it can notice the interactions a pair of independently-computed vectors has already thrown away. It's far too slow to run over a whole library, which is why you run it on a shortlist.

We already had one. It had been re-ranking the transcript channel's own candidates for a long time. But the list a user actually sees is the fused, cross-channel one — and nothing was re-scoring that. The single best-evidenced defense in the paper was installed one level below where the final ranking gets decided.

The detail that matters: it blends with the existing fusion score rather than replacing it. A result with no text to score — a purely visual match, say — would otherwise be silently punished for having nothing for the cross-encoder to read. It keeps its fused standing; only text-bearing results get moved by the new signal.

Measured, holding everything else fixed:

Fused multi-channel search	MRR
Before (fusion only)	0.508
After (fusion + cross-encoder re-scoring)	0.532

About +4.7% relative. Stated honestly: that's on our 62-query fusion set, which is small enough that we'd want a wider set before calling it a headline number — the same caution we apply to every metric we publish. The direction is consistent with the paper and it cost no new dependency: the model was already in the app.

Change 2 — stop indexing the same frame sixty times

The video-understanding work at the conference converged on one idea: video's redundancy is overwhelmingly in the time axis, and you should collapse it before spending real compute. Systems like LongVU discard roughly half their frames on similarity alone before any heavy encoding, with no measured quality cost. VideoNSA, taking a sparse-attention route, runs on 3.6% of the attention budget at 128K tokens and still beats full dense attention.

MediaFind samples one frame every two seconds and embedded every single one. For a two-minute talking-head clip that's sixty near-identical vectors of the same person in the same chair. Wasteful on disk, and worse, redundant frames crowd each other in results.

There's also a sharper reason, and it comes straight back to the ceiling. Our visual embedding is CLIP ViT-B/32 — 512 dimensions, whose best-case ceiling is around 500,000 items. Frames are not files: a few hundred hours of video at one frame per two seconds produces millions of vectors. At frame granularity, a large library genuinely approaches the limit the paper describes. Thinning redundant frames isn't only a storage win; it keeps the index in the regime where the embedding can still do its job.

So frames are now compared against the last one we kept, and dropped if they're near-identical. The nice part is that it's almost free — the embeddings needed to make the decision have already been computed a moment earlier for exactly this purpose.

Change 3 — four times smaller, at no measurable cost

Every vector we store was a 32-bit float. A single CLIP frame vector is 2,048 bytes, and as established, big libraries have millions of them.

The compression work presented this year has a clear finding and a clear warning. The finding: retrieval tolerates aggressive quantization far better than intuition suggests. The warning is sharper — naive fixed-point integer formats fail on normalized embeddings, because the values cluster in a way that a fixed scale represents badly. You need the quantization to be fitted to the actual distribution of your vectors.

So each vector is now stored, optionally, as 8-bit integers with a per-vector scale factor: 2,048 bytes becomes 512. We measured whether that costs anything:

Visual search, identical queries	MRR	Bytes per vector
float32 (unchanged)	0.963	2,048
int8 quantized	0.963	512

Identical retrieval quality at a quarter of the size. This one ships switched off by default — a change to how data is written to disk deserves more evidence than one benchmark before it becomes the default for existing libraries, and the storage layer was built to let both formats coexist so the switch can be flipped without a migration.

Every number here was gated, not asserted

Each of these changes went through the same evaluation harness that scores search quality on every commit — measured before, measured after, and compared against a pinned baseline that fails the build on a regression. The harness also verifies which backend produced each result, so a number can't quietly come from a weaker stand-in encoder than the one it claims.

That's why the figures above are stated as deltas against a known bar rather than as absolutes: 0.508 to 0.532 on the same fixed query set, 0.963 to 0.963 across a storage format change. Improvements that don't survive that comparison don't ship as improvements.

What we'd tell another team

If you're building retrieval on embeddings, the actionable version of all this is short:

• Don't rely on a single dense vector alone. Keep a keyword/exact channel in the mix; it doesn't share the ceiling, and on compositional queries it can be the only thing that works.
• Put a cross-encoder on the final shortlist — the actual list users see, not an intermediate one. It's the best-evidenced fix available and it's cheap when bounded to a shortlist.
• Watch your granularity. The ceiling applies to whatever you index. Per-frame or per-chunk indexing can push a modest-looking library into millions of vectors.
• Quantize, but fit it to your data. Free storage is real; the naive integer version quietly isn't.

None of this needed a bigger model, a cloud service, or an API key — which is rather the point. The research that helped most this year wasn't about scale. It was about knowing precisely where a familiar tool stops working, and building around the edge.