The right way to fuse vision and text on-device. We measured it.

“On-device” has become a marketing checkbox. It tells you where the computation happens, not whether the result is any good — and for a category built on trust, that gap matters. We wanted an honest answer to the obvious question: how much does combining vision and text actually buy you when everything runs locally? Not a cherry-picked demo. A number, on queries we didn't write, that anyone with the same public data can reproduce.

The short version: combining vision and text does win — but only with the right fusion. The method turns out to matter more than the models. Here's what we found, caveats and all — including the correction we made along the way.

The setup

We used Ego4D Natural Language Queries (NLQ) — a public benchmark of egocentric video with human-authored questions (“where did I put the watering can?”) and answer windows that were created independently of our system. That independence is the whole point: we can't quietly grade ourselves against a target we invented. We learned that one the hard way — an earlier internal experiment looked great on a benchmark we designed, then collapsed on real human queries once the ground truth wasn't rigged in our favor. Never again.

We built a cross-video pooled index over four Ego4D videos:

• 5,821 video frames sampled at 1 fps, embedded with CLIP ViT-B/32;
• 1,463 narration segments — the dataset's own provided text — embedded with MiniLM-L6.

Ego4D is head-mounted camera footage, so its NLQ questions are first-person “where did I leave it” memory queries — exactly the kind you'd ask your own recordings. A few real ones from our set:

“Where did I put my phone?” · “What did I put in the pot?” · “Where is the pipe metal?” · “What time did I eat the food?” · “Where did I put a cup?” · “Did I leave the microwave open?”

We ran 97 of these against several fully on-device methods. Two single channels:

• text — rank narrations by MiniLM similarity to the query;
• visual — rank CLIP frame embeddings against the CLIP text encoding of the query;

and two ways to fuse them: RRF (reciprocal-rank fusion — combine the two rankings, the standard first reach) and score fusion (sum each channel's confidence-normalized similarity scores). Plus an oracle that always routes to whichever channel is right — an unreachable ceiling, not a real method.

The protocol, precisely, so a skeptic can see exactly how the table is computed. Every method searches across all four videos — crucially, none is restricted to the query's own video (getting that wrong is exactly the bug we hit; more below). Each single channel ranks its own units (narration segments for text, frames for visual); the fusion methods rank the union of both. A query counts as a hit at rank k if a top-k moment falls inside the query's ground-truth answer window (±1s) in the correct video. The cross-video pool makes this harder than standard within-video NLQ, so these numbers are not comparable to the official NLQ leaderboard.

Everything ran locally on an Apple M4 Max laptop via Metal Performance Shaders (MPS). No cloud, no API keys.

The result

The fusion method matters more than the models

Start with the obvious approach and you'd conclude the opposite of our headline. Reciprocal-rank fusion (RRF) — rank each channel, then combine the ranks — is the standard first move, and on its own it doesn't clear the bar here: 0.289 R@10, a touch under text-only's 0.299, and it trails on R@5 and MRR too. (It edges text at R@1 — 8 hits vs 7 — but that's within the noise of 97 queries.) Simply combining ranks isn't enough.

Why? RRF looks only at the rank of each item, throwing away the similarity magnitude. So a barely-above-noise visual guess gets the same vote as a confident one — and since the visual channel is much weaker here (R@10 0.144 vs text's 0.299), its uncertain votes dilute strong text hits. Rank fusion asks each channel for its opinion but never asks how sure it is. That's the piece worth keeping.

What actually works: confidence-aware score fusion

The fix is to keep the confidence signal. Instead of fusing ranks, we normalize each channel's similarity scores (per query, z-scored to a common scale) and add them. This puts both channels on the same footing while preserving each channel's internal score gaps — so a channel with a clear standout casts a strong vote, while a flat, uncertain channel's votes stay small. Same two channels, same candidates — just a fusion that respects how peaked each channel's scores are.

That version beats the best single channel on every metric: R@10 0.330 (+10% over text-only's 0.299), R@1 0.124 (+72% over the best single channel's 0.072), MRR +17%. It has zero parameters tuned on the test set, and the win holds across normalization choices (R@10 0.320–0.330). The oracle ceiling — always route to the right channel — sits at 0.381, so there's still headroom: score fusion captures roughly a third of the gap between the best single channel and a perfect router.

Concrete examples — real queries from the run

The position each method gave the correct moment (#1 is the top result; “miss” = not in the top ten).

Score fusion holds on to a correct answer that rank fusion lets slip. When text alone already nails it, RRF's equal-weight visual vote can drag it down — sometimes out of the top ten — while score fusion keeps it at the top:

And score fusion genuinely adds signal — surfacing moments neither channel ranked highly on its own:

What we're not claiming

We're allergic to hype, so here's the honest boundary:

• This is proof scale — four videos, 97 queries, clustered within those videos — not a full leaderboard, and not a statistically-powered study. The margins are small (R@10 is 29 → 32 queries out of 97), so treat these as directional: the RRF-vs-score-fusion contrast and the R@1/MRR gains are the takeaways, not any single decimal. (The one thing we did stress-test is the score-fusion R@10 win, which holds across normalization schemes, 0.320–0.330.)
• The text channel is Ego4D's narrations, not our ASR, so this says nothing yet about our speech-to-text end to end.
• This is not a clean modality ablation. The channels differ in more than text-vs-vision — different encoders, different candidate counts. Read it as a comparison of retrieval methods, not a controlled causal test.
• We did not benchmark against a cloud model. The comparison is across on-device methods — which is the point — not a claim that local beats a frontier vision-language model.
• CLIP ViT-B/32 at 1 fps is a low-cost baseline, chosen on purpose. A stronger encoder, denser sampling, or a temporal model could move the visual and fusion rows either way.

Why publish this at all

Because “on-device” has become a marketing checkbox, and checkboxes don't tell you if the product works. The useful, non-obvious finding here: combining vision and text on-device really does beat the best single channel — but you get there with confidence-aware score fusion, not the rank fusion you'd reach for first. That's the kind of thing you only learn by measuring — and by being willing to publish the correction when the first measurement was wrong.

This is deliberately the outward-facing half of a pair. Internally we run a per-commit regression eval that guards search quality on our own curated sets on every push; this post is the complement — an external, public benchmark, on queries we didn't write, that a stranger can rerun. One keeps us from regressing; this one lets you check our homework — bug and all.