Scene tags that actually fire: a trained head on the embeddings you already stored
Our scene tags were zero-shot and timid — one in five landscapes came back tagged “people.” A linear classifier on the CLIP embeddings MediaFind had already cached roughly doubled coverage, for one extra matrix multiply and no re-indexing. Here’s the win — and the one class it made worse.
MediaFind indexes your audio and video library locally and lets you search it — by what’s said, who’s on screen, what brand appears, and where a shot takes place: kitchen, beach, office, a city street. That last one, scene tagging, was quietly the weakest thing we did. Here’s how we made it a lot better for almost no compute — and the three things that surprised us on the way.
The weak link
Our scene tags were zero-shot. We embed a handful of text prompts once with CLIP — “a photo of one or more people,” “a landscape or nature scene,” “a city street or buildings” — and for each frame we pick the prompt with the highest cosine similarity to that frame’s image embedding. No model trained, none downloaded. Keyless and on-device, which is the whole point.
The trouble is that CLIP is confidently similar to almost any plausible prompt. Left alone, every frame gets a tag and half are wrong. So we’d bolted on a negative-anchor gate: a label fires only if it also beats prompts like “an abstract graphic” by a margin. That suppresses the nonsense, but it’s a blunt band-aid — it throws away real signal to stay precise.
You could see it in the failures. A photo of a mountain came back tagged “people.” A sunset, “people.” On our own hand-labelled set, the zero-shot tagger correctly labelled only about one in five genuine landscapes. It wasn’t wrong so much as timid — the gate couldn’t tell signal from noise, so it kept quiet.
The insight: the expensive part was already done
The obvious fix is “train a scene classifier.” The obvious objection is “that’s a whole model, and we’re a keyless, on-device app.”
But we’d already paid the expensive bill. To do visual search at all, MediaFind runs every frame through CLIP once and stores the embedding. Logos, actions, colours, scenes — they all read from that one cached vector. A scene classifier doesn’t need to touch pixels again. It needs to look at a 512-dimensional vector already sitting in the index.
And a classifier over a fixed embedding is just a linear layer:
proba = sigmoid(embeddings @ W.T + b) # (n_frames, n_classes)That’s it — one matrix multiply against vectors we already have. No re-decoding video, no re-embedding, no second model at query time. Training is a logistic regression you can fit on a laptop in minutes; the shipped artifact is a weight matrix measured in kilobytes.
This isn’t a clever technique. Linear-probing a frozen CLIP is the canonical thing to do with CLIP — it’s in the original paper. That’s exactly why it was right: the cheapest, most boring option was sitting on top of infrastructure we’d already built.
What the numbers said
We pulled a few hundred labelled images per class from public datasets, embedded them, and fit one one-vs-rest logistic head per scene label. Then we A/B’d it against the live zero-shot tagger on a held-out split. Three things surprised us.
Here’s what the real set said:
| Metric | Zero-shot | Trained head |
|---|---|---|
| Coverage — correct tag on real scenes | 0.52 | 0.85 |
| Precision — right when it fires | 0.86 | 0.99 |
| False-positive rate — abstract frames wrongly tagged | 0.10 | 0.03 |
Coverage nearly doubled — and precision rose while false positives fell. Not a trade; strictly better. Per class, the lift landed exactly where the band-aid had hurt most:
| Scene class | Zero-shot recall | Head recall | Δ |
|---|---|---|---|
| nature | 0.19 | 1.00 | +0.81 |
| people | 0.25 | 1.00 | +0.75 |
| food | 0.36 | 0.92 | +0.56 |
| cityscape | 0.72 | 1.00 | +0.28 |
| animal | 0.94 | 0.94 | ±0.00 |
| vehicle | 0.67 | 0.22 | −0.45 |
And the honest one: one class got worse. vehicle recall fell from 0.67 to 0.22. We traced it — our vehicle training images were all cars on clean studio backgrounds, so the head learned “clean background” as much as “car,” and calibration had to crank vehicle’s threshold sky-high to stay precise, which killed its recall. That’s not a flaw in the method; it’s single-source training data doing what single-source training data does. The fix is diverse sources and smarter calibration, and it’s on the list. We mention it because a write-up that only shows the wins is a brochure.
Shipping it without changing anything
The feature is real and merged, but here’s the part we’re proudest of: turning it on changed nothing.
The head loads behind a seam. If no weights file is present — the default — the runtime never even resolves the classifier; it falls straight through to the existing zero-shot path. Same code, same output, byte for byte. The trained head activates only when someone ships weights and those weights match the runtime’s CLIP space. Absent, disabled, or mismatched, you get exactly today’s behaviour.
That let us land the whole mechanism — loader, integration, training and eval tooling — as an inert change, verified by proving the classifier runs zero times under the default config. The interesting model work and the risky production rollout became two separate decisions instead of one scary one.
What to take from it
- Look for the embeddings you’ve already computed. The most expensive part of a “new” feature is often already in your index. A linear head on a cached embedding is nearly free at inference and trivial to train.
- Pin your encoder identity. If a model produces vectors you store and reuse, record which model and refuse mismatches loudly. Silent geometry drift is a miserable bug.
- Distrust a benchmark until you’ve counted its rows. Ours looked like a coverage metric and was a five-example precision guard. We almost optimised against noise.
- Ship the mechanism dark. Land the plumbing so it’s provably inert by default, then flip the switch as a separate, reversible decision.
The headline — roughly double the scene-tagging coverage — came from the least glamorous option available: a logistic regression on vectors we’d already stored. Sometimes the model you need is already in the database.