What is near-duplicate detection?

Exact vs near duplicates

An exact duplicate is the same file, byte for byte. Tools find these by computing a checksum (a hash such as SHA-256) of each file and grouping matching hashes. It is fast and has essentially no false positives.

A near duplicate has different bytes but the same content: a photo saved as both HEIC and JPG, a video trimmed by a second, a screenshot cropped slightly, or a clip re-uploaded at lower quality. Checksums can't see these, because changing a single pixel changes the whole hash.

How near-duplicate detection works

Instead of hashing the bytes, the software describes what the content looks like and compares those descriptions:

• Perceptual hashes shrink an image to a tiny grayscale version and encode its pattern of light and dark. Similar images produce hashes that differ in only a few bits.
• Image embeddings from a vision model capture content more robustly, so a crop or color change still lands nearby.
• For video, frames are sampled along the timeline and compared as sequences, along with duration. Audio fingerprints can confirm a match.

A concrete example

A family photo library has 40,000 images after years of phone backups. An exact-duplicate scan finds 3,000 identical copies. A near-duplicate scan then finds groups like: the original HEIC, a JPG exported for sharing, and a smaller copy saved from a messaging app. It also groups the eight burst shots of a birthday cake, where you probably only want the sharpest one.

Near-duplicate detection is more expensive than checksum matching because every file has to be decoded and analyzed, and comparisons have to be made between many pairs. Tools speed this up by indexing the fingerprints so they only compare files that are already roughly similar, rather than every file against every other file.

Pitfalls and how to delete safely

Near-duplicate detection involves judgment, so it can be wrong in both directions. Two genuinely different photos of the same scene may be flagged as duplicates, and a heavily edited version may be missed. A threshold setting controls how similar is “similar enough.” Before deleting anything:

• Review each group rather than bulk-deleting.
• Keep the highest-resolution or original file, not the smaller copy.
• Move candidates somewhere reversible first instead of deleting them outright.
• Keep a backup of anything irreplaceable.

In MediaFind

MediaFind finds exact duplicates for free. With Pro, smart cleanup adds near-duplicates, similar videos, and blurry or low-quality files, with keeper suggestions for each group. Every candidate goes to a reversible quarantine before anything is deleted, and the analysis runs on your computer. See how to find duplicate videos or the free MediaClean app.

Frequently asked questions

What is the difference between a duplicate and a near duplicate?

A duplicate is an identical file. A near duplicate shows the same content but differs in the data, for example because it was resized, cropped, re-encoded or saved in another format.

Can near-duplicate detection delete the wrong photo?

It can flag photos that are similar but not the same. Review each group, keep the best original, and use a tool that lets you undo.

Does near-duplicate detection work on videos?

Yes. Tools sample frames from each video and compare them as sequences, so re-encoded or slightly trimmed copies can still be matched.