What is a local LLM?

How it works

A language model is a large file of learned weights plus software that runs them. To run locally, you download the weights once and a runtime loads them into memory, ideally on the GPU, and generates text one token (a word piece) at a time. After that, no internet connection is needed.

The limiting factor is memory. Models are described by their parameter count, and each parameter takes space. Quantization stores weights with fewer bits, commonly 4 to 8 instead of 16, which shrinks a model several times over with a modest quality cost. As a rough guide, a 7 to 8 billion parameter model at 4-bit needs around 4 to 5 GB of memory, while small models of 1 to 3 billion parameters fit comfortably on most recent laptops. Speed is measured in tokens per second and depends mainly on memory bandwidth and the GPU.

An example

A lawyer wants to ask questions of 40 deposition transcripts but can't send them to an outside service. A local LLM paired with local search can find the relevant passages and answer “What did the witness say about the March meeting?” with the lines it used, all on a laptop with Wi-Fi off. The same pattern, search first and then generate, is called retrieval-augmented generation.

The trade-off shows up in the answers. A small local model may phrase things more plainly or miss a subtle point that a very large cloud model would catch. For a question whose answer is sitting in a few retrieved passages, that gap is usually small; for open-ended writing or reasoning, it is larger.

Trade-offs

• Privacy and control: prompts and documents never leave your machine, and no account or API key is needed.
• Cost: no per-token billing; you pay once in disk space and download time.
• Offline: it keeps working on a plane or in a secure room.
• Capability: small local models reason less well and know less than the largest cloud models, and they can state wrong things confidently. Grounding answers in your own documents and showing sources helps a lot.
• Hardware: larger models need more RAM and a capable GPU, and a heavy model can slow down other work while it runs.

In MediaFind

MediaFind's Ask assistant and summaries run on an on-device language model. A small one downloads the first time you use Ask, and larger ones are a one-time download too, so you can pick one that fits your machine. There are no API keys and no per-token billing, and no prompt leaves your computer. Answers cite the moments or messages they used, so you can check them. More in running an LLM locally, choosing a model and on-device AI.

Frequently asked questions

Can a local LLM run on a laptop?

Yes. Small and quantized models run on most recent laptops, and Apple Silicon Macs handle mid-sized models well because the GPU shares system memory.

Is a local LLM as good as ChatGPT?

Generally not at open-ended reasoning or general knowledge. For answering questions about your own documents with the sources shown, a smaller local model is often good enough.

Does a local LLM need the internet?

Only to download the model the first time. After that it runs fully offline.