Local models on your laptop

Running an AI model on your own hardware: no API key, no internet, no logging, no monthly bill. Smaller models, sure — but also full privacy and zero latency from your machine.

What local models actually do

A local model runs entirely on your own machine — your laptop, your phone, your server. The weights are downloaded once, and from then on every prompt is computed locally. No API key, no internet round-trip, no provider logging your conversations.

You give up two things in exchange: top-tier quality (frontier models are still bigger and smarter) and convenience (you have to manage hardware). You gain privacy, zero per-call cost, sub-second latency from your own machine, and the freedom to use AI in places where the cloud won't go — air-gapped networks, regulated industries, on a plane.

That's the whole concept. Below: which tools to use, the sizing math, and the quantisation trick that makes 70B-class models fit on a laptop.

When you'd reach for it

Pick a local model when at least one of these matters more than absolute quality:

• Privacy — journals, therapy notes, medical records, legal docs, anything regulated.
• Cost at scale — a fixed-cost server beats per-call billing for high-volume internal tools.
• Offline — planes, ships, field work, secure facilities.
• Latency — no network hop, every token is local.
• Air-gapped enterprise — defence, finance, healthcare where data can't leave the building.

You wouldn't reach for a local model for the hardest reasoning, the longest context windows, or the latest multimodal capabilities — frontier APIs still own those.

How it's actually built

To run a model locally you need three things:

1. The weights — a file (or set of files) typically 1–80 GB. Downloaded once, stored on disk.
2. A runtime — software that loads the weights and runs inference. Examples: llama.cpp, Ollama, LM Studio, vLLM.
3. Hardware that fits — RAM and (ideally) a GPU big enough to load the model.

A rough sizing guide:

Quantisation — compressing the weights to lower precision (fp16 → int8 → int4) — typically halves or quarters memory needs at small quality cost. A "70B Q4" model fits in ~40 GB instead of ~140 GB. This is why open-source models suddenly became consumer-friendly.

The major tools

Ollama

Easiest entry point. One install, simple commands. ollama run llama3:8b downloads and runs the model. Built-in chat REPL, plus an API your own code can call. Perfect for quick experimentation.

LM Studio

GUI for the same use case. Browse a catalogue of models, click to download, chat in a friendly interface. No terminal required. Most accessible option for non-technical users.

llama.cpp

The plumbing under both of the above. C++ implementation that runs models efficiently on CPU, Apple Silicon, and various GPUs. Powerful, with a CLI and an OpenAI-compatible HTTP server. The serious choice when you want control.

vLLM / TGI / SGLang

Production-grade serving frameworks. High throughput, batching, KV-cache reuse. What you'd run if you were serving local AI to many users on a server. Overkill for laptop use.

llamafile

Single executable file containing a quantised model + the runtime. Download one file, run it. Works on Mac, Linux, Windows with no install.

Where it bites in real life

What local models are good and bad at

Genuinely strong

• Privacy — nothing leaves your device.
• Latency — no network round-trip; near-instant TTFT.
• Cost — zero per-call cost after the one-time hardware investment.
• Customisation — full fine-tuning, LoRA, custom system prompts, weird experiments.
• Offline use — planes, secure environments, anywhere the internet is bad.
• Simple tasks — summarising, classifying, drafting, basic Q&A.

Genuinely weak

• Hardest reasoning — frontier models still pull ahead on math, code, and complex multi-step tasks.
• Long context — long-context support varies by local model; usually shorter than frontier APIs.
• Recency — training cut-off baked in; no built-in web access (unless you build it).
• Multimodal — fewer good vision/audio options at small sizes.

Try it yourself (~15 minutes)

The fastest path:

1. Install Ollama (Mac/Linux/Windows): ollama.com/download.
2. Open a terminal and run: ollama run llama3.2:3b (small, fits anywhere).
3. Chat at the prompt. Try the same questions you'd ask ChatGPT. Notice quality differences.
4. Try ollama run qwen2.5:7b if you have ~8 GB free RAM. Compare.
5. (Optional) Disable your wifi and confirm it still works.

You're now running an LLM with no internet, no API key, no logging, on your own hardware. For the right tasks (privacy-sensitive, offline, simple), this is a real option.

Further reading

• Ollama — the easiest way to run local models.
• LM Studio — GUI alternative.
• llama.cpp — the engine under most local AI tools.
• Hugging Face — Open LLM Leaderboard — independent benchmarks of open models.
• llamafile — single-file portable LLMs from Mozilla.
• vLLM — production-grade serving stack for self-hosters.