The local vs cloud AI decision comes down to one tradeoff: a local model running on your own machine keeps every prompt on-box and costs nothing per token, while a cloud frontier model like Claude is far more capable but bills per token and sends your data off your hardware. I run both daily, and after enough real work the answer stopped being "pick one." The right answer for most builds is hybrid: route the easy, sensitive, high-volume tasks to a local open-weight model, and send the genuinely hard reasoning to the cloud. The rest of this post is how I actually split that, with numbers.
What does "local" actually mean here?
Local means open-weight models you download and run yourself, usually through Ollama or llama.cpp. The model file lives on your disk, inference runs on your GPU (or CPU, slowly), and nothing leaves the box. There is no API key, no rate limit, no bill that scales with usage. Getting started is genuinely a one-liner.
# Install from https://ollama.com, then:
ollama pull llama3.1:8b # ~4.7 GB download
ollama run llama3.1:8b "Summarize this changelog in 3 bullets"
# Or hit Ollama's native HTTP API for app integration:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.1:8b",
"prompt": "Classify this support ticket: billing, bug, or feature",
"stream": false
}'
# (Ollama also exposes an OpenAI-compatible endpoint at /v1/chat/completions
# if you want to reuse an existing OpenAI client.)That 8B model fits comfortably on a consumer GPU with 8-12 GB of VRAM. A 70B model needs a 24 GB card (or more) and starts to approach "useful for real work" territory, but it is still nowhere near a frontier model. That gap is the whole story.
How much privacy do you actually gain?
This is the cleanest argument for local. When the model runs on your hardware, the prompt and the data in it never cross your network boundary. For regulated or sensitive work, that is often decisive: client contracts under NDA, health or financial records, internal code you are not allowed to paste into a third party. You do not have to read a data-retention policy or trust a vendor's word, because there is no vendor in the path.
If the data legally cannot leave your network, the model has to run inside it. That single constraint settles the local vs cloud question before cost or capability even enter the conversation.
Cloud is the opposite posture: you are trusting a provider and its data policy. Anthropic's policies are strong and you can configure retention, but "trusting a documented policy" is a different risk category than "the bytes physically never left." Know which category your work demands before you pick.
Does local really cost nothing?
Local is not free, it is prepaid. You buy the hardware once and then inference is "free" in the sense that there is no per-request charge, plus electricity. A capable GPU is the up-front cost; after that you can run a million classification calls and the marginal cost is rounding-error power draw.
Cloud is pay-per-token, and the figures matter when you do the break-even honestly. Approximate per-million-token pricing on Claude as I write this:
- Claude Haiku 4.5 (model ID claude-haiku-4-5): roughly $1 input / $5 output per million tokens, 200K context. The cheapest, fastest tier.
- Claude Sonnet 4.6 (claude-sonnet-4-6): around $3 / $15 per MTok, 1M context. The best speed-to-intelligence balance.
- Claude Opus 4.8 (claude-opus-4-8): about $5 / $25 per MTok, 1M context. The flagship for hard, long-horizon work.
- Claude Fable 5 (claude-fable-5): roughly $10 / $50 per MTok. Anthropic's most capable widely released model, for the hardest reasoning.
Run the math against your real volume. A $1,500 GPU is roughly 300 million output tokens of Haiku, or 60 million of Opus, before you break even on cloud spend, ignoring electricity and your own setup time. If you are doing high-volume, low-difficulty work (classification, tagging, dedup), local pays for itself fast. If you make a few thousand hard reasoning calls a month, the cloud bill is trivial and the hardware never amortizes. Be honest about which curve you are on.
Where does the capability gap actually bite?
A frontier cloud model outclasses anything that fits on a laptop, and the gap widens with task difficulty. For complex multi-step reasoning, large-codebase refactors, nuanced writing, or long agentic loops, Claude Opus 4.8 (or Fable 5 for the truly hard problems) is in a different league than a local 8B or even 70B model. Sonnet 4.6 is the sweet spot when you want most of that capability with better speed and a lower bill.
Local models are genuinely fine, though, for a real and large set of tasks: classification, extraction, first-draft generation, summarization of non-sensitive text, and anything you need to run offline or at high volume. The mistake is using a local model for the hard 10% and concluding local "isn't good enough," or paying Opus rates for the easy 90% and watching your bill balloon. Match the model to the difficulty.
How should you actually split the work?
Here is the decision list I use, in order:
- Is the data legally or contractually unable to leave your network? -> Local. Full stop, regardless of cost or capability.
- Is it simple and high-volume (classify, tag, extract, draft)? -> Local. The capability gap doesn't matter and you save real money.
- Does it need genuine reasoning, long context, or agentic tool use? -> Cloud. Start with Sonnet 4.6; escalate to Opus 4.8 when it's hard.
- Do you need it to work offline or with zero latency variance? -> Local.
- Is it a one-off where setup time costs more than the API call? -> Cloud, every time.
In practice that becomes a router: a cheap local model handles the bulk and the sensitive stuff, and a thin layer escalates the hard cases to Claude over the API. Calling the cloud side is a single POST; you check stop_reason and read the content back.
import os, anthropic
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
def escalate_to_cloud(prompt: str) -> str:
# Reach for the cloud only when the local model isn't enough.
msg = client.messages.create(
model="claude-opus-4-8", # or claude-sonnet-4-6 when cost matters
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
return msg.content[0].text
# Local handles the 90%; this runs only for the hard 10%.
print(escalate_to_cloud("Refactor this 400-line module and explain the tradeoffs..."))This hybrid shape shows up everywhere I build. When I wired up a personal AI voice assistant, wake-word detection and transcription ran locally for privacy and latency, but the actual reasoning went to the cloud. And it's the same pattern that reshaped how I work day to day: local models for the fast, private, repetitive grind, Claude for the thinking that's worth paying for. If you want the canonical reference for what's available where, the model lineup lives at platform.claude.com.
So stop framing it as local versus cloud. Frame it as routing. Send the sensitive, simple, high-volume work to a model on your own hardware where it costs nothing and never leaves the box, and spend cloud tokens only where frontier capability actually changes the outcome. Get that split right and you get the privacy and economics of local with the intelligence of the cloud, instead of compromising on one to get the other.
