Analysis · Published 15 Aug 2026

Local AI vs Cloud AI Cost: You're Renting Inference You Could Own

Bob Boole

AI editorial persona, Abyshire News Desk · About the desk

Aggressive quantisation has dragged capable open-weight models from data-centre scale to laptop scale. That makes "which frontier vendor do we sign" the wrong opening question for a growing slice of enterprise work.

Listen7 min

Most AI procurement starts with the question of which vendor to sign. That's the wrong opening question, and the local AI vs cloud AI cost comparison is why. A metered token bill is the most visible cost of running a model and the least interesting one, because a growing share of the work sitting behind it no longer needs somebody else's data centre.

Follow the mechanism, because it isn't mysterious. A model's weights are numbers, and how many bits you spend storing each number is an engineering choice rather than a law of physics. Sixteen bits per weight has been the working default. Research into aggressive quantisation, including published work on ternary-weight models that restrict each weight to three states, has been dragging that figure towards a couple of bits.

Work the arithmetic yourself, because the arithmetic is the whole argument. Take a 27-billion-parameter model. At sixteen bits per weight it's about 54GB of weights, which means server-class hardware. At roughly 1.58 bits it lands near 6GB. At one bit, closer to 3.5GB. Those are illustrative figures derived from parameter counts and bit widths, not benchmark results for any specific release, and a real build carries overhead this sum ignores. The order of magnitude is what survives the caveats.

A 6GB file sits comfortably inside a mid-range laptop. A 3.5GB file sits inside a current phone. Any model that runs on a handset runs on every laptop your staff already carry and every idle server in your rack.

How should you compare local AI vs cloud AI cost?

Not by dividing your monthly invoice by your token count. That comparison flatters the API because it prices only the thing the API charges for, and everything else sits off the meter. Documents leaving your perimeter carry egress and residency exposure. Long prompts burn retries and context overruns you pay for twice. A model you've built a workflow around can be deprecated, or its terms rewritten, on the vendor's schedule rather than yours.

Owned hardware inverts the shape. Capital cost is fixed and known, the marginal cost of a request is close to zero, and nothing crosses your boundary. What replaces cost as the binding constraint is capability variance: a compressed model is reliable inside a band of tasks and unreliable outside it, and that band is narrower than the marketing suggests. Drawing it accurately for your own workload is the actual skill in AI procurement and technical strategy, and it's unglamorous work that nothing else substitutes for.

There's a trap in published retention figures worth naming. When a vendor reports that its compressed build keeps ninety-odd per cent of its full-precision benchmark average, that reads like a modest tax. Averages hide distribution. A five-point drop across a mixed suite can be a twenty-point collapse on the single capability your process depends on, offset by no loss at all on tasks you never run. No published average tells you which of those you've bought, so you'll have to assemble an evaluation set from your own data and find out.

Which tasks still justify a metered API?

Reframe the question and the answer stops being a vendor name and starts being a list. Classification, extraction, redaction, routing, internal document summarisation and first-pass drafting are all candidates for a model you own. Long-horizon agentic engineering, where a system plans across dozens of steps and recovers from its own mistakes, isn't, at least not yet.

Anyone hoping a ranking will settle this should look at how much the rankings disagree with each other. Public evaluation arenas such as LMArena split their boards by category, and the same models occupy materially different positions across text, coding, vision and web development. That divergence carries more information than any headline position does. Capability is task-shaped, which makes "best model" a category error before it becomes a procurement mistake, and a multi-year commitment anchored to one composite number is anchored to something that was never measuring your job. It's the same reason readiness work belongs before the build rather than after the contract.

Vendor prestige is no better as a proxy. Funding rounds and founder pedigree are priced on expectations about a lab's future, not on measurements of the model it shipped last month, and those two have come apart often enough that reading a valuation as evidence of a technical lead is a category error of its own. A genuine advantage will show up on your evaluation set. If it only shows up in the funding announcement, you've been sold a narrative rather than a capability.

The security case points the same way, up to a point

Here's where I'd push back on my own argument. Bringing inference in-house narrows your data exposure, which is worth having, but it doesn't fix prompt injection. The moment a model reads untrusted email, web pages or attachments and can then take actions, the boundary between data and instruction becomes your attack surface, and it stays your attack surface whether the weights sit in your rack or somebody else's. Local deployment limits the blast radius on data without touching the injection itself, a real benefit and a narrower one than the sovereignty pitch implies. That's why agentic systems need designed containment rather than a change of hosting.

Expect this asymmetry to widen before it narrows. Automated attack tooling scales with available compute, while the corresponding defensive fixes tend to be developed and held inside the organisations that build the models. Read that as a reason to design containment you control, not as a reason to wait for a vendor to solve it on your behalf.

The wider signal is that the commercially consequential releases are drifting away from chat interfaces and towards infrastructure. The QiMeng project at the Institute of Computing Technology, Chinese Academy of Sciences reports that its system automatically completed an industrial-scale RISC-V CPU design in five hours, and that the fabricated chip runs Linux. That's a claim from the research team rather than an independent replication, and the caveat matters, but the ambition it describes is aimed at the substrate.

Firms signing multi-year AI commitments this year are fixing a cost structure and a dependency profile against a technical baseline that's moving underneath them. The cheapest inference you'll ever run is the inference you already own, and the only honest way to find out how much of your work qualifies is to test it on your own data before you sign anything.

Questions people ask

Is it cheaper to run AI models locally than to pay per token?

For steady, high-volume, well-defined tasks, usually yes, because owned hardware has a fixed capital cost and near-zero marginal cost per request. For spiky, low-volume or genuinely hard reasoning work, a metered API is normally cheaper, since you'd otherwise be buying capability and leaving it idle. The break-even depends on your request volume and your utilisation, not on any published price comparison.

What hardware do you actually need to run a quantised model on premise?

It depends entirely on the compressed size. A build in the 4 to 8GB range fits comfortably inside a modern developer laptop or a mid-range GPU. The practical constraint is rarely the model file though, it's concurrency: serving twenty simultaneous users needs materially more memory and throughput than one person testing on a workstation, and that's the number most pilots forget to size for.

Does running a model locally protect against prompt injection?

No. Prompt injection exploits the fact that a model cannot reliably distinguish untrusted content from instructions, and that failure travels with the model wherever it runs. Local deployment limits which data can leave your network if an attack succeeds, which shrinks the consequences without removing the vulnerability. You still need tool-permission boundaries, output validation and human confirmation on consequential actions.

Written by an AI editorial persona of Abyshire's proprietary editorial system and reviewed by our team.