Self-Hosting vs API: The Break-Even Math Nobody Shows You
Every engineering team hits this question at some point: token bills keep climbing, open models keep getting better, and someone says "we could just rent an H100 and run Llama." Sometimes that's a brilliant call. More often it's a spreadsheet fantasy that dies on contact with a concept called utilization. Here's the honest math, with live prices on both sides of the equation.
The seductive headline number
An H100 currently rents from $1.49/hour (Vast.ai, checked 2026-08-02 — full comparison on our GPU rental pricing page). A well-tuned inference server (vLLM or similar) pushing a modern ~70B-class open model does on the order of 800 output tokens/second with batching. Run the division:
- $1.49/hr ÷ ~2.9M tokens/hr ≈ ~$0.5174 per million output tokens
- That undercuts every flagship API ($25–50/M output) and most mid tiers — it even beats a mainstream budget model like Google Gemini 3.5 Flash Lite ($2.50/M output).
- Worth knowing before you celebrate: the ultra-budget end of the API market (Mistral Mistral Nemo at $0.03/M) is already cheaper than your best-case self-hosting number. If that tier passes your quality bar, the build-vs-buy question is over before it starts.
Against the flagships you'd actually be replacing, self-hosting looks like a 10–50× win. That number is real — and it is also the most misleading number in AI infrastructure.
The utilization trap
The API charges you per token. The GPU charges you per hour, whether tokens flow or not. That single difference decides most build-vs-buy calls:
| GPU utilization | Effective cost / 1M output tokens | Verdict vs budget APIs |
|---|---|---|
| 100% (theoretical) | $0.5174 | clearly wins |
| 50% (very good in practice) | $1.03 | still ahead of mid tiers |
| 20% (typical business-hours app) | $2.59 | ties mainstream budget tiers |
| 5% (spiky or early-stage traffic) | $10.35 | loses to almost everything |
Real applications have nights, weekends, and lulls. Unless you run sustained heavy load — or can batch offline work to fill the valleys — your effective cost is 2–20× the headline number. APIs have no idle cost: they are utilization-proof.
The costs that never make the spreadsheet
- Redundancy: one GPU is one point of failure. Production needs at least two nodes — double the bill before the first token.
- Engineering time: serving stack setup, quantization choices, CUDA driver archaeology, model updates every few months. Budget several engineer-days per month — often the largest real cost.
- Cold starts and scale-up: traffic spikes force you to over-provision (pay for idle) or queue requests (pay in user experience).
- Quality gap management: if the open model is slightly weaker on your task, you pay in eval time, prompt engineering, or a hybrid fallback to a paid API anyway.
- Marketplace reliability: the cheapest rentals are peer-to-peer marketplaces — great prices, variable reliability. Datacenter-grade providers cost 1.5–3× more (see the provider comparison).
When self-hosting genuinely wins
- Sustained high volume: if you burn 50M+ tokens/day around the clock, utilization stays high and the math holds. Check your daily spend in the API calculator — if it's consistently above ~$50/day on budget-tier models, run the comparison seriously.
- Batch-heavy workloads: offline embedding, labeling, and generation jobs can be queued to keep GPUs at near-100% — the one case where the headline number is real. (Though compare the Batch API's 50% discount first.)
- Hard data-privacy constraints: when data cannot leave your infrastructure, the comparison isn't about price anymore.
- Fine-tuned small models: a 7–8B model fine-tuned on your task can beat flagship APIs on quality and run on a $0.35/hr consumer GPU — the strongest self-hosting case there is.
The 5-line break-even check for your workload
- Pull your monthly token volume from your provider dashboard (or estimate it in the calculator).
- Price it on the cheapest API model that passes your quality bar — that's your buy number.
- Estimate GPU-hours: monthly output tokens ÷ ~2.9M/hr, then divide by your realistic utilization (be brutal: 30–50% is good).
- Multiply by 2 for redundancy, add ~20 engineer-hours/month at your loaded rate — that's your build number.
- If build isn't at least 40% cheaper, take the API — the option value of zero ops is worth the margin.
Both sides of this equation move monthly — API prices fall, GPU rental prices fall, open models improve. The live numbers on this page and the GPU pricing table refresh so the comparison stays honest.