The Batch API: A Flat 50% Off for Anything That Can Wait
All three major providers — OpenAI, Anthropic, and Google — run the same quiet promotion: send your requests as an asynchronous batch instead of one-by-one, accept results within 24 hours instead of seconds, and pay half price on both input and output tokens. No negotiation, no volume threshold, no enterprise contract. It is the single easiest large discount in the AI industry, and most teams never use it.
Why providers pay you to wait
GPU fleets are sized for peak interactive demand and sit partially idle between spikes. Batch jobs let providers fill those idle valleys with flexible work — so they share the savings. In practice, results usually arrive far sooner than the 24-hour ceiling; the ceiling is a guarantee, not a typical latency.
What belongs in a batch (more than you think)
- Document pipelines: summarization, extraction, classification, translation of backlogs.
- Content generation: product descriptions, SEO metadata, image alt-text at catalog scale.
- Data enrichment: lead scoring, sentiment tagging, entity extraction over datasets.
- Evaluations: running your model-quality benchmarks and regression suites.
- Embedding refreshes and periodic re-indexing jobs.
- Nightly reports and any cron-scheduled generation.
The litmus test is one question: does a human sit waiting for this response? If not, it is batch work — and running it interactively means paying double for latency nobody uses.
The math on a real pipeline
A nightly job classifies 50,000 documents (3,000 input + 200 output tokens each) on a model priced at $0.75/$4.50 per million:
| Mode | Input cost | Output cost | Monthly (30 runs) |
|---|---|---|---|
| Interactive | $112.50 | $45.00 | $4,725 |
| Batch (−50%) | $56.25 | $22.50 | $2,363 |
Same model, same output quality, same code apart from the submission format — $2,362 a month back for tolerating a delay on a job that ran at 3 AM anyway.
Provider specifics
- OpenAI: upload a JSONL file of requests, poll for the output file; 50% off, 24h window; separate, higher rate limits that don't touch your interactive quota.
- Anthropic: Message Batches API, up to 100K requests per batch; 50% off; results retrievable per-request as they finish; stacks with prompt caching.
- Google: batch mode in Vertex AI at 50% off interactive pricing, with BigQuery/GCS input-output integration for large-scale data jobs.
Stacking the discounts
Batching multiplies with the other levers rather than replacing them. That 50,000-document pipeline with a shared 1,000-token instruction prefix: caching cuts the repeated prefix by ~90%, batching halves what's left. Combined with choosing a mini-tier model over a flagship (the biggest lever of all — often 10–30×), total savings of 95%+ versus the naive setup are routine, not exceptional. Model your own pipeline with the batch toggle in our API cost calculator, and see the prompt caching guide for prefix discipline.
When not to batch
- Anything user-facing and synchronous — chat, autocomplete, live agents.
- Workflows where step N+1 depends on step N's output within the same run.
- Jobs with hard sub-hour deadlines: the 24h window is a ceiling you must design for, even if typical latency is much lower.
Pricing facts current as of 2026-08-02; check provider documentation for exact batch mechanics before implementation.