Token cost is the easy part. Idle compute, vector storage, cold-start latency, retries, and observability are where bills get out of hand. A field guide to designing for cost from day one.
The five hidden costs
- Idle compute. Self-hosted models that run on dedicated GPUs and serve sporadic traffic. Pay-per-token APIs are dramatically cheaper at low utilization.
- Vector storage and indexing. Re-indexing on every doc update is expensive. Incremental indexes pay for themselves in a quarter.
- Retry storms. Naive retry on rate-limit errors can quadruple your bill in a bad hour. Exponential backoff with jitter; honor Retry-After headers.
- Observability. Storing every prompt and completion at scale costs real money. Sample, summarize, expire.
- The eval bill. A 500-example eval against a frontier model on every PR adds up. Tier your evals.
Designing for cost
- Pick the cheapest model that hits your quality bar. Re-test quarterly; mid-tier models keep getting better.
- Cache aggressively. Identical inputs should not produce two model calls. Hash-key the prompt + context.
- Stream where users see output. Doesn't change cost, dramatically improves perceived speed.
- Batch where users don't. Background processing should hit batch endpoints when available — often half-price.
- Set per-tenant quotas. Or one customer's runaway loop will eat your monthly budget.
FinOps for AI
Tag every call with tenant, feature, environment. Build a daily dashboard that shows cost-per-feature. The teams that hit their cost targets are the teams that look at this dashboard every morning. The teams that don't, don't.
RULE
You can't optimize what you don't measure. You can't measure what you don't tag.