Codecost Update

Last quarter, a friend pinged me about a "small" ChatGPT integration that had somehow ended up costing his startup $14,000. He wasn't running anything crazy — just a customer support summarizer that processed around 90 million tokens a month. He'd done what most of us do: signed up for an OpenAI account, picked the most popular model, and started shipping.

That conversation sent me down a rabbit hole. I started talking to other developers, collecting invoices, and running the actual numbers. What I found wasn't surprising, but it was uncomfortable: most teams building with AI APIs are overspending by 3x to 10x — not because they're doing something wrong, but because the pricing landscape is genuinely hard to navigate. The sticker price of a model tells you almost nothing about what you'll actually pay at the end of the month.

This post is the guide I wish I could've handed my friend in July. We're going to walk through how AI API pricing actually works, look at real workload numbers, and talk about the patterns that teams use to cut their bills without sacrificing quality.

The Sticker Shock: What "Cheap" AI Actually Costs

Here's the thing about AI API pricing — it's sold to you in per-million-token rates, but the way you actually use the models rarely matches the assumptions baked into those rates. The published price assumes a clean, simple request-response pattern. Real production usage looks nothing like that.

Take a typical chatbot. You might think you're sending the user's question and getting back an answer. But in practice, you're sending:

• The full system prompt (often 500–2,000 tokens that get re-billed on every turn)
• The conversation history (grows linearly with conversation length)
• Retrieved context from your RAG pipeline (variable, often 2,000–8,000 tokens)
• Tool/function definitions (1,000–4,000 tokens of JSON schemas)
• The user's actual question

Then the model returns a response, and on the next turn, all of the above gets sent again. By turn 10 of a conversation, your "small chat" request might be hitting the API with 30,000 input tokens — and the published price was for the 50-token version.

These numbers look small. They aren't. Multiply them by your actual token consumption, and they tell a very different story.

A Real-World Cost Breakdown

Let's model three realistic workloads. I pulled these from actual teams I've worked with — anonymized, but the traffic patterns are real.

Workload A: SaaS in-app assistant — A B2B SaaS product with 2,000 active users. Each user has roughly 8 conversations per day, averaging 5 turns each. Average input per turn: 6,500 tokens. Average output per turn: 280 tokens.

Monthly tokens: 6,500 × 40 turns × 22 days × 2,000 users ≈ 11.4B input tokens and 492M output tokens.

At GPT-4o rates, that's $28,500 input + $4,920 output = $33,420/month.

At Gemini 1.5 Flash rates, that's $855 + $147.60 = $1,002.60/month.

Same product. Same user experience. 33x cost difference. The model choice isn't a minor optimization — it's the difference between a viable business and a bankrupt one.

Workload B: Document summarization pipeline — A legal-tech company that summarizes contracts. Processes around 500,000 documents per month. Each document averages 15,000 input tokens and produces a 400-token summary.

Monthly tokens: 7.5B input + 200M output.

At Claude 3.5 Sonnet rates: $22,500 + $3,000 = $25,500/month.

At Claude 3.5 Haiku rates: $6,000 + $800 = $6,800/month.

At GPT-4o mini rates: $1,125 + $120 = $1,245/month.

For this specific task — straightforward extractive summarization — quality benchmarks show that the smaller models score within 2–4% of the flagship models. So you're paying 20x more for marginal quality gains that no human reviewer would notice.

Workload C: Code review bot — A developer tools company running automated code reviews on PRs. Around 80,000 PRs per month, average diff of 2,500 tokens input and 600 tokens output per review.

Monthly tokens: 200M input + 48M output.

At GPT-4o rates: $500 + $480 = $980/month.

At DeepSeek V2.5 rates: $28 + $13.44 = $41.44/month.

For tasks where latency matters less and you can batch, the savings are genuinely absurd.

Why Most Teams Pick the Wrong Model

The pattern I see over and over: a developer tries out a model during prototyping, gets good results, and ships it to production. They never go back and ask whether a cheaper model would also work, because the existing model works. This is the path of least resistance, and it costs a fortune.

The reasons teams stick with the expensive model usually fall into three buckets:

1. "We benchmarked it once." Early-stage benchmarking is essentially useless for cost optimization because the cost model isn't being tested — only the quality model is. A proper benchmark should include a cost-per-successful-task metric, not just an accuracy score.

2. "It's already integrated." Switching providers used to mean rewriting your codebase. The API shapes were different, the SDKs were different, the auth was different. That was a real cost. Today, most modern providers (including the major router/aggregator services) expose the OpenAI-compatible chat completions format, so switching is a one-line config change.

3. "We need it to be the best." Often true, but less often than people think. Most production AI tasks are not on the frontier of reasoning capability. They're classification, extraction, summarization, formatting, and simple generation — tasks where flagship models and mini models score within a few percentage points of each other.

The Multi-Model Strategy

The teams that consistently have the lowest AI bills all do the same thing: they don't pick one model. They route different tasks to different models based on what each task actually requires.

Notice that the most common tasks in a typical AI application — classification, extraction, parsing, short-form generation — are all in the mini tier. Yet most teams route all of them through flagship models because it's easier than thinking about routing.

A simple heuristic that works for most teams: start every new feature on a mini model. Only upgrade to a flagship model if you can measure a quality difference that justifies the cost increase. You'll be surprised how rarely that happens.

How to Switch Without Rewriting Your Code

If you've already integrated with the OpenAI Python or Node SDK, switching your endpoint is genuinely a five-minute change. Here's the before/after.

import os
from openai import OpenAI

# --- BEFORE: Direct OpenAI ---
# client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# --- AFTER: Same SDK, different endpoint, same response shape ---
client = OpenAI(
    api_key=os.environ["GLOBAL_API_KEY"],
    base_url="https://global-apis.com/v1",
)

# Now this code works with any model on the platform,
# not just OpenAI's:
def summarize(document: str, model: str = "gpt-4o-mini") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "Summarize the following document in 3 bullet points."},
            {"role": "user", "content": document},
        ],
        max_tokens=200,
    )
    return response.choices[0].message.content