AI Application Cost Engineering: Token Ledgers, Unit Economics, and Cost Governance — AI Development Tutorial

Defining Cost Engineering: Turning "Reading the Bill at Month End" into an Everyday Operating Metric

Most teams first become aware of AI cost problems on the day the vendor's monthly invoice lands in the finance inbox. That invoice carries exactly one number: how much was spent last month. It answers "are we getting more expensive," but it cannot answer "why are we getting more expensive," "which line of business got more expensive," or "which line dies first if it doubles." So what usually follows is nearly always the same script: hold a meeting, guess at a few causes, adjourn, and look again next month. That loop can run for six months with zero improvement, because the shape of a billing statement does not support attribution in the first place.

The first thing cost engineering does is not save money — it changes the unit of data. It reduces "money" to attributable cost events, and reconstructs "monthly total" as real-time accumulation plus dimensional slices plus rate. Only once cost is mapped to the objects an organization actually cares about — features, tenants, models, channels — does cost stop being a financial outcome and become an operating signal. And operating signals have a specific character: you can look at them daily, set thresholds on them, let them trigger actions, and assign them to an owner.

One easily confused boundary needs to be drawn first. Cost engineering is not inference optimization. Inference optimization asks how to finish the same job with less compute under the same load; its scope is the engine, quantization, and batch scheduling layer. Cost engineering asks whether the commercial shape of the product matches its inference overhead; its scope is accounting, attribution, pricing, budgeting, and product design. The two are mutually dependent: without cost engineering you cannot tell how much money an inference optimization actually saved, because the denominator is undefined; without inference optimization, cost engineering can only allocate and brake, never produce structural improvement. This article is about the former, but it will lean on conclusions from the latter repeatedly.

1.1 Seven Different Kinds of Money Hide Inside the Bill

Take an LLM invoice apart and a deceptively simple statement contains at least seven categories of spend that are entirely different in nature — different cost drivers, different attribution paths, different optimization levers:

Uncached input tokens. This scales linearly with business volume and with context length, which makes it the easiest thing to run away.
Prefix-cache-hit input tokens. The unit price is usually significantly lower than ordinary input, and its cost depends on how long your stable prefix is, how generous the TTL is, and how "clustered" your request order is.
Output tokens. These are driven by two factors at once — answer length and the model's autoregressive decoding cost — and their contribution to latency is far larger than input's.
Reasoning / thinking tokens. Chain-of-thought models produce a large block of tokens the user never sees but that are billed all the same. Traditional BI reports routinely fold this into output, which distorts every "output is too expensive" analysis.
Tool and external service costs. Search, code execution, database queries, third-party APIs: these don't come from the model vendor, yet they count toward the real cost of a task. If unit economics leaves them out, the gross margin it reports is inflated.
Retry and fallback costs. Retrying after a timeout, switching providers after rate limiting, regenerating after a structured-output parse failure — all of these are silent cost multipliers.
Non-generative inference costs. Embedding, reranking, intent classification, guardrail models, image and speech: these share the same "inference budget" concept as the main chat model, but their price structures and optimization levers are completely different.

Collapsing these seven categories into a single total is the same as compressing entirely different cost drivers into one scalar. The first discipline of cost engineering is to never maintain a single number called "AI cost" — maintain itemized accumulations only. Any attempt to manage through one grand total breaks down the first time cost misbehaves, because you have no drill-down path.

1.2 The Two Sources of Cost: Traffic and Unit Price

Total cost = unit price × usage. This sentence looks like a tautology, but it is the core framework of cost engineering, because it dictates that all optimization has exactly two paths:

Cutting unit price: switch to a cheaper model, quantize, negotiate cache discounts, pick a model size better matched to the task. These are one-off or low-frequency engineering moves with predictable effect, but they carry a quality cost.
Cutting usage: shorten context, drop a few turns, raise the cache hit rate, batch whatever can run offline, route on demand to smaller models, redesign away the tasks that only a large model could serve. These are continuous product moves, theoretically lossless in quality, but they demand long-term governance.

Worth noting: cutting usage is usually worth more than cutting unit price, and has fewer side effects. Plenty of teams reach for a model swap first, when the real problem is that the task never needed that many tokens in the first place: three overlapping retrieval results injected, history messages never structurally trimmed, tool definitions re-injected every turn, uncertain tasks re-asked over and over. That's pure waste, and no model swap fixes it.

1.3 Why "Reading the Bill at Month End" Is Doomed to Fail

The monthly review pattern has three structural defects:

Latency. The overspend happens in week one; you learn about it at the end of the month. There is no correction window at all.
No attribution. A rising total could come from one large customer's usage growth, a new feature ramping behind a canary, a crawler-like traffic anomaly, or a newly launched recursive call. All four are the same number in the total.
No denominator. "It's more expensive this month" is an absolute-value statement. What you need is unit cost — per session, per successful task, per active tenant — to tell whether the trend means "we got more expensive" rather than "we used more."

The goal of cost engineering can be stated in one sentence: at any moment, be able to answer within two minutes which feature, which tenant, and which model has a rising unit cost, and whether that rise came from a change in unit price or from a change in usage mix. If you can't hit that response speed, the ledger isn't built right.

Get the Ledger Right First: The Cost Event Model

The ledger is the foundation of cost engineering. Many implementations fail not because the algorithm is wrong, but because a modeling mistake was made at the very bottom: treating "one HTTP request" as one cost event. That assumption inevitably collapses in any scenario with retries, routing fallbacks, or tool loops, because a single user action usually sits on top of several billable calls.

2.1 A Minimum Viable Cost Event Schema

A cost event should be the atomic unit of the ledger. Its definition is: a model call that is visible to the billing party and independently attributable. One user action (trace) is composed of several cost events; one session is composed of several traces. This layering lets you answer both "how much did this interaction cost" (trace level) and "how much did this user spend this month" (session level).

Below is the recommended minimum field set. The key points are the three-way token split and the price snapshot:

from dataclasses import dataclass, field
from decimal import Decimal
import time, hashlib

# 成本事件：一次"可归因的计费事实"，而不是一次 HTTP 请求。
# 一次用户动作可能触发多次上游调用（重试、路由回退、工具循环），
# 所以账本粒度必须比请求更细，否则所有重试成本都会被静默吞掉。
@dataclass(frozen=True)
class CostEvent:
    event_id: str            # 幂等键：hash(trace_id, attempt, call_seq)
    trace_id: str            # 贯穿一次用户动作
    session_id: str
    tenant_id: str
    feature: str             # 功能维度：chat / summarize / embed / rerank ...
    route: str               # 逻辑路由名，如 "chat.default"
    model: str               # 实际服务的模型（可能被路由改写）
    provider: str

    # 三段式 token，比单一 total 更适合归因
    prompt_tokens: int
    cached_prompt_tokens: int   # 命中前缀缓存的输入部分
    output_tokens: int
    reasoning_tokens: int      # 思考型模型产生、用户看不到但照常计费的部分

    # 价格是"当次生效"的快照，而不是当前配置。
    # 供应商调价后，历史事件仍按当时价格计算，避免账目被追溯改写。
    price_snapshot: dict        # {"in": Decimal, "cached_in": ..., "out": ...}
    unit: str = "1k_token"

    attempt: int = 1            # 第几次尝试，>1 即为重试
    ts_ms: int = field(default_factory=lambda: int(time.time() * 1000))
    env: str = "prod"           # prod / staging / eval / replay
    labels: dict = field(default_factory=dict)

    @property
    def billable_input(self) -> int:
        # 未命中缓存的输入部分
        return max(self.prompt_tokens - self.cached_prompt_tokens, 0)

    @property
    def cost(self) -> Decimal:
        p = self.price_snapshot
        total = (
            Decimal(self.billable_input) * p["in"]
            + Decimal(self.cached_prompt_tokens) * p["cached_in"]
            + Decimal(self.output_tokens) * p["out"]
        )
        return total / Decimal(1000)

def make_event_id(trace_id: str, attempt: int, call_seq: int) -> str:
    """幂等键必须由业务侧稳定字段构成，不能用自增主键或时间戳，
    否则上游重投、消费端 at-least-once 都会造成重复计账。"""
    raw = f"{trace_id}|{attempt}|{call_seq}".encode()
    return hashlib.sha256(raw).hexdigest()[:24]

Three design decisions deserve separate mention.

First, tokens must be split three ways rather than collapsed into a single total. In the usage payload returned by the vast majority of model APIs, input, cached input, output, and reasoning tokens come back separately — yet many teams sum them into one number the moment they land in storage. The direct consequence: the cache hit rate becomes uncomputable, the retry multiplier becomes uncomputable, reasoning cost gets charged as output cost, and every downstream analysis rests on a broken foundation.

Second, price must be a snapshot rather than a reference. If the event only stores model="xxx", then the moment a vendor changes prices all of your historical cost gets recomputed, and yesterday-versus-last-month comparisons lose all meaning instantly. Storing a snapshot is a very small extra cost, but it guarantees the immutability and auditability of the books — a hard requirement when finance reconciles.

Third, the attempt field must be a first-class citizen. Retry rate is one of the core metrics of cost health: under normal conditions retries should be a very small fraction, and a rising retry rate for a given model or a given time window usually means upstream quality has degraded, your timeout is set too tight, or some particular class of bad request is firing repeatedly. It should be the first queryable field, not something buried in logs.

2.2 Dimension Design: Feature, Tenant, Model, Channel

The design principle for dimensions is: every dimension must correspond to a real decision action. Don't add a dimension that has no corresponding action, because dimension explosion makes aggregation queries expensive and nobody ends up using them.

feature (feature): this is the most important dimension and also the easiest one to design wrong. It must map to product navigation or a user-visible functional unit, not to a code module name. The most common mistake is grouping by service name — "chat-service" holds both the free toy and the paid deep analysis, all mixed together. Under that dimension design you can never answer "which feature is losing money," which is precisely the one question a product manager cares about.
tenant_id (tenant): the only entry point for unit economics. Without a tenant dimension you can only compute things like total revenue minus total cost — averages with no decision value. The tenant dimension must also come from real authentication context, not from a forgeable field in the request body.
model / route (model and route): note that these are two different things. route is the logical route name you configured; model is the model that actually served the request. Storing them separately is necessary because once you route "complex tasks" to a strong model, the change on the route dimension reflects a change in business mix, while the change on the model dimension reflects a change in price or behavior. Mix them together and you can't even find out who a price change actually affected.
provider (channel): mandatory in any multi-provider setup, used to evaluate switching gains and to investigate cost anomalies during an outage.
env (environment): this one is ignored by many teams, and the result is that evaluation traffic, offline replay, and internal dogfooding all get mixed into production cost. A cost report carrying an env label should be able to explain the gap between its "production cost" figure and the vendor invoice with one or two queries; without that label, the gap is a black box you can never explain.

2.3 Billing Precision and Idempotency

There will always be a gap between the ledger and the vendor invoice, and how you handle it depends on what the numbers are for:

External financial reporting: use the vendor invoice as the ground truth and let the ledger do nothing but allocation. This requires the ledger to stay as close as possible to the invoice on the "total" dimension — otherwise the allocation coefficients are wrong, and then every allocated number is wrong.
Internal operating reporting: use the ledger as the ground truth, because it carries dimensions. Carry the gap in an explicit "unallocated balance" account instead of smearing it across features and pretending it doesn't exist.

The implementation has to hold two engineering disciplines. First, event writes must be idempotent. Model calls can report duplicate events under retry scenarios, message queues usually give you at-least-once delivery, and floating-point accumulation drifts over a long run. A workable approach is to accumulate with a business idempotency key plus exact decimals (such as Decimal) or integerized minimum billing units, rather than recomputing the historical sum on every write. Second, the ledger needs a reconciliation channel. Some costs simply aren't knowable at the moment the call happens: a stream cancelled midway, a batch job tallied after the fact, follow-up calls produced by asynchronous tool callbacks. These events need late posting with an explicit reconciliation timestamp, or you'll watch a cost curve jump at the end of the month.

Decomposition Conventions: Three Ways to Read the Same Spend

Once the ledger is in place, the same spend has to be read through at least three different conventions. Confusing conventions is the single most frequent source of error in cost conversations: product says "our feature is cheap," finance says "costs went up," and neither is wrong — they're just looking at different facets of the same thing.

3.1 Request-Level Truth: What a Single Call Actually Cost

This is the lowest and least error-prone convention, taken directly from the usage data returned by the provider. Its job is to calibrate: every higher-level convention should converge down onto it. If the cost roll-up for some feature dimension doesn't reconcile with request-level truth, you have lost or duplicated events — fix the data before you talk about analysis.

3.2 Feature-Level Allocation: Splitting a Trace Across Features

A single user action is rarely pure generation: it may trigger intent detection (one small model call), vector retrieval (one embedding batch), reranking (one rerank call), main generation (one large model call), and a post-output guardrail check. Feature-level allocation has to answer the question: "The user clicked the 'generate weekly report' button — what did the total cost of that click amount to, including every intermediate step?"

There are two routes to allocation:

Direct attribution: every sub-call carries the final feature label from the moment it is issued. This is the most accurate, but it requires the call chain to propagate context labels reliably, which in practice means putting the label in a request header or context object and preventing middleware from dropping it.
Post-hoc allocation: aggregate all events by trace, then allocate by some weighting scheme. This is simple to implement, but it needs an explicit allocation rule — and the rule itself becomes a source of error.

The practical recommendation is: use direct attribution on the main path, use post-hoc allocation as a fallback, and treat "the share of traffic falling back to allocation" as a monitored metric in its own right. If that share keeps rising, label propagation has broken somewhere — usually a newly added piece of middleware or an async job that forgot to carry the label.

3.3 Tenant Level and Billing-Period Alignment

There's an easily overlooked precondition for a correct tenant-level roll-up: the aggregation window must include sessions that haven't closed yet. A streaming response may still be running its last few tokens on the server after the user closes the page; async jobs, bulk backfills, and webhook-triggered follow-up calls may generate cost hours or even days later. If you hard-split on the calendar month, monthly tenant gross margin will carry a systematic time mismatch.

Two ways to handle this. One is to introduce an "unbilled session" account, carrying the cost of incomplete sessions at the end of the window as a separate accrual and settling them in the next period. The other is to define the billing period by session end time rather than request start time, which is fairer for long sessions. The first is more common in external financial reporting, the second is more sensible for internal analysis — the key is never to mix both conventions in the same report.

Per-Session Cost: Conventions, Lifecycle, and the Formula

"How much does one session cost?" is the most-asked and most-easily-answered-wrong question in this whole area. The reason it's answered wrong is almost always that the convention was never stated: who is doing the calculation, how far it goes, whether cache is counted, whether reasoning tokens are counted.

4.1 A Session Is Not a Request

The real cost of a single session contains at least five parts:

every model call across all turns of the conversation (including those that were retried and those that fell back);
every auxiliary call triggered within the session — retrieval, embedding, rerank, intent recognition;
the external cost of the tool calls themselves (search APIs, databases, code sandboxes);
the "continue generating" calls triggered after a tool returns — this one is frequently missed entirely, because its trace isn't in the same chain as the original request;
if the session produces side effects (writing to the vector store, generating and storing a long document), there's non-token storage and indexing cost on top.

Counting only "the handful of main chat calls" is the most common convention error. It systematically understates sessions that involve tools, and that misjudgment then corrupts product decisions: you assume agent scenarios cost about the same as ordinary chat, when in reality a tool loop multiplies cost — sometimes by an order of magnitude.

4.2 Billed Tokens vs. Effective Tokens

Maintain two metrics side by side, because their uses are completely different:

Billed tokens: what was actually generated and billed, used for reconciliation and spend control. It includes retries, includes thinking tokens, and includes intermediate output that was discarded.
Effective tokens: what actually contributed to delivering a result to the user — typically the final visible output plus the input that the model genuinely consumed and that wasn't cached. It excludes retry amplification, excludes thinking tokens, and excludes useless generations caused by system bugs.

The ratio between the two — the amplification factor — is itself the most valuable health metric you can have. If billed tokens / effective tokens stays high over time, the system is generating uselessly. If the share of thinking tokens is too high, the task may not need reasoning that deep. If the share of retried tokens is climbing, that's a stability problem rather than a cost strategy problem, and it needs engineering to get involved. Putting the amplification factor on your monitoring dashboard is far more useful than staring at total cost, because it's far more sensitive to anomalies: total cost carries business volume as an enormous noise term, while the amplification factor is a normalized signal.

4.3 The Session State Machine and Reconciliation

Modeling the session as a state machine is a precondition for computing per-session cost correctly. Typical states:

会话状态机（简化）——每个状态迁移都是一个可能产生成本事件的点

CREATED
  └-> CLASSIFYING      意图/路由判断（小模型，可能多次）
        └-> RETRIEVING   embedding + 向量库查询 + （可选）rerank
              └-> GENERATING   主模型流式生成
                    ├-> TOOL_CALL     模型请求调用工具 → 暂停
                    │     └-> TOOL_RESULT  工具返回 → 回到 GENERATING
                    │           （每次回路都是一次新的 GENERATING，成本叠加）
                    ├-> GUARDRAIL     输出后检查
                    └-> COMPLETED

异常分支：
  GENERATING --timeout--> RETRYING --> GENERATING      （attempt+1，成本叠加）
  GENERATING --client abort--> CANCELLED                 （已生成的 token 仍计费）
  GENERATING --overload--> FALLBACK_MODEL --> GENERATING（不同模型，价格快照不同）

The direct engineering conclusion the state machine yields is: cost events should be written on state transitions, not on the HTTP handler. Only by hanging the write on the transition do tool loops, timeout retries, and client cancellations avoid going unrecorded. At the same time, every session needs an explicit cost_settled flag; a session without it is "unbilled" and must be visible on its own in reporting.

Unit Economics: Revenue per Customer, Inference Cost, and Gross Margin

If the ledger answers "where did the money go," unit economics answers a sharper question: does this product make money, and whose money does it make. This is where cost engineering graduates from a technical function into a business function, and it's also the part most often done wrong.

5.1 Switching From Aggregate Thinking to Distribution Thinking

The most common wrong move is: monthly revenue minus monthly cost, that gives a gross margin, take it into the business review. The problem with that number is that it is entirely sensitive to the user distribution. If revenue is uniformly distributed while cost follows a power law (a few heavy users burning most of the tokens), the aggregate gross margin gives the same answer for two completely different business shapes: a healthy subscription business, and a marginal business being dragged under by a handful of large customers.

The right move is to segment users by at least some quantile and compute the unit cost per segment. Common segmentation dimensions are consumption tier (light, medium, heavy), plan tier, and feature-usage profile (whether tools are enabled, whether context is long, whether the batch API is used). Every segment must be able to answer "what is the gross margin of this segment" independently.

One statistical trap has to be flagged here: never divide total revenue by total tokens to represent the cost of a "typical user". The mean of total tokens is pulled up severely by the long tail, whereas the cost of a typical user should be characterized by the median or a quantile. Every expression of the form "the average user" in cost engineering should state explicitly whether it means an arithmetic mean or a quantile — under a power-law distribution, the two can differ by an order of magnitude.

5.2 Contribution Margin and Layered Statements

The recommended cost structure model has three layers, narrowing from the outside in:

Layer one: revenue − direct inference cost = inference gross margin. This is the coarsest layer, covering all costs across all tenants and all features. Its value is that it provides a total-volume constraint; it is not suitable for making decisions.
Layer two: inference gross margin after segmenting by tenant. The core metric here is "gross margin contribution per tenant" — that tenant's revenue minus all inference cost attributed to it (including retrieval, reranking, guardrails, tools). This layer is the foundation for pricing and plan design.
Layer three: task gross margin after segmenting by feature / scenario. The core metric here is "cost per successful task." This layer feeds product design — it tells you which features deserve continued investment, which should be rate-limited, and which should be taken down and rebuilt.

The three layers must be self-consistent: the sum of layer two across tenants should equal layer one, and the sum of layer three across features should equal layer two. If they don't reconcile, the difference must be explicitly exposed as an "unallocated" account — do not smooth it away pro rata. Smoothing leaves every downstream analysis resting on an untrustworthy base, and this class of error often goes unnoticed for months.

5.3 Three Ratios That Must Never Be Mixed

When reporting to a non-technical audience, the following three ratios get confused most often, and confusing them leads to exactly opposite conclusions. Every chart must be labelled with which one it uses:

Cost per active user: the denominator is the number of active users, which is heavily influenced by retention swings. During a growth phase it naturally declines, and that gets misread as "optimization is working."
Cost per session: the denominator is the number of sessions, which is heavily influenced by product interaction design. Split a task that could be completed in one turn into three turns and cost per session drops substantially — while cost per task may actually rise.
Cost per successful task: the denominator is the number of "successful" tasks. This metric is closest to real economics, because it pushes the cost of failed retries into the numerator. But it requires a clear definition of success, and defining "success" is a product problem rather than a technical one.

The practical recommendation is: use cost per successful task on the internal day-to-day dashboard; use absolute contribution margin plus trend for management reporting, and always state which denominator is in play. You need both — but don't tell them as one story.

5.4 The Cost Floor: The Worst Case Pricing Must Cover

Once you have cost per successful task, pricing acquires a clear constraint. The key question is: should a feature's price be aligned to its median cost or to a cost quantile. The answer is a quantile — specifically a higher quantile. The reason is that the long tail of the cost distribution is not an anomaly; it is part of the product's shape. Long-context tasks, complex multi-turn tasks, and tool-enabled tasks are exactly the highest-value shapes for this kind of feature, and part of their high cost is value creation itself. It cannot be squeezed out.

The corollary is: cost optimization should target the body of the distribution first, not the tail. If p50 cost is high, most users are being consumed inefficiently — that signals systemic problems in prompt structure, routing policy, or turn design, and you should fix that first. If p50 is fine but p95 is high, a specific class of usage is eating the budget, and the right lever is product-side tiering (turn heavy workloads into a separate premium tier or a dedicated feature) rather than continuing to optimize the average. The two diagnoses call for completely different actions; conflating them sends optimization in entirely the wrong direction.

Another issue that must be handled explicitly is the cost floor of free traffic. Free trials, public demos, and internal dogfooding all consume inference resources, and their cost is ultimately borne by paying users. There are three ways to handle it: set a hard cap on the free allowance, distinguish free from paid cost accounting by feature, or simply accept it as part of customer acquisition cost and include it in CAC. The actual mistake is neither accepting it nor booking it anywhere — that inflates gross margin and leaves decisions about the free strategy with no basis at all.

Cost Reduction Techniques and a Framework for Quantifying the Gains

There are many cost reduction techniques, but most teams make the same mistake when evaluating them: they account for the change in cost but not the change in quality. Save tokens and let task success rate fall from 90% to 70%, and the true unit cost may actually go up. So you need a unified evaluation framework — build the framework first, then apply it technique by technique.

6.1 A General Framework: The Cost-Quality Exchange Rate

The core metric for evaluating any cost reduction technique is not "what percentage of cost did we save," but how cost per successful task changed:

from dataclasses import dataclass

@dataclass
class ArmResult:
    arm: str               # 方案名，如 "baseline" / "gpt-mini-routed" / "cached-v2"
    n: int                 # 评测样本数
    cost_total: float      # 方案总成本（绝对值，评测集固定时才可跨方案比）
    success: float         # 端到端任务成功率，不是 BLEU、不是人工主观分
    p95_latency: float
    escalation: float      # 转人工 / 二次求助 / 任务放弃率

def unit_cost_per_success(r: ArmResult) -> float:
    """单位成功任务成本 = 每次尝试成本 / 成功率。
    这是唯一可以跨方案、跨模型、跨缓存策略比较的指标。"""
    per_attempt = r.cost_total / max(r.n, 1)
    return per_attempt / max(r.success, 1e-6)

def marginal_change(base: ArmResult, cand: ArmResult) -> float:
    """返回相对基线的单位成本变化率，负数表示变便宜。"""
    b = unit_cost_per_success(base)
    c = unit_cost_per_success(cand)
    return (c - b) / b

def pareto_front(arms: list[ArmResult]) -> list[ArmResult]:
    """在"单位成本 - 成功率"平面上做帕累托筛选。
    一个方案只有同时更便宜且不掉成功率，才值得上线；
    只更便宜会掉成功率的方案，应当以质量损失为代价单独做产品决策。"""
    pts = sorted(arms, key=unit_cost_per_success)
    front, best = [], None
    for a in pts:
        if best is None or a.success > best:
            front.append(a)
            best = a.success
    return front

Three points on using this framework:

The evaluation set must be fixed and stratified. An aggregate success rate hides structural change — one variant might raise success substantially on easy tasks and drop it just as substantially on hard ones, leaving the net looking unchanged. Only stratified evaluation exposes the false optimization of "trading one task set for another."
It must include the hardest tail samples. If your evaluation set contains only medium-difficulty tasks, you will never learn that a cost reduction technique collapses on extreme inputs (very long documents, complex multi-step reasoning) — which is precisely where cost is most likely to run away.
Latency cannot be a free cost. Batching, deeper reasoning, and longer prompt-compression preprocessing all trade latency for cost. Batching pushes interactive latency from seconds to hours, which is unacceptable in many products. p95_latency and escalation must enter the decision table alongside cost.

6.2 Prompt Caching: Fix the Hit-Rate Definition Before Talking About Gains

Prefix caching is one of the techniques with the most direct payoff, and it is also the most easily misused. Plenty of teams see a dollar amount in the "cached input" column of the invoice and conclude that caching is working. In reality, for most teams the problem isn't that cache pricing is too high — it's that the hit rate is too low, and a low hit rate usually comes from system design rather than configuration.

To get this right, the definition of hit rate has to be nailed down first:

# 命中率的三种口径，必须在报表里注明用的是哪一种
hit_token_rate = cached_prompt_tokens / prompt_tokens
#   "token 口径"：被缓存命中的输入 token 占比。适合衡量提示工程效果。

hit_call_rate = count(events with cached_prompt_tokens > 0) / count(events)
#   "调用口径"：有多少次调用吃到了任何缓存。适合衡量系统设计的聚集性。

hit_prefix_len = min(P, ...)   # 稳定前缀长度（token 数）
#   "前缀口径"：真正被复用的稳定前缀有多长。
#   这个口径最接近收益来源：只有稳定前缀才产生可观的复用率。

# 三者的关系：
#  token 口径 ≈ 调用口径 × 单次命中的平均覆盖比例
# 所以"调用口径很高但 token 口径很低"= 缓存系统在工作，
# 但你的提示里绝大部分内容仍然在缓存边界之外被重算。

The common root causes of a low hit rate, ordered by how often they actually show up:

The stable prefix is being broken. Timestamps, request IDs, random values, user nicknames — anything that changes every round — get inserted ahead of the system prompt. Caching only hits on an "exactly identical prefix," and a single differing byte invalidates the cache all the way to that position. The canonical practice is to place all dynamic content after the prefix, and to enforce that rule as a lint check or a hard constraint in the template layer.
Request ordering isn't clustered. Caching has capacity and a time window, and requests from different tenants and different features evict each other. If your traffic is long-tailed and uniform, the hit rate is naturally very low. At that point "sharding the cache pool by tenant or by feature" is usually more effective than any cache parameter tuning.
Multi-turn conversation pushes the prefix ever longer. In multi-turn scenarios the history keeps changing, which means each turn's prefix differs from the previous turn's. The remedy is cache-anchor segmentation: fix the unchanging long-lived part (system prompt, tool definitions, long-term memory) as an independent prefix, put the session's short-term history after it, and keep the long-lived part permanently cacheable.
TTL doesn't match session rhythm. If the cache lifetime is shorter than the interval between two requests sharing the same prefix, the hit rate approaches zero. This requires deriving the TTL requirement from the real distribution of inter-request intervals, rather than copying the default.

The boundary needs to be stated clearly: caching optimizes "repeated input," not "redundant input". If your problem is that a single task injects thirty thousand tokens of duplicated material, the correct move is to delete the unnecessary content first — deleted content generates neither cache cost nor cache benefit. Caching optimizes on top of a correct prompt structure; it is not a substitute for one.

Also watch out for fraudulent attribution of cache gains. A common wrong move is: ship caching, look at how much the invoice dropped, and credit the entire difference to caching. That badly overstates the benefit in three situations: prompt slimming happened in the same period (input tokens were already falling), the business mix shifted in the same period (a lower share of short requests also lowers cost), and you moved to the batch API in the same period (batching carries its own discount). The correct attribution method is a controlled comparison: fix the evaluation set, use the same time window, toggle only the cache on and off, and compare cost per successful task for the same batch of tasks under both configurations. Without that control, any number claiming "how much did caching save" is marketing material and cannot enter a decision.

The last easy-to-miss point is that there is a trade-off between cache capacity and concurrency. The VRAM or host memory that caching consumes competes with inference resources; under tight GPU memory pressure, expanding the cache can directly reduce concurrency, which then raises unit cost. Cache tuning therefore has to be observed on both "hit rate" and "throughput" at once — watching hit rate alone can lead you to a choice that damages total cost.

6.3 Prompt Compression: Gains Must Be Corrected for Task Success Rate

Prompt compression (summarizing history, extracting structured state, trimming retrieval results, dropping redundant examples) looks like pure gain: fewer input tokens, cost naturally falls. But it carries an implicit cost: the information you compressed away may be exactly what the task needed. And that cost surfaces late — a week after compression everything may look fine while an edge case that needs a long chain of information starts failing, and because in metrics that case is merely a slow slide in success rate, it gets attributed to "something else."

So the validation discipline for prompt compression is:

Report compression ratio separately; never conflate it with cost savings. The fraction of input tokens removed is the compression ratio; cost savings also depend on cache hit behavior — if what you compressed happened to be the stable prefix, compression may actually reduce cache benefit.
A success-rate regression is mandatory, and observation must be delayed. Compression effects must be evaluated on sessions that occurred after compression took effect; you cannot compare against pre-compression historical data. The evaluation window should cover at least several of the compression strategy's typical failure cycles.
Be conservative on "information-preserving" tasks. If the task genuinely requires full recall of multi-turn detail, summarization is simply the wrong technique; what you should build is structured storage plus on-demand retrieval. The correct shape of compression is "changing how information is obtained," not "reducing how much information there is."

6.4 Quantization: How Precision Loss Propagates to Business Metrics

Quantization (reducing the numerical precision of weights and activations) belongs to the "cut unit price" path. Its economics are straightforward: smaller bitwidth means less memory bandwidth consumed and higher effective throughput, which in turn means more concurrency served on the same hardware. Its cost propagation chain is: quantization → lower marginal cost per token → higher maximum concurrency under a given latency target → lower unit cost.

But precision must be evaluated along the correct chain, or the conclusions won't be reliable. The common mistake is to look directly at the change in perplexity or leaderboard scores and extrapolate that to business impact. Perplexity is a language modeling metric in the average sense, and it is insensitive to exactly the capabilities the business cares most about (long instruction following, validity of structured output, adherence to instruction constraints, argument correctness in tool calls).

The correct evaluation surface is a set of task-level capability metrics, read with attention to differences across difficulty strata. A common pattern is that metrics on easy tasks barely move while metrics on hard tasks drop noticeably, leaving the aggregate average looking almost unchanged. That means the benefit of quantization is probably "trading significant degradation in tail capability for a linear drop in marginal cost" — whether it's worth it is purely a product decision. The right input to that decision is stratified data plus a stratified revenue distribution, not a single overall mean.

6.5 Small-Model Routing: The Cost of a Router Misjudgment Is Asymmetric

Model routing (small model for simple tasks, large model for complex tasks) is a structural cost reduction technique, because what it lowers is the "average unit price" rather than the price of every task. Its economics can be described with a very clean framework:

# 模型路由的经济学（用变量描述口径，不依赖任何具体价格）
#
# 设：
#   p        = 判定"任务简单"的概率
#   q        = 判定为简单、且小模型确实能答对的概率（判定的精度）
#   r        = 大模型单独完成该任务的正确率
#   s        = 小模型单独完成该任务的正确率（s < r）
#   C_small, C_large = 两种模型的单次尝试成本
#   C_route  = 路由器判定本身的成本（可用一个极小模型，通常接近 0）
#
# 期望成本：
#   E[cost] = C_route + p * (q * C_small + (1 - q) * C_large)
#                    + (1 - p) * C_large
#
# 期望正确率：
#   E[acc]  = p * (q * s + (1 - q) * r)   # 判错时回退大模型，acc 接近 r
#            + (1 - p) * r
#
# 关键结论 1：误判代价是不对称的。
#   简单任务被判为复杂 -> 多花钱，但结果正确（只是浪费）
#   复杂任务被判为简单 -> 省钱，但结果错误（业务损失 + 可能引发重试，
#                                    重试会再次调用大模型，总成本可能反而更高）
#   所以路由器的评估指标应该是"漏判率"（把复杂判成简单的比例），
#   而不是"总体分类准确率"。
#
# 关键结论 2：当 q 很低时，路由几乎不省成本。
#   因为每次判错都要再走一遍大模型，等于做了两次大模型调用。
#   评估路由方案时，漏判率必须换算成"期望额外调用次数"来看。

Several practical judgments follow from this framework:

The router's miss rate is a hard metric that must be monitored separately with alerting. In the system it behaves more like a reliability component than an optimization component.
Design it as a cascade rather than a binary choice: let the small model answer first, use rules and confidence to decide whether to let the answer through, and fall back to the large model when uncertain. This turns the problem from "classification accuracy" into "confidence calibration," which is usually easier to get right and easier to tune by trading cost against quality via a threshold.
A conservative threshold is a reasonable starting point. Before cost pressure appears, the routing threshold should lean conservative (route less), because once you count the business loss caused by misjudgments, the payback bar is far higher than intuition suggests. Wait until the ledger has accumulated a few weeks of real data and the miss rate is under control, then loosen it gradually.

6.6 Batching and Async: The Preconditions for Trading Latency for Cost

The core value of a batch API is turning "low-latency single requests" into "high-latency bulk requests" in exchange for a lower unit cost. Its benefits come from sharing weights and KV space across the batch, reducing scheduling overhead, and occupying compute units more fully. This kind of optimization changes no model behavior whatsoever, so it is quality-neutral — a significant advantage relative to the other techniques.

The applicability boundary must be stated clearly:

It only applies to scenarios where the user isn't waiting: offline summarization, bulk classification, evaluation replay, overnight indexing, report generation. Interactive chat, real-time customer support, and any scenario where the user is watching the screen for a result are all out of scope.
It requires queue and backpressure design. Batch API quotas are usually separate from online quotas, so if you reroute online traffic to a batch queue, watch how quotas and latency commitments change. The queue needs explicit priorities, backlog alerting, and a drop-on-timeout policy — otherwise you get the situation where one stuck job drags down an entire batch.
Whether the benefit materializes depends on how well the batching window matches task arrival rate. If traffic is sparse, batches never fill up and the system degenerates into inefficient single requests. So the correct order is: look at traffic characteristics first (does it arrive in periodic bursts?), then decide whether to adopt batching — rather than switching over just because "batching is cheaper."

6.7 Reducing Turn Count: The Most Underrated Category of Savings

The previous sections all talked about "making each call cheaper," but the biggest lever is often on call count. A user task that takes five turns to complete, if you can finish it in two, cuts cost by more than half — and user satisfaction usually improves as well (less typing, less waiting).

Actionable directions include: collect all the information in one pass instead of asking turn by turn; infer user intent from conversation history instead of reconfirming every time; use structured forms or options instead of open-ended questions; change "confirm, then execute" into "execute, but allow undo"; and put an explicit constraint in the system prompt to "check that you have all necessary information before answering."

The metric that measures this lever is the distribution of turns per task, not the average. Turns are strongly power-law too: the vast majority of tasks finish in one or two turns while a few get stuck in long loops. Looking only at the average conceals two opposite problems at once — "most tasks are too verbose" and "a few tasks are spinning" — so you should look at quantiles, and separately monitor the count of long-tail loops along with their termination conditions.

Budgets, Rate Limiting, and Alerting

The ledger and unit economics make cost visible; budgets and alerting make it controllable. The goal of this layer is not to precisely control the spend of every individual request, but to cut it off in a controlled way before it runs away.

7.1 A Budget Is an Allocation Problem, Not a Total

Setting a "monthly cost ceiling" for an AI business is close to worthless, because nobody makes a better decision just because the total happens to be approaching the ceiling. A budget is an allocation mechanism, and there are three common axes:

Per tenant: each tenant gets an independent allowance or soft allowance. This suits usage-based billing scenarios, and it is the only way to keep a single tenant from dragging down the whole system. A hard allowance cuts service off with no warning to the customer, which makes for a poor experience; the better approach is a tiered policy: degrade at the soft threshold (switch to a cheaper model, shorten the context, turn off non-essential tools), throttle only at the hard threshold, and say so explicitly in the product.
Per feature: each feature has its own budget and unit cost ceiling. This suits products where features differ enormously in cost — a free trial conversation versus a paid deep report, for instance.
Per model or channel: this stops a single route from eating the entire budget under abnormal conditions. It is routinely overlooked, yet it is exactly where sudden incidents are most likely to happen.

The three can be stacked. In practice, converge on them in the order "channel → feature → tenant," narrowing from coarse to fine. That way, whenever any layer trips, you know who is spending and why they are spending.

7.2 Five Dimensions of Rate Limiting

Limiting RPM alone is not enough. At a minimum you need to cover:

Rate: requests per minute and tokens per minute (set separately for input and output, because their cost structures differ).
Concurrency: the number of sessions in flight at the same time. Concurrency is the coupled variable that determines both latency and cost, so capping it protects latency commitments better than capping the rate does.
Periodic quota: a per-tenant daily or monthly token ceiling. This is the main lever for holding monthly cost in check.
Per-call limits: a maximum input length for a single request, a cumulative token ceiling for a single session, and a maximum turn count per session. These three limits are what prevent a "single-point explosion," and they are the necessary defense line against resource-exhaustion attacks driven by prompt injection.
Priority: which traffic to preserve when the global budget runs tight. This requires the cost system to be wired into the scheduling system rather than each going its own way.

The degradation chain you design for when a limit trips matters more than the limit itself. The ideal order is: switch to a simplified prompt with a higher cache hit rate → switch to a smaller model → reduce the number of retrieved results → turn off non-essential tools → queue and wait rather than fail outright. Every level of that chain should have been validated for quality and latency by load testing in advance, not improvised on the day of the incident.

7.3 Alerting: Budget Burn Rate

Fixed-threshold alerting has two problems: set too low it is noise, set too high it arrives too late. Cost alerting should be built on the burn rate — the ratio of actual spend progress to time progress. This metric is independent of the absolute budget amount, so it adapts to different business scales:

def burn_rate(used: float, budget: float, elapsed_fraction: float) -> float:
    """烧穿速率 = 花费进度 / 时间进度。
    = 1.0  : 完全按计划花钱（正常）
    > 1.0 : 花得比计划快，1.5 表示按当前速度会在周期的 2/3 处花完全部预算
    < 1.0 : 花得比计划慢（业务量不足，或用量被优化掉了）
    """
    if elapsed_fraction <= 0:
        return float("inf")
    return (used / max(budget, 1e-9)) / elapsed_fraction


def alert_level(used: float, budget: float, elapsed: float) -> str:
    r = burn_rate(used, budget, elapsed)
    if r >= 2.0:
        return "P0: 已确定会超支，立即定位并处置"
    if r >= 1.5:
        return "P1: 高概率超支，当日内定位"
    if r >= 1.2:
        return "P2: 偏离计划，纳入观察"
    if r <= 0.5 and elapsed > 0.5:
        return "P3: 花得太慢，检查用量上报是否丢失或路由是否被误降级"
    return "ok"


# 告警必须按维度分别计算，而不是只看全局：
#   全局 burn rate 正常 + 某租户 burn rate = 5.0  →  必须告警
#   全局 burn rate 正常 + 某模型单位成本突增    →  必须告警
# 建议的最小告警集：
#   1) 周期总预算烧穿速率（P0/P1/P2）
#   2) 按租户的烧穿速率 top-N（单租户拖垮）
#   3) 单位成功任务成本的同比/环比突变（效率退化，与总量无关）
#   4) 成本放大系数（账单 token / 有效 token）异常（重试/思考膨胀）
#   5) 账本与账单的差额率（数据管道故障的早期信号）
#   6) 缓存 token 命中率的突降（配置或流量结构变化）

Item 5 deserves emphasis. A broken data pipeline disguises itself as a drop in cost: if the reporting path goes down, total cost collapses, no alert fires, and you only discover that a full month of data is missing when you reconcile at the end of the month. Treating the "ledger versus invoice" discrepancy rate as a first-class monitoring signal is one of the very few things in cost engineering that surfaces infrastructure failures early.

Item 3 matters just as much. A sudden shift in unit cost is an earlier signal than a sudden shift in the total, because the total is drowned in the enormous noise of business volume. A rise in total cost driven by volume growth is healthy; a rise driven by cost-efficiency degradation is dangerous. The two must be alerted on separately.

7.4 Where Hard Constraints Belong

A hard budget constraint must ultimately land in the gateway or the scheduling layer, not in a conditional check inside application code. The reason: application-level checks are unreliable under concurrency (several requests read "balance remaining" at the same instant and then all let themselves through), and they cannot cover calls between internal services. The recommended structure is this:

Check and reserve quota at the request entry point (optimistic deduction), settle against real usage when the cost event is recorded (correcting the estimation error), and reset and roll over at the period boundary. The two-phase reserve-then-settle design is necessary, because actual usage is unknown when the request is issued (streaming generation especially) and can only be estimated; but the estimate must be correctable after the fact, otherwise the ledger drifts away from the invoice for good.

Nine Common Traps in Cost Attribution

This is the most practically useful section of the article. All nine traps below show up in nearly every AI product running at scale, and most of them make cost analysis land on a directional error rather than a small deviation.

Trap 1: Retry Amplification Silently Swallowed

The most typical symptom is a perfectly smooth cost curve on the dashboard with no spikes at all, while the actual invoice comes in far above expectation. The cause is that retried requests land on a different service instance with no identifier linking them back to the original request, so the retry cost gets booked against a different time period, a different feature, or even a different tenant.

The defense: trace_id must be propagated across the entire retry chain, including retries inside the gateway, retries at the SDK layer, and client-side resends on timeout. attempt must increment strictly, starting at 1. The report needs a "retry cost share" metric, and that metric belongs on the first screen of the cost dashboard — it is a leading indicator that usually goes anomalous long before the bill runs over.

Trap 2: Silent Context Bloat

Every team knows that "longer context costs more," yet very few go on to measure what the context is actually made of. In a real system, the bloat usually comes from sources that are not obvious at all:

Tool definitions (function schemas) sit in the input on every turn, and the number of tools grows as features are added;
Several sets of retrieval results are injected at once and overlap each other heavily;
The few-shot examples in the system prompt are nowhere near as effectively used on real traffic as they are billed for on every turn;
The JSON Schema for structured output inflates as fields are added;
On a failed retry, the complete previous output is carried into the context as well;
In multi-agent setups, a sub-agent's full output is passed verbatim to the main agent, amplifying at every layer.

The defense is to treat context as an object that needs budget management: during development, use tooling to measure the distribution of input tokens for each class of task and set ceilings on business quantiles (not averages); after launch, keep watching the quantile trend of input tokens instead of just the mean. The long tail of the input token distribution predicts cost incidents far better than the mean does.

Trap 3: Evaluation Traffic Mixed into Production Stats

Offline evaluation, A/B experiments, regression test-set replays, human annotation — the model configurations, prompt templates, and parameters behind this traffic may be entirely different from production, yet it draws down the same vendor account. The subtler case is internal staff dogfooding features in the production environment: that traffic neither generates revenue nor counts as evaluation, but in the ledger it is simply "prod."

The defense: the env label must be injected uniformly at the gateway layer, not left to callers to pass voluntarily. You also need to run a periodic "ledger versus invoice" discrepancy attribution and calibrate a trustworthy ratio for explaining that difference. The test is simple: if a stable gap between your production cost and the vendor invoice cannot be explained with two or three queries, some traffic is unlabeled.

Trap 4: Interrupted Streams Billed in Full

A user hits send, waits two seconds, decides it feels slow, and closes the page. The server may still be generating, or it may have produced a large number of tokens before the disconnect was detected. Those tokens are billed as usual, yet in many implementations they are never recorded because the response did not terminate normally. The result is a ledger that persistently understates production cost — and the understatement correlates with how dissatisfied the user was: the most expensive, worst-experience traffic is precisely the easiest to miss.

The defense: cost events must be recorded in the response stream's cancel callback, not only on successful completion. The portion already generated is settled at the actual token count. That metric deserves monitoring in its own right — a sudden rise in "average tokens generated before interruption" is often an early signal of a degrading latency experience or a model turning more verbose.

Trap 5: The Cache Hit Rate Has the Wrong Denominator

When the hit rate is computed as cached_prompt_tokens / prompt_tokens, requests with no cacheable prefix at all (short requests, dynamic prefixes) still enter the denominator but can never contribute a hit, pulling the overall rate down. That leads you to wrongly conclude the cache has failed, when the only real problem is that a batch of requests is inherently uncacheable.

The correct approach is for the denominator to include only requests that are structurally cacheable, and to track the share of inherently uncacheable traffic as a separate metric in its own right. The two numbers have to be shown side by side, otherwise your judgment on cache optimization stays persistently skewed by a structural bias.

Trap 6: Long Context That Looks Cheaper and Costs More

A counterintuitive but common phenomenon: after enabling a long context to avoid "having to re-ask once the input is truncated," total cost goes up instead. The reason is that users no longer need several rounds of clarification and the one-shot success rate rises sharply, which looks like an efficiency gain; but the per-call cost of a long context is nonlinear, while the old "ask a few more rounds" path only pays the extra cost when it fails. Worse, in many scenarios most of the documents placed into the long context are never actually used — you simply paid for them.

The defense: run a separate unit economics analysis for long-context features, and use "information utilization" as a supporting metric — measure what share of the injected context actually influenced the final answer. Context with persistently low utilization should switch to on-demand retrieval rather than unconditional full injection.

Trap 7: Misallocation in A/B Experiments

Cost comparison during an experiment is where teams most easily reach the wrong conclusion. Common problems: the traffic mix between treatment and control is not the one the design assumed; experiment traffic concentrates on a certain kind of user (usually the ones more interested in the feature); the experiment period happens to overlap an upstream price change or a shift in traffic mix; and the two arms have naturally different cache hit rates (the new approach may share part of its cache with the old one, so costs end up wrongly diluted).

The defense: cost analysis during an experiment must use unit cost rather than totals, must correct for the traffic mix, must treat cache hit status as a stratification dimension, and must keep in mind that experiment traffic is usually unrepresentative — its behavior on long-tail users is often the opposite of production.

Trap 8: Timezone and Billing-Period Misalignment

This one is purely technical but has severe consequences. The billing system cuts periods by UTC, your business system cuts them by local timezone, and a tenant's billing cycle is cut by contract signing date. When you ask "which tenant had the highest inference cost last month," you are in fact intersecting three different time windows. If you do not notice that, you get a ranking that is entirely wrong, and then make pricing or packaging decisions on top of it.

The defense: every cost report should state its time window, timezone, and cutting basis explicitly; tenant-level reports should aggregate by session completion time and break the unbilled portion out as a separate line.

Trap 9: Treating a Drop in Cost as Optimization Success

A drop in cost has four possible causes: the usage mix changed, business volume fell, traffic got routed onto degraded paths, or the data reporting broke. Only the first one, combined with stable unit quality metrics, counts as optimization success. This is the trap most likely to make you let your guard down, because it looks like good news.

The defense: any conclusion about cost improvement has to satisfy three conditions at once — cost per successful task goes down, the success rate does not drop, and the ledger-versus-invoice discrepancy rate stays stable. Miss any one of them and you mark the conclusion "to be verified" beside it, rather than writing it into the business review.

Rollout Path and Organizational Ownership

Cost engineering does not have to be finished all at once. What follows is a path ordered by dependency, in which each step takes the previous step's output as its input. Before laying out the three phases, one principle is worth stating up front: every step must leave behind a deliverable artifact, not just a configuration change. A deliverable might be a reconciliation script that actually runs, a dashboard with fixed definitions, or a number solid enough to support a pricing discussion. Artifacts are the only way to build trust across teams — without them, the platform team says "cost is fine" and the business team says "cost is way too high," and both sides can only argue from impressions. Argument is the single most common reason cost engineering stalls.

A second principle is to avoid committing to an optimization target before the ledger exists. Plenty of teams set a quarterly "cut cost by X percent" goal, which is nearly guaranteed to fail while the ledger is missing: on one hand the team does not know the baseline, and on the other even hitting the target cannot prove the optimization deserves the credit (the volume mix may have shifted). Get the measurement solid first; targets can wait.

9.1 Three Phases

Phase one: make cost visible first. The goal is to reach "the ledger reconciles" within two weeks. Concrete actions: inject trace, tenant, feature, and environment labels uniformly at the gateway layer; build the cost event table and guarantee idempotent writes; use a price snapshot field so history stays immutable; implement daily "ledger versus invoice" reconciliation and surface the difference explicitly. Do no optimization at all in this phase — make the foundation solid first. The acceptance criterion is: the ledger-versus-invoice gap on any given day can be explained and can be drilled down along three dimensions — feature, tenant, and model.

Phase two: make unit economics hold up. The goal is to be able to answer "which features and which tenants are making money." Concrete actions: unify the definition around "cost per successful task"; produce gross margin reports stratified by tenant and by feature; establish a definition of "success" agreed with the business side; identify the long tail of cost outliers (high-cost tenants, long-context tasks, high-turn sessions). The output of this phase usually leaves product and business with a completely different view of cost — generally less optimistic than the engineering team's own estimate, which is itself the value. The acceptance criterion is: you can make a data-backed decision in a meeting about which feature to put under a limit.

Phase three: make cost operable. The goal is to have budgets, rate limiting, alerting, and the degradation chain all in place. Start with soft alerts for budgets (notify but do not block), run two periods to confirm the thresholds are sensible, and only then turn on hard constraints. The key discipline in this phase is: shipping a hard constraint requires a degradation chain alongside it, and that chain must already have been validated under load testing — otherwise rate limiting simply drives availability down, trading cost for lost revenue. The acceptance criterion is: you can run a full cost incident drill, from detection through degradation to postmortem.

9.2 Division of Labor: The Platform Owns Definitions, Product Owns Structure

Cost rarely runs out of control because of insufficient technical capability; far more often it is because nobody owns unit economics. A sensible division of responsibility looks like this:

The platform / infrastructure team owns the ledger, the billing definitions, unit prices, and the rate limiting and alerting mechanisms. The job is to make cost attributable and constrainable, not to own the business cost structure.
The business / product team owns unit economics and the usage mix. The job is to make cost worth it — bringing cost down to a competitive level through product design, packaging, and prompt and interaction design.
Both sides jointly own the quality floor of the degradation chain. Degradation costs quality, and that trade-off cannot be decided unilaterally by either party.

There is a practical consideration behind this split: the platform team has no incentive to guess which feature should not be so expensive, and the product team has no ability to change unit prices or compute allocation. What cost engineering actually does is give these two sides a ruler they can both hold.

9.3 The Anti-Pattern Checklist

To close, here are a few practices that should clearly be avoided, all of which show up very frequently in practice:

Optimize before you measure. Optimizing without a ledger makes it impossible to prove the benefit afterwards, and the team gradually loses trust in cost engineering.
Optimize input but ignore call count. The lever in call count is usually larger than the lever in per-call cost, and ignoring it is the biggest structural waste.
Treat cost as a property of the model. Cost is a property of the system — with the same large model, different prompt structures, routing policies, and session designs can produce very different costs.
Use aggregate metrics to judge efficiency. Volume growth drowns out efficiency signals, so every efficiency conclusion needs a normalized metric.
Ship a hard budget with no degradation chain. It will take your revenue down with it at the first traffic peak.
Make the business side fill in labels by hand. Labels must be injected by the system at a trusted point, or their data quality cannot support any decision.

The mark of maturity in cost engineering is not "the bill got smaller," but that cost has become an everyday operational metric sitting alongside latency and availability: it has a definition, instrumentation, a dashboard, alerts, an owner, a budget constraint, and a degradation plan. At that point cost is no longer bad news at the end of the month; it is a number you see every morning when you open the dashboard — and only then does cost reduction happen on its own, without a series of one-off optimization campaigns.