Agent Reliability Engineering: Timeouts, Retries, Idempotency, and Fault Isolation

Treat the Agent as a Distributed System: Reliability Problems Do Not Live in the Model

The first thing most teams do when they start hardening an Agent for stability is prompt tuning and swapping out models: turn the temperature down a notch, move to a model that follows instructions better, add a few few-shot examples. Those moves improve quality, but they do almost nothing for reliability, because Agents almost never crash from a lack of model intelligence. Nearly all of it comes from engineering gaps that sit outside the model.

If you lay a single Agent run out flat, this is what you get: the model generates an unpredictable control flow, that control flow nests real calls into external systems, the calls mutate state, the state feeds back into the control flow, and finally the whole chain of results gets handed to a human to read. Every link in that chain can fail, and the failure modes amplify one another. It looks like a distributed system, but it is harder to make fault-tolerant than most distributed systems, for three structural reasons.

First, the control flow has no static upper bound. In an ordinary service the retry boundary is determined by the call graph — how many layers, how many attempts each — and it is written down in code. An Agent's step count, which tool each step calls, and what arguments it passes are all decided by the model at runtime. There is no retry graph at compile time. You cannot compute a timeout budget or configure a circuit breaker for a graph that does not exist yet.

Second, every call is nested several layers deep. Under a single tool call there are still the orchestration layer, the gateway layer, downstream services, and the database. If every layer gets a fixed timeout, the timeout budget inflates multiplicatively with the nesting depth. A twenty-layer path with a three-second timeout can hang at the outermost layer for seventy-five minutes before it gives up, while your session timeout may only be one minute.

Third, reasoning and side effects are coupled into the same trace. In an ordinary service, if reasoning fails you retry reasoning; if the database write fails you retry the write. The two paths are cleanly separated. In an Agent, a single retry may re-run the reasoning, or it may re-trigger a write that already took effect. That is the root of the idempotency problem, and the reason the second half of this article spends so much space on it.

So this article is organized by splitting reliability into three layers. The control layer governs the loop and the state, covering stuck detection, step budgets, and circuit breaking. The data layer governs context and deadlines, covering how the timeout budget is allocated and inherited. The effect layer governs the outside world, covering idempotency, retries, compensating transactions, and rate limiting. Failures in the three layers infect each other, and in the end chaos testing validates all three together.

The Six Failure Modes Specific to Agents

Before writing any mechanism, spell out the disease you are trying to cure. The six below are the highest-frequency failures in an Agent setting, and every one of them has a counterpart in an ordinary microservice — where it shows up in a far more extreme form.

One: Missing Timeouts — A Long Transaction With No Upper Bound

The typical symptom is a tool call with no timeout parameter that waits by default until TCP-level retransmissions are exhausted; or a timeout that is written down but never passed through the middle layer between the orchestration layer and the tool adapter, so it ends up using some looser framework default. What the user sees is a button spinning for forty minutes, followed by an empty response.

The subtler version configures the overall timeout but not the idle timeout. In a streaming response the peer establishes the connection and then sends not a single byte, so the server just hangs there; or the agent emits a heartbeat every five seconds while the tool died long ago but the connection is still alive. Time-to-first-byte and inter-chunk interval timeouts have to be configured separately, and this is a rule that is still routinely missed in MCP and in every flavor of Agent tool protocol.

Two: Non-Idempotent Retries — One Timeout, Three Emails

This is the most expensive class. The tool already succeeded; only the response was lost on the way back, the client cannot see it, so it retries, and now the email send, the charge, the order placement, and the release each happen twice. Plenty of teams swap in a test stub that makes every tool return a fixed value before launch. Success rates look great, and then the first week in production brings a support ticket about double charges.

The subtler variant is model-level self-healing: stuff the raw tool exception back into the context and let the model decide to call again. Those retries bypass the retry policy you wrote into the framework entirely. You will not see retry counts at the gateway; you will only see inference spend quietly double.

Three: Partial Failure With Nowhere to Go — Three Succeeded, the Fourth Timed Out

The Agent plans five steps; the first three writes succeed and the fourth times out. The system now faces a question it has to answer immediately: do those three already-effective records stay, or do they get undone? A lot of implementations pretend nothing happened and throw the exception back out to wait for the user to retry — and the user, on retry, re-executes the first three steps that already succeeded.

Four: Loops That Never Converge — It Looks Like Thinking, but It Is Going in Circles

Model reasoning is stochastic, and the same state can generate thousands of differently worded useless actions. You cannot catch these loops by "detecting repeated strings," because every step's text is different. What you need to detect is semantic lack of progress: the state summary has not changed, the goal has not moved any closer, only the phrasing has.

Five: Fan-Out Overload — One Plan, Thirty-Six Calls

The model sees the sentence "pull every channel's data for this quarter" and plans thirty sub-calls. If the orchestration layer dutifully runs them concurrently, the same downstream service instantly receives thirty times the traffic, tripping rate limits and possibly triggering an avalanche. Per-tool rate limiting is completely ineffective here, because from that service's point of view every sub-call is a different request.

Six: Degradation That Does Not Propagate — Upstream Is Already Busy, Downstream Keeps Pulling Hard

The model gateway returns 429 or 503, the orchestration layer does not parse those status codes, treats them uniformly as tool errors, hands them to the model, and the model rephrases and tries again, punching through the quota several times in a row. The signature of this failure is that cost rises faster than errors. You will find it on the bill first, and only later track down the cause on the dashboard.

Timeout Budget: Allocated as Context, Not Passed Down Layer by Layer

The first course correction about timeouts is this: what you manage is a deadline, not a timeout length. A timeout is a length; a deadline is an absolute moment. When a call chain has three layers and each layer may retry, passing the timeout down layer by layer lets the budget run wild, because no layer knows how much the layer above it has left.

The difference comes down to one sentence: a timeout answers "how long can this wait at most," a deadline answers "by what time must this be finished at the latest." Only the latter decays correctly through nested calls.

The Three Shapes of Budget Allocation

In practice there are only three shapes of allocation. Sort out which one you have before talking about inheritance.

Serial chains: several calls execute in sequence and share a single wall-clock budget. The way to divide it is to roll it forward based on what's left, not to slice it up front — because an early stage may finish early, and the time it saved should go to the stages after it.

Parallel fan-out: N branches run at once and must divide up the remaining budget. Note the word "divide": giving each branch its own copy instead makes the parallel total equal the parent budget times the branch count, which is the single most common source of timeout explosion.

Nested Agents: a child Agent inherits the parent's remaining allowance, usually multiplied by a ratio and capped at an absolute ceiling. The ratio stops one expensive subtask from eating the whole budget; the cap stops long-tail requests from being amplified as they propagate up through the layers.

Coupling the Budget to Model Inference

Half of the timeout story gets ignored: after a tool call fails, the model has to run inference again to decide what to do, and that inference costs time and tokens too. If the budget covers the tool but not the inference, you run out of allowance at exactly the moment a trade-off has to be made. The outcome is either a brutal cut-off that emits "unable to complete due to timeout," or a quiet extra inference allowed through with no allowance left, so the actual wall-clock time exceeds expectations. Neither is a good outcome: the first throws away tool work that already succeeded, the second makes the budget meaningless. So the budget should cover at least three spans — one model inference, one tool round trip, and the re-decision inference after a failure.

The corollary is that inference itself needs a ceiling too. Once the remaining budget is smaller than the typical duration of a single inference, letting the model keep thinking is pure waste; the better move is a degraded path with minimal output, where the model organizes what it already knows into an answer using the fewest possible tokens, or hands off to a deterministic template entirely. This "not enough budget, so degrade to minimum output" behavior shows up far more often at the tail of long tasks than people expect, and it deserves its own design rather than being treated as a catch-all.

There is also a positive use of the budget that tends to get ignored: let the budget constrain planning in the other direction. If the planner sees the remaining allowance while it is generating the action list, it will tend to "plan three steps because three steps fit" rather than plan thirty and get cut off midway. The latter leaves a pile of side effects that took effect but were never completed; the former never produces them in the first place. That is far cleaner than intercepting at the execution layer, and the cost is having to express the budget clearly in the prompt or the tool description, and then confirming that the model actually reads it.

Budget Inheritance for Child Agents

Once you bring in child Agents or parallel expert models, the budget becomes a resource that crosses processes and even vendors. Three rules are workable. First, a child Agent inherits the parent's remaining allowance multiplied by a ratio, with an absolute cap; the parent's total share to children must be reclaimable by the parent, otherwise the number of parallel branches — which the model decides — lets the total budget run away. Second, whatever a child consumes must be written back, accounted through a token or ledger threaded through the entire call tree, so parent and child do not both believe there is money left. Third, a child Agent's irreversible side effects must propagate back into the parent's ledger, otherwise the recovery logic only sees the parent-side records and misses the messages the child already sent.

The third rule is the one most often overlooked. A child Agent usually has its own execution context and its own persistence, and a parent doing crash recovery generally loads only its own state. Without a shared ledger between them, recovery opens a hole of the form "the parent thinks it never happened, the child Agent actually did it," and the only option left is to guess. Worse, this class of problem never surfaces at runtime — it fails intermittently on the recovery path and is extremely hard to locate. A shared ledger is not expensive; usually it just means writing one extra record with a parent-child link when you write a side effect.

A Side-by-Side Look at the Three Shapes

Shape	How it is allocated	The two most common mistakes	Consequence
Serial chain	Rolled forward from the remaining amount; a stage that finishes early leaves more for later stages	Slicing fixed shares up front	A fast early stage wastes allowance; a slow one runs dry midway
Parallel fan-out	Divides the remaining allowance, allocated by weight	Giving each branch a full copy of the parent budget	Total budget inflates with the branch count and the whole thing jams against rate limits
Nested Agent	Remaining amount times a ratio with an absolute cap, consumption written back	Inheriting without accounting	Both parent and child think allowance remains, and the actual total far exceeds expectations

The two mistakes in the parallel row of that table point in opposite directions but land in the same place: one inflates the total budget, the other exhausts the allowance on the first attempt so that the compensation stage has nothing to spend. The first amplifies, the second depletes, and the common outcome is a run that never finished plus a pile of dangling side effects. So allowance for parallel branches has to be thought about together with the compensation budget, not split into two pieces on the assumption that they do not interfere.

Three Rules That Must Be Hard-Coded

1. Budgets cannot be reset. A retry must reuse the same deadline moment; it must never restart the clock. This is the first gate that pulls retry from "amplifier" back to "recovery mechanism."
2. If the budget is insufficient, do not start. Launching a call that is destined to time out wastes a connection, occupies downstream quota, and may leave half a side effect behind. Check before the call, not after the timeout.
3. Remaining allowance must propagate across processes. Tools are often independently deployed HTTP services, so the remaining milliseconds have to go into a request header, and downstream services must be allowed to write consumption back. Otherwise the budget on the far side of a service boundary is a pure guess.

Implementation: The Budget as a Context Object

# 预算不是参数，是上下文：一次 run 只有一个墙钟截止时刻，
# 所有下游调用从它继承剩余额度，而不是各自拿一份新的 timeout。

from __future__ import annotations

import time
from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass, field

_budget: ContextVar['Budget | None'] = ContextVar('budget', default=None)


class BudgetExceeded(Exception):
    pass


@dataclass
class Budget:
    total: float                      # 本次 run 的墙钟预算（秒）
    started: float = field(default_factory=time.monotonic)
    name: str = 'root'

    @property
    def remaining(self) -> float:
        return max(0.0, self.total - (time.monotonic() - self.started))

    @property
    def expired(self) -> bool:
        return self.remaining <= 0.0

    def check(self, need: float = 0.0) -> None:
        # need 是这次调用预计要花的时长。剩余量大于 0 但不够跑完时
        # 就该放弃，而不是发出去再等它超时。
        if self.expired or self.remaining < need:
            raise BudgetExceeded(
                f'budget {self.name} exhausted: '
                f'need={need:.2f}s remain={self.remaining:.2f}s'
            )

    def child(self, name: str, weight: float = 1.0, cap: float | None = None) ->Budget':
        # 并行分支按权重瓜分「剩余」额度；不给每个分支一份完整副本，
        # 否则 N 路并行总预算就变成了 N 倍父预算。
        share = self.remaining * weight
        if cap is not None:
            share = min(share, cap)
        return Budget(total=share, name=f'{self.name}/{name}')


DEADLINE_HEADER = 'x-agent-deadline-ms'


def outbound_headers() -> dict:
    b = _budget.get()
    if b is None:
        raise RuntimeError('no budget in context: run must enter run_with_budget()')
    return {DEADLINE_HEADER: str(int(b.remaining * 1000))}


def adopt_downstream(headers: dict) ->Budget':
    # 工具在另一个进程里：从请求头恢复自己的预算，继续向上衰减
    left_ms = int(headers.get(DEADLINE_HEADER, '0'))
    return Budget(total=left_ms / 1000.0, name='downstream')


@contextmanager
def run_with_budget(total: float, name: str = 'root'):
    token = _budget.set(Budget(total=total, name=name))
    try:
        yield _budget.get()
    finally:
        _budget.reset(token)


def call_tool(fn, spec, *args, **kw):
    # spec 是工具声明的延迟画像（p50/p99），用来决定给它多少额度
    b = _budget.get()
    allowance = min(spec.p99 * 1.5, b.remaining)
    b.check(need=allowance)
    return fn(*args, deadline=time.monotonic() + allowance, **kw)

The easiest thing to overlook in that implementation is the early check inside check(need=...). There is real craft behind using p99 times a factor as the per-call allowance ceiling: a p99 too close to the mean makes long tails exhaust the budget constantly, and one too loose lets a single slow tool eat the whole run's time. Most teams start at p99 times 1.5 to 2 and then tune it against the production distribution.

One more detail is worth its own paragraph: a tool declaring a latency profile (p50/p99) only means anything when the profile comes from a histogram you observed yourself. Using a made-up constant as p99 means building budget decisions on a guess that has never been validated. The approach is crude but effective: each tool records its percentile latency over the last hour and refreshes its profile periodically, and any tool that has not cleared the sample-size threshold skips budget allocation entirely and falls back to a conservative fixed value.

Retries Presuppose Idempotency

One sentence summarizes this section: a retry is not a fault-tolerance strategy; a retry is a guess at a side effect whose effect you cannot confirm. When the side effect is idempotent, guessing wrong costs nothing. When it is not, guessing wrong is data corruption. So the correct order for a retry policy is: classify the tools first, then decide which ones are safe to retry, and only then talk about backoff algorithms.

Splitting Tools into Three Classes by Retryability

The classification criterion is not "read or write" but "does repeating it change the semantics." Applying that criterion to three classes turns out to be clearer than intuition suggests.

Class	Criterion	Typical tools	Automatic retry
Naturally idempotent	Repeating produces the same result, or it is read-only in semantics	Search, query, read file, list directory, translate, summarize	Allowed, up to a bounded number of retries
Writes with a natural idempotency key	The server de-duplicates on some business key, collapsing duplicate writes	UPSERT, persisting by message ID, booking against an external order number	Allowed, provided the idempotency key really does reach the server
Irreversible	The side effect is already externally visible and cannot be de-duplicated by a business key	Send email, send a message, take a payment, spawn a subprocess, ship a release	Automatic retry forbidden; compensation only

The practical value of this table is that it turns "should this tool be retried?" from an open question that has to be revisited every time into an attribute you can read straight off the tool registry. The class and the construction of the idempotency key need to be declared when the tool is registered; by the time you discover in production that some tool is not retryable, it is almost always too late.

LLM inference calls deserve their own note. They look naturally idempotent — same input, same output? Not so. An inference retry produces no external side effect, but it burns money and it keeps alive a branch that should perhaps have been abandoned. More importantly, when inference and tool calls live in the same loop, the number of inference retries counts toward the model's self-correction budget, and that count needs a ceiling or you land squarely in the loop-breaker topic of the next section.

Idempotency Keys Must Bind to Intent

How the idempotency key is constructed determines whether the whole mechanism holds up. The most common mistake is binding the key to the attempt: generate a fresh UUID on every retry, the server sees a new key, and it accepts everything. That is the same as having no idempotency at all. The correct binding target is the logical intent: every attempt, every retry, and every cross-process replay of the same intent shares one key.

The key should be built from four parts: tenant identifier, intent identifier (generated and persisted by the orchestration layer at planning time), tool name, and a normalized digest of the arguments. The normalization step is frequently skipped, so the model passes {'q': '天气'} on the first attempt and {'q': '天气 '} on the retry, the two keys differ, and de-duplication silently fails. Normalization has to cover the usual differences: whitespace, letter case, dictionary order, timestamp placeholders, and random ids.

There are two more easily missed prerequisites. The server must genuinely persist the key-to-result mapping rather than merely glancing at a cache, and the lifetime of that store must cover the maximum retry window plus the process-restart recovery window. A lot of implementations set the TTL to one hour, and then a cross-day breakpoint recovery replay punches straight through de-duplication.

Inference Retries and Tool Retries Must Be Layered Separately

These two kinds of retry often get folded into a single policy, and the result is that both the budget and the counters become meaningless. A tool retry triggers on a clear transport or server error and re-executes the same call; an inference retry triggers when the output fails a format or validation check and re-samples the model. Their budgets, attempt ceilings, and observability metrics should all be counted separately. Merge them and you can see neither downstream jitter nor model-quality drift, leaving you with nothing but a vague "today felt unstable."

More critical is that you must forbid unbounded model-initiated self-healing. Stuffing the raw tool exception back into the context and letting the model decide to retry feels natural in terms of experience, but the price is total loss of control: no ceiling, no budget, no backoff, and therefore no circuit breaker to work with. A workable compromise is to declare a "model self-heal allowance" per tool, defaulting to one or two, and to mark the tool unavailable for the rest of the run once that is exceeded, forcing the model onto a degraded path instead of letting it keep probing.

This self-heal allowance complements the circuit breaker rather than duplicating it. The circuit breaker answers "is this tool broken overall," based on statistics across runs; the self-heal allowance answers "is this particular call worth another try," based on step count within a single run. Mixing them produces two classic errors: when the breaker is too loose the self-heal allowance becomes the only line of defense and the model hammers the same broken tool; when the breaker is too tight, a little jitter trips it and users frequently see "temporarily unavailable" even though the task would have finished moments later.

Respect the Server-Side Rate Limit Signals

When a downstream returns an explicit rate-limit or overload signal, the retry policy has to defer to the server's instruction. The most basic rule is to adopt the wait time given in the response rather than applying your own backoff formula — the server's suggestion comes from its real queue depth, which local backoff knows nothing about. A home-grown backoff is only a fallback for when the response carries no signal at all, and even then it should be more conservative rather than more aggressive, because "no signal" often means the other side is already too busy to produce one.

One level up is distributed rate limit coordination. When several Agent instances get throttled by the same upstream, everyone backing off independently still produces synchronized spikes, because their backoff clocks all start at the moment the same upstream returned. There are two workable approaches: briefly cache the upstream's throttling state in shared storage so all instances share one cooldown window; or introduce a long-tailed jittered distribution of retry times so requests spread across the whole window instead of clustering at its front edge. Neither is complicated, but without them a retry budget can only constrain a single instance's total, not the cluster peak.

There is also a model-side signal worth noting: an overload error returned by an inference request and an overload error returned by a tool mean completely different things to the user. Tool overload usually still has a degraded path (use cache, switch data source, narrow the scope), while model overload often means the whole task cannot be done. If both are classified as "retryable," you get a bad combination: during a tool overload the model retries frantically, burns through inference quota that could have held, and then nothing gets through.

Make Retries Visible to the User

A call that takes twenty seconds to succeed feels to the user like "it's hung." Reliability buys a purely experiential gain here: if retries carry explicit progress feedback — which attempt this is, why it is retrying, how much longer — the perceived cost of those twenty seconds drops significantly. The implementation cost is trivial: put the reason and the estimated time into the retry event. No interaction redesign required.

Conversely, if a retry is destined to fail anyway, failing fast and saying so clearly is friendlier than burning through every retry. The test is whether the remaining budget can still cover at least one success: if it can, retry; if not, return a structured failure directly. This rule aligns the retry budget with the user's waiting time and avoids the worst combination — neither a success nor a wasted thirty seconds of the user's patience.

Backoff and the Retry Budget

Exponential backoff plus jitter is common knowledge, but what really decides whether a storm happens is the concept of the retry budget: retry requests must be capped at a fraction of total requests, usually within twenty percent. Once you exceed the budget, fail fast and let the caller degrade instead of continuing to add pressure.

Jitter has to be full jitter, meaning a uniform sample over [0, upper bound] rather than a small perturbation around that bound. The reason is direct: exponential backoff with a small perturbation still synchronizes at scale, because every client's backoff curve has the same shape. Only spreading the delay randomly across the entire interval flattens the spikes.

Implementation: Classification, Retry Budget, and Full Jitter

# 重试的前提是幂等：先给工具分级，再决定退避与预算。
# 幂等键固定在意图上，而不是尝试上 —— 这是整套机制成立的前提。

import hashlib
import json
import random
import threading
import time
from dataclasses import dataclass

READ, IDEM_WRITE, IRREVERSIBLE = 'read', 'idempotent_write', 'irreversible'


@dataclass(frozen=True)
class ToolSpec:
    name: str
    kind: str
    p50: float
    p99: float


def classify(exc: Exception) -> bool:
    # 5xx / 429 / 超时可重试；4xx 业务错与参数错重试一万次也是错
    code = getattr(exc, 'status_code', None)
    return code is None or code == 429 or code >= 500


class RetryBudget:
    '总请求数里只有一小部分允许是重试，超出就快速失败。'

    def __init__(self, ratio: float = 0.2):
        self.rate = ratio
        self.tokens, self.last = 10.0, time.monotonic()
        self.lock = threading.Lock()

    def allow(self) -> bool:
        with self.lock:
            now = time.monotonic()
            self.tokens = min(20.0, self.tokens + (now - self.last) * self.rate * 60)
            self.last = now
            if self.tokens < 1.0:
                return False
            self.tokens -= 1.0
            return True


def normalize(args: dict) -> dict:
    # 剥离空白与不稳定字段，否则同一意图会算出不同的键
    return {k: (v.strip() if isinstance(v, str) else v)
            for k, v in sorted(args.items())
            if not k.startswith('_') and k not in ('trace_id', 'request_id')}


def idempotency_key(spec: ToolSpec, args: dict, intent_id: str, tenant: str):
    if spec.kind == IRREVERSIBLE:
        return None                     # 不可逆：不重试，交给补偿事务
    payload = json.dumps(normalize(args), sort_keys=True, ensure_ascii=False, default=str)
    raw = f'{tenant}|{intent_id}|{spec.name}|{payload}'.encode('utf-8')
    return hashlib.sha256(raw).hexdigest()[:32]


def backoff_delay(attempt: int, rng: random.Random) -> float:
    # 全抖动：把尖峰摊到整个区间，而不是在上界附近小幅扰动
    return rng.uniform(0.0, min(8.0, 0.25 * (2 ** attempt)))


def call_with_retry(spec, args, intent_id, tenant, invoke, budget, deadline, rng=None):
    rng = rng or random.Random()
    key = idempotency_key(spec, args, intent_id, tenant)
    for attempt in range(4):
        try:
            return invoke(args, key=key)
        except Exception as exc:
            left = deadline - time.monotonic()
            if not classify(exc):                   # 4xx 业务错不重试
                raise
            # 先判次数上限，再判幂等键，让「不幂等绝不重试」成为短路规则
            if attempt >= 3 or (key is None and spec.kind != READ):
                raise
            if left <= spec.p50 or not budget.allow():
                raise                               # 预算不足 -> 快速失败
            time.sleep(min(backoff_delay(attempt, rng), max(0.0, left - spec.p50)))

The ordering inside that line if attempt >= 3 or key is None and spec.kind != READ is worth expanding on. Checking the attempt ceiling before the idempotency key exists so that the rule "never retry when not idempotent" becomes an unbypassable short-circuit at the code level, rather than one option scattered across several conditions. Once such a rule is written as scattered conditions, some half-year refactor is guaranteed to miss one of them.

Also notice that the backoff sleep is clamped with min to stay inside the remaining budget. There is a counter-intuitive conclusion here: backoff time spends budget just like call time does. A lot of implementations only count the tool's p99 in the budget and forget the backoff, so an actual retry chain consumes three p99s plus two backoff intervals and blows the budget on the spot.

There is one more easily missed cost dimension: prefix caching. When a model call is retried, if the retried prompt prefix is byte-identical to the previous one, the KV cache may hit and the marginal cost of the retry is very low. That property makes a "retry storm" look far less alarming on the bill, which in turn makes it harder to discover. So cost alerting cannot look at the total alone — it has to look at the average number of inferences per run.

External Side Effects and Compensating Transactions

Retries do not solve the problem of irreversible tools. An email that has gone out cannot be recalled, and money that has been charged cannot be un-charged. There are only two things you can do: reduce irreversible actions beforehand, and afterward push the state back into an acceptable range with a new action. The latter is the compensating transaction, and it is the only tool you have.

Do the Subtraction First: Ordering Irreversible Actions

Before introducing any compensating mechanism, do the cheapest and highest-payoff thing first: schedule irreversible actions as late in the orchestration as possible. If a task needs to send three confirmation emails and write one database record, the order should be "write the record, validate, prepare the email content, and only on the last step send the mail." Then a failure at any earlier step leaves nothing to compensate.

This principle can land directly in the planner's prompt constraints, and equally in the orchestration framework's dependency declarations. A step further is to tag tools with a reversibility level so that the planner places high-irreversibility actions at the top of the dependency graph when it picks an order. There is one counter-example to watch for specifically: letting the model send the email and take the payment in parallel, because "they both have to happen anyway." The compensation order of parallel branches is close to unsolvable, and they have to be serialized first.

Compensating Actions Come in Strengths

Treating compensation generically as "the inverse operation" leads to over-optimism. Compensations vary enormously in reliability, spanning at least three grades, and handling them all the same way leaves you lost the moment a compensation fails.

Grade	Characteristics	How to handle it	Typical examples
Strong compensation	Precisely reversible and naturally idempotent	Run automatically; on failure it can be retried safely until the budget is exhausted	Cancelling a not-yet-consumed task, deleting a draft you created
Weak compensation	Can only push the state toward, not back to, consistency	Run automatically once; on failure hand off to a human, no infinite retries	Marking an order as awaiting manual confirmation, sending a correction notice
No compensation	Externally irreversible with no substitute action	Mostly avoided before the run; at runtime only alerting and reporting	Payment already taken and the reconciliation window has closed

Grading buys two direct benefits. First, failure handling for automatic compensation finally has a basis: a strong compensation can be retried with confidence, because repeating it costs nothing; a weak one only needs one retry, because repeating it may itself be a second injury — sending a duplicate correction notice is worse than not compensating at all. Second, actions with no compensation can be avoided up front during planning, instead of discovering at runtime that there is no way back.

There is also a practical trap here: the time to take effect of a compensating action matters more than the action itself. Deleting a record may take effect immediately, while a correction email has to wait for the recipient to read it, and the two mean very different things inside a consistency window. To judge whether a compensation is sufficient, look at how long it takes to pull the externally visible state back into an acceptable range — not at whether it succeeds inside your own system.

Compensation Is Not Rollback

A database rollback restores state to some historical point; compensation is executing a new action forward so the system reaches a business-acceptable neighboring state. The two differ a great deal, and the difference has to be made explicit at design time.

The difference shows up in three places. First, visibility: a rollback is invisible inside the database, while a compensating action is externally visible and may itself fail. Second, reversibility is not necessarily symmetric: a refund may fail, and a message recall may time out. Third, the time window: the later a compensation runs, the more it costs. Discovering ten minutes after a charge that something was wrong may mean the refund flow has already passed the reconciliation window.

In practice, write three lines of metadata for every side effect: compensate is the compensating action, compensatable_until is the deadline past which compensation is meaningless, and compensation_fail is what to do when compensation fails (alert, hand off to a human, or accept the inconsistency). Keeping these three lines in the tool registry is far more reliable than keeping them in orchestration logic.

The Ledger Must Be Written Before Execution

Compensation presupposes that you know what was done. If the ledger lives only inside the Agent's context, a process crash, a context truncation, or a context compaction will erase it. The ledger must be persisted in the same transaction as the Agent state, and it must follow an ordering: first write "I am about to do X" into the ledger, then execute X, and afterwards update the status to succeeded.

This write-intent-first ordering gives crash recovery a single unambiguous outcome: if the crash happened after the ledger write but before execution, recovery sees a pending record and has to decide whether X really ran. If it cannot decide, assume it ran and trigger compensation — over-compensating once is strictly better than missing one. The compensating action itself must be idempotent, so that it can absorb being executed one extra time.

Implementation: A Saga Ledger with Reverse-Order Compensation

# 副作用台账：正向动作与补偿动作都要在「执行前」落盘。
# 顺序固定为 记录意图 -> 执行 -> 标记结果，三步缺一不可。

import asyncio
import time


class CompensationFailed(Exception):
    pass


class Ledger:
    def __init__(self, store, run_id):
        self.store = store
        self.run_id = run_id

    async def declare(self, step, intent, compensate=None, ttl=None):
        rec = {
            'run_id': self.run_id,
            'step': step,
            'intent': intent,
            'compensate': compensate,          # 补偿动作的可调用标识
            'compensatable_until': ttl,        # 过了这个时刻补偿就无意义了
            'status': 'pending',               # pending / succeeded / failed /
                                             # compensated / uncompensatable / manual
            'result': None,
        }
        await self.store.insert_if_absent(rec)
        return rec

    async def mark(self, step, status, result=None):
        await self.store.update_where(
            {'run_id': self.run_id, 'step': step},
            {'status': status, 'result': result},
        )

    async def settled_but_uncompensated(self):
        rows = await self.store.find({'run_id': self.run_id, 'status': 'succeeded'})
        return [r for r in rows if r['compensate']]


async def resume_and_compensate(ledger: Ledger, registry, actor):
    # 恢复入口：先读台账，再决定继续还是补偿。不要凭上下文猜。
    rows = await ledger.store.find({'run_id': ledger.run_id})
    rows.sort(key=lambda r: r['seq'], reverse=True)   # 逆序补偿，后做的先撤

    for rec in rows:
        if rec['status'] in ('compensated', 'failed', 'uncompensatable'):
            continue
        comp = registry.get(rec['compensate'])
        if comp is None:
            await ledger.mark(rec['step'], 'uncompensatable')
            continue
        if rec['compensatable_until'] and time.time() > rec['compensatable_until']:
            await ledger.mark(rec['step'], 'manual')   # 过期补偿只能转人工
            continue

        for attempt in range(3):
            try:
                await actor.call(comp, args=rec['result'])   # 补偿自身必须幂等
                await ledger.mark(rec['step'], 'compensated')
                break
            except Exception as exc:
                if attempt == 2:
                    await ledger.mark(rec['step'], 'manual')
                    raise CompensationFailed(rec['step']) from exc
                await asyncio.sleep(0.5 * (2 ** attempt))

The line rows.sort(key=lambda r: r['seq'], reverse=True) in that recovery logic looks unremarkable but is frequently dropped in real projects. Reverse-order compensation is the defining requirement of a Saga: if you send the email before taking the payment, then on rollback you have to refund first and retract the notification second, because the content of the notification depends on the payment having happened. Reversing the order produces states like "refunded but the user never got the notification" that are extremely hard to explain.

Also pay attention to how the 'pending' status is handled. In the code above, pending records are excluded from compensation because with no result there is no way to tell whether compensation is needed. A stricter approach treats pending as "possibly already executed" and uses query-type tools to check the actual state before deciding. Writing that branch out explicitly is far more reliable than leaving it to implicit downstream handling.

On where to keep the ledger, a pragmatic choice is this: if your orchestration framework already provides durable checkpoints (writing each step's state to external storage and supporting recovery from any checkpoint), then the ledger can degrade into a field of the checkpoint itself rather than a separate table. But confirm that the checkpoint meets two conditions: the write happens before every side effect, and the recovery granularity is a single step. A lot of frameworks checkpoint at the granularity of some stage of a whole run, which is not enough.

Reconciliation: The Backstop Layer Beneath Compensation

Compensation is event-driven — there has to be a failure for there to be a compensation. That means it can only handle failures that were observed. If the ledger write itself was lost, or the process crashed between two critical steps leaving the status stuck at pending forever, compensation will never fire. This kind of "silent failure" is the hardest of all to defend against, because it produces no logs at all.

The countermeasure is periodic reconciliation: scan the ledger on a schedule, find every record sitting in an abnormal state, and check it against the server's actual state. There are four categories worth fishing out specifically — records stuck in pending past their timeout (possibly crash leftovers); records marked succeeded but past their validity window with no compensation conclusion (compensation was skipped); records marked as failed compensation that were never retried (a missed human handoff); and records where the server state contradicts the ledger, meaning some write was neither recorded as succeeded nor as failed.

The cost of reconciliation sits mostly in the "query the server's actual state" step, so grade by record type: for idempotent writes you can query directly by idempotency key, which is cheap; for irreversible actions you often have to ask the other side's system to confirm, which is expensive. So the frequency can differ by record type — idempotent records can be swept frequently, irreversible ones daily.

Handling what you find also has to be graded. Anything that can be fixed automatically (the server never actually executed it, so mark the ledger failed and release the quota) gets fixed automatically. Anything that needs judgment (the server executed it but the outcome is unknown) goes to a human, packaged with all the information needed to decide, using the same format as the human handoffs described earlier.

One last emphasis: reconciliation cannot replace compensation, it can only back it up. Reconciliation's detection latency is typically minutes to hours, whereas essentially all of compensation's value comes from timeliness — users accept "revoked ten minutes later" and "revoked on the spot" very differently. So reconciliation belongs at the last line of defense, guaranteeing the system does not permanently leave an inconsistency behind, not in the normal path as an error-correction mechanism.

Loop Stalls and Circuit Breaking

An Agent's loop fails to converge in three completely different shapes, and any single detection method covers exactly one of them. These three are worth treating separately, because their root causes differ.

Shape One: Genuine Repetition

The same action with the same arguments executes over and over. This is the easiest to detect — hash the action into a fingerprint and count. The root cause is usually a tool returning something the model cannot digest, such as an empty list or an error code, and the model keeps retrying the same path.

Shape Two: No Progress

The actions change, the wording changes, but the state summary does not change. The model is spinning in place: it queries the same page three different ways, gets the same result each time, decides "insufficient information," and queries again. This is the most expensive shape, because every step involves a real tool call and real inference, and it is the hardest to detect.

The detection method is to hash the state summary: if the state summary is byte-identical across several consecutive steps, the outside world has not changed, no amount of action-switching will surface new information, and the run should stop and report "the current information is not enough to make progress." That state summary has to cover the model's real basis for progress — hashes of files already read, return digests of tools already queried, the contents of the todo list — not merely a hash of the conversation text.

Shape Three: Budget Exhausted but Still Not Converged

The loop itself varies but never finishes the task. Only a hard budget stops this. The key decision is what happens when the budget is hit: it must not be a silent truncation followed by an answer that looks complete, because that is lying to the user; it has to be an explicit failure that hands back both the finished part and the unfinished part.

A Step Budget Should Not Be a Global Constant

Writing the maximum step count as a global constant is, in essence, an admission that you have not modeled task difficulty. A more workable approach lets the budget vary with task type: a single-step query from one clear instruction might need two steps, a reconciliation task spanning three systems might need twenty, and an open-ended research task might run to a hundred steps and still not converge. If you do not have task classification yet, at minimum make the budget adaptive — gather the historical distribution of converging step counts per task type and take a quantile plus margin as the budget for that class.

The risk with adaptivity is being skewed by outliers. If hitting the ceiling simply causes the budget to be raised, you get a loop of "the harder it is the more you spend, the more you spend the less it converges." The sound approach is to take a quantile between the median and a high percentile, to put an absolute ceiling on the per-task budget, and to watch the hit rate as a long-term metric. A rising hit rate means the budgeting system itself has a problem, not that tasks got harder — the latter normally comes with success rates rising in step, the former never does.

Health Memory Across Runs

A breaker scoped to a single run only protects that one trajectory. What is genuinely valuable is recording tool health outside the run, so that a new run knows from the start that "this tool has just failed eight times in a row." That memory is usually a shared cache plus a sliding-window counter. It is cheap, and the improvement to experience is direct: a new task does not have to hit the wall before routing around it, and you save that string of useless probe calls.

Health memory should also feed into planning. Append a dynamic "recently unstable" note to the tool description so the planner prefers stable tools when an alternative path exists; that is effectively injecting an operations-side signal into the reasoning-side view. It is far gentler than hard-blocking at the execution layer and it fits how Agents work — the model is very good at making the right trade-off when it has enough information. The only problem is that it assumes all tools are equally reliable by default.

Circuit States Need a User-Visible Explanation

When a breaker trips, what the user eventually sees should not be "task failed." Handle it in two cases. If the task can be completed in degraded form, say explicitly: "part of this information comes from cache rather than a live query, because the live query is temporarily unavailable." If the task genuinely depends on that tool, say clearly: "the live query service is currently unavailable; we recommend retrying later, and the parts already completed will not run again." The second sentence matters especially, because it conveys both the breaker state and the idempotency guarantee, which noticeably reduces the user's hesitation about retrying — and it quietly accomplishes the user education that reliability mechanisms usually skip.

Circuit Breakers: Putting a Ceiling on a Tool

Loop detection governs the inside of a single run; the circuit breaker governs the same dependency across runs. You need both: within one run, cycling through three tools in circles is something a breaker will not stop, and when an entire downstream service degrades, having each run retry on its own amounts to amplifying the failure N-fold.

The standard three-state circuit breaker fits tools well: the closed state accumulates failures against a rate threshold and opens; the open state fails fast through a cooldown window; the half-open state lets a small amount of probe traffic through. Agent scenarios need two adjustments. First, consecutive failures of the same tool have to be counted separately, because alternating between two broken tools dilutes the failure rate. Second, the error returned in the open state must tell the model in structured form that "this tool is temporarily unavailable, do not retry," otherwise the model probes on its own and pushes the breaker straight back open.

Implementation: Fingerprints, Repeat Detection, and the Triple Budget

# 死循环不是异常，是「有进展地重复」。三种症状分别用三种检测。

import hashlib
import json
import time


class StepBudgetExhausted(Exception):
    pass


class LoopDetected(Exception):
    pass


class NoProgress(Exception):
    pass


class LoopGuard:
    def __init__(self, max_steps=40, max_repeat=3, window=4):
        self.max_steps = max_steps
        self.max_repeat = max_repeat
        self.window = window
        self.steps = 0
        self.fingerprints: list[str] = []
        self.state_log: list[str] = []

    @staticmethod
    def _digest(value) -> str:
        body = json.dumps(value, sort_keys=True, ensure_ascii=False, default=str)
        return hashlib.sha256(body.encode('utf-8')).hexdigest()[:16]

    def _action_fp(self, tool: str, args: dict, state_digest: str) -> str:
        return self._digest({'t': tool, 'a': args, 's': state_digest})

    def observe(self, tool: str, args: dict, state_digest: str):
        self.steps += 1
        if self.steps > self.max_steps:
            raise StepBudgetExhausted(
                f'step budget {self.max_steps} exhausted at {tool}'
            )

        fp = self._action_fp(tool, args, state_digest)
        self.fingerprints.append(fp)
        self.state_log.append(state_digest)

        # 形态一：同一动作在同一个状态下反复出现
        repeats = self.fingerprints.count(fp)
        if repeats > self.max_repeat:
            raise LoopDetected(f'tool={tool} repeated {repeats}x on same state')

        # 形态二：动作在变，但外部状态摘要连续不变 -> 无进展
        recent = self.state_log[-self.window:]
        if len(recent) == self.window and len(set(recent)) == 1:
            raise NoProgress(f'state digest unchanged for {self.window} steps')


class ToolBreaker:
    '三态熔断：closed / open / half_open。反馈给模型时必须结构化，
     否则模型会自己反复试探，把刚打开的熔断又顶回关闭态。'

    def __init__(self, fail_threshold=5, cooldown=30.0, half_open_probes=2):
        self.fail_threshold = fail_threshold
        self.cooldown = cooldown
        self.half_open_probes = half_open_probes
        self.state = 'closed'
        self.fails = 0
        self.opened_at = 0.0
        self.probes = 0

    def allow(self) -> bool:
        if self.state == 'closed':
            return True
        if self.state == 'open':
            if time.monotonic() - self.opened_at > self.cooldown:
                self.state, self.probes = 'half_open', 0
                return True
            return False
        if self.state == 'half_open':
            if self.probes < self.half_open_probes:
                self.probes += 1
                return True
            return False

    def on_success(self):
        self.state, self.fails = 'closed', 0

    def on_failure(self):
        self.fails += 1
        if self.state == 'half_open' or self.fails >= self.fail_threshold:
            self.state = 'open'
            self.opened_at = time.monotonic()

The most important design choice in the "no progress" detection is the coverage of the state summary. Hashing only the conversation history treats the model's redundant restatements as progress, so it never fires; hashing only tool return values misses the progress the model makes at the reasoning level. A reasonable approach summarizes three things: the set of external facts obtained, the changes to the todo list, and the set of actions attempted. Only when all three change is it real progress.

There is one more frequently overlooked dimension in the triple budget: the cost budget. A step ceiling cannot control the case where a single step is expensive. A run burning a hundred thousand tokens per step for twenty steps never exceeds its step count, but the money is already gone. Making token count and dollar cost hard budgets on a par with step count, and failing explicitly when they are hit just as the wall-clock budget does, is a baseline requirement for productionizing Agents in 2026.

For what to do once a breaker opens, one pragmatic piece of advice: do not fail silently and do not let the model hammer at it — turn it into a planning input. Give the model a structured error that plainly states "tool X is temporarily unavailable," and hint that it may either take a degraded path or explain the situation to the user. That way the breaker stops the failure from amplifying while the Agent keeps its ability to degrade on its own.

Model-Side Failures: Not Timeouts, but Just as Fatal

Everything so far has dealt with "tools and infrastructure will fail." But Agents have another class of failure that happens entirely on the model side: there is no timeout and no error — the model simply got it wrong. This class needs no chaos test to reproduce, and it is often harder to discover than network jitter, because the system looks perfectly healthy the whole time.

Output Truncation and Structural Collapse

The most common case is structured output being cut off mid-generation. The cause may be hitting the maximum token length, or a streaming connection dropping. On the surface it shows up as a JSON parse failure, but what is really happening is that a half-finished result is being treated as a failure. The key distinction here is that a parse failure and a generation failure must be separated. The former means the model may already have figured it out and merely did not finish talking; the latter means the model itself produced nothing usable. The correct handling of the former is to continue generation or lower the output requirement, not to start over.

The most effective practice is to constrain output size explicitly in the prompt and to prepare a fallback parse path for parse failures — for instance, salvaging the fields that are already complete from the truncated JSON. It is not an elegant approach, but on real traffic its payoff far exceeds one expensive and uncertain retry.

Tool Hallucination: Calling Things That Do Not Exist

The model calls tool names that are not in the registry at all, or passes parameters that do not exist for a real tool. This failure is especially frequent when the tool name and the actual function name disagree: if the name in the description differs from the registered name by a single character, the model will call the one from the description.

Protection has to happen before execution, not as a fallback inside an exception handler. Validation comes in three layers: is the tool name in the registry, are all required arguments present, and are the argument types and value ranges legal. All three belong in the orchestration layer — do not expect the downstream service to return a 400 and let the model self-correct. That is an expensive round trip, and the model has a substantial probability of sending the very same request again. The more thorough approach is to establish a single source of truth for tool names between the prompt and the registry, generating the registry documentation instead of hand-writing it, eliminating the physical origin of the inconsistency.

Accumulation of False Facts

This is a failure mode specific to Agents and the subtlest of them all: after a tool errors out, the model writes the error message into its reasoning as if it were a fact, and plans on top of it. A query returning "no permission" may be read by the model as "the data does not exist," producing a conclusion that is actually biased. A write that times out may lead the model to assume the write succeeded and carry on, when in reality nothing happened at all.

The way to counter it is to strictly separate "fact" from "unresolved." The structure returned by a tool must carry an explicit field marking the status of that call, and while reasoning the model may treat only successful returns as facts; unresolved or failed returns count only as items to handle. If this constraint lives merely in the prompt, adherence is not high. The more reliable approach is to have the framework, when assembling the next round of context, replace failed calls with a structured placeholder explanation instead of passing the raw error text straight through. The raw error text carries far more information than the model can use, and its ambiguity is precisely the raw material of hallucination.

Parameter Hallucination and Up-front Validation

More insidious than a missing parameter is a parameter value that looks plausible but does not exist: ask the model to fill in an order ID and it may copy a similarly shaped one out of the context, or invent one following some pattern. Downstream systems often cannot tell, and can only return "no such order," at which point the model tries again, invents another one, and lands in a loop that is both slow and non-convergent.

Effective protection moves validation as early as it can go somewhere it can be decided: an argument must come from a set returned by a tool rather than being freely generated by the model; it must satisfy some checkable pattern, such as a fixed format or length; and it must pass an existence check. When an up-front check fails, return a structured clarification request rather than an error, so the model fills in the gap instead of guessing again. For fields like amounts and times, it is better to let an explicit selector supply the value, whether from the user or from the system, than to have the model hand-write it.

After a Timeout: Resume From the Breakpoint Instead of Starting Over

A timeout is the signal most easily mishandled as "this attempt failed," when its real semantics are the outcome is unknown. The call may never have been sent, may have run halfway, or may have completed with only the response missing. That uncertainty dictates a handling strategy for timeouts that is completely different from that for explicit business errors.

A Timeout Is a Kind of Indeterminate Outcome

Indeterminate outcomes have three handlings, corresponding to three classes of tool. For read-only tools, retrying directly is optimal, because repeating them costs nothing. For idempotent writes, first query the previous result by idempotency key and use it if you find one, retrying only if you do not — this step is steadier than retrying directly, because it avoids one unnecessary duplicate execution. For irreversible tools, you must query the status before deciding and enter the compensation flow when necessary. A blind retry is never acceptable.

Filing "timeout" and "explicit failure" under the same error category is one of the deepest hidden mistakes in a lot of implementations. It looks like a single line of classification code, but the real consequence is that the system performs retries on irreversible tools that should never be retried. And such bugs never show up in test environments, because test environments generally do not time out.

Three Granularities of Resume

After a run times out, there are three granularities of resume available, and their costs and payoffs differ enormously.

The lightest is restarting from the current step: completed steps are skipped via their idempotency keys, and the timed-out step is retried. It is the simplest to implement and suffices for the vast majority of scenarios. The second is resuming the whole run from a checkpoint: context, completed actions, and the remaining budget balance are all persisted and reloaded on recovery. For very long-running tasks this is the only option, because replanning from scratch is both expensive and capable of changing the conclusion.

The third is degraded delivery: stop trying to finish, and instead organize what was completed into a result, hand it to the user, and explain what remains. It is often treated as failure, when it is actually the most reasonable ending for a long task whose budget has run out. For an interactive product, a timely answer with its gaps clearly marked is worth far more than a progress bar that is still spinning.

Resume Must Be Visible to the User

Whenever a resume has happened, the returned result should carry an explicit resume marker: which step it continued from, what was skipped beforehand, and whether any side effect might have executed twice. A user who sees "resumed from step three; the first four steps will not repeat" trusts the system very differently from one who sees a response identical to the first attempt. The latter makes people afraid to click retry, and the problem just sits there.

There is also one easily missed point: a resume must reuse the original intent identifier, not mint a new one. If a resume generates a new intent identifier, the idempotency key changes, the steps that already succeeded get executed again, and the entire meaning of the resume mechanism is inverted. This constraint should be enforced by the framework, not left to callers remembering to pass the right argument.

Resource Failures: Subprocesses, Handles, and Leaks

Compared with an ordinary service, Agents have one more systemic failure mode: resource leaks. It is not a timeout, not a retry, not an idempotency problem, yet it still makes the system gradually unavailable — and its symptoms are often misdiagnosed as "load is too high."

Leak Sources Specific to Agents

Three sources are the most common. The first is subprocesses left behind by tool calls: a command-execution tool starts a long-running job, the call times out and returns, and the process keeps running. If tools execute concurrently, a few loop iterations accumulate a large pile of zombie processes that eat the machine's CPU and memory. The second is file handles and temporary files: files a tool opened, intermediate artifacts it downloaded, browser sessions — all of these have to be closed on the exception path, and an implementation that only handles the normal return path leaks them all at once when the failure rate rises.

The third is sessions and connections: opening a new connection per tool call without reusing it or returning it to the pool makes the connection count grow monotonically under load until it exhausts the file descriptors. A fourth one that is easily missed is resource references inside the context: shoving a large file or a full page of HTML straight into the context blows it up within a few steps, and the truncation that follows degrades the model's behavior along with it.

Leases and Reclamation

The general tool for fighting leaks is the lease: every resource registers a lease at creation, and the lease has an expiry; a reclaimer scans on a schedule and collects expired leases. That is far more reliable than cleanup at the end of a call, because it holds up against both abnormal exits and timeouts.

Lease registration has to cover all resource types, including subprocesses, temporary files, open handles, and downstream quota that has been leased out. Quota deserves special emphasis: in Agent scenarios quota is often allocated per run, so if a run crashes and never finishes cleanly the quota leaks, and over the long run that slowly drains the entire tenant's allowance. The signature of this failure is "some tenant's usage grows for no visible reason," which makes it brutally hard to investigate.

Resource Guardrails on Tool Calls

Guardrails can live at the adapter layer, where the rules are fairly stable: subprocesses must go into a process group and the whole group must be killed on timeout rather than just the parent; temporary files must be registered in the lease table with a cleanup path; each tool call needs its own memory and file-size ceiling; and content beyond a threshold must be truncated automatically with an explicit note attached, rather than passing the full content straight through to the model.

"Truncating returned content" is worth singling out, because it is a reliability problem and a quality problem at once. A tool that returns a hundred thousand lines of logs blows out the context, slows down inference, and leaves the model hunting through irrelevant information for the one useful line. A sound guardrail should preferentially keep the structured key fields (error code, status, the target field), collapse large bodies into a placeholder note, and tell the model "details are collapsed; query them on demand if you need them."

Cost Is a Budget Too

Cost gets pulled out on its own because it is often treated as a by-product of performance tuning, when in fact it is one of the hard constraints on reliability. An Agent whose spend budget runs away has failed even if every single step succeeded — it burned the budget, consumed the quota, and left other users of the same tenant queuing.

The Marginal Return of a Retry Decreases

The expected payoff of the first retry is usually high: it absorbs transient jitter and succeeds most of the time. The second and third decline noticeably, because requests that survive two retries are usually real faults rather than jitter. Yet a lot of implementations get the backoff policy wrong precisely here: they decide whether to retry based on the error type rather than on how many retries have already happened. The correct approach makes the retry count itself part of the decision and explicitly charges the cost of a retry against the budget, so that the second retry is automatically vetoed when the first has already spent most of the budget.

The more fundamental judgment is this: a call is worth retrying if and only if the probability that it succeeds on retry, multiplied by its value, exceeds the budget it consumes. Retrying a ten-dollar operation five times is obviously a bad deal, while retrying a hundred-thousand-dollar transfer is worth it. This judgment can be written as a simple expected-value function; it does not need to be very precise, since getting the order of magnitude right is enough to avoid obvious waste.

Put Cost into the Objective

The most effective practice is to set a cost budget per class of task and change behavior once a fraction of the budget is consumed: for example, if half the budget is gone and the task still has not converged, tell the planner to simplify the remaining steps; at eighty percent, stop exploring and just wrap up. The degraded path taken when the budget runs out must be designed in advance, because by the time you get there there is no room left for complicated decisions.

Another related metric is cost per success, not cost per call. It rolls in the overhead of retries, spinning loops, and failed attempts, so it is much closer to the real economics. A team may see a very low cost per call and then discover on the bill that each completed task cost five times what was projected; the difference is hiding inside retries and no-progress loops.

When You Must Hand Over to a Human

Even a well-built set of automatic mechanisms has an endpoint, and you need a clear escalation path. Treating "hand over to a human" as the backstop rather than as failure means you have to make explicit, inside the automatic machinery, which situations are no longer attempted.

Three Escalation Tiers

The first tier is irreversible with no compensation: the action is already externally effective and cannot be undone, or its compensation window has closed. These cases should stop the automatic flow immediately and go to a human, with no further speculative retries or compensation attempts. Continuing to automate has exactly one outcome, which is a larger loss.

The second tier is repeated failure: several different strategies against the same goal have all failed, or the breaker keeps reopening. The root cause is usually outside the framework — misconfiguration, missing permissions, an incompatible downstream interface — and automatic mechanisms cannot fix it, so continuing just burns budget. The criterion has to be based on "failures across different strategies" rather than on total failures, otherwise one bad strategy gets retried forever.

The third tier is uncertainty beyond tolerance: compensation would succeed but the business cannot accept it, or continuing carries legal or compliance risk. This tier requires business knowledge, so the system has to be able to package up everything needed for the decision and hand it to a person, rather than merely reporting "it failed."

What the Handoff Package Should Contain

Handoff quality determines the cost of human takeover. A qualified handoff package needs at least: a complete timeline of what happened (each action, a digest of its inputs, its result), the current certain and uncertain items (which are confirmed in effect, whose state is unknown), the strategies already tried and not yet tried, and an account of every in-flight action (whether it might still be running, and whether to cancel it first).

The item "whether it might still be running" is the one most easily dropped. A subprocess or async task that timed out but may still be executing will very likely be duplicated during human takeover if the handoff package does not say so. The correct approach is to hand over the lease information for in-flight resources as well, so the person taking over knows what to check first.

Regression Verification for Reliability

Every mechanism described above is easy for a later refactor to break silently, and such breakage usually does not surface in functional tests. Reliability therefore needs its own regression verification.

Version the Fault Scenario Set

Treat the chaos test scenario set as code: give each scenario a number, a description, and its expected invariant outcome, and version it alongside releases. The value is that when a release makes the breaker stop firing, or compensation stop executing, the regression test notices immediately instead of waiting for a production incident.

The scenario set should cover every injection layer mentioned above and deliberately include combinations — "the tool succeeds but returns a failure, the downstream is simultaneously rate-limiting, and the process simultaneously crashes." Combinations where three things happen at once are what actually test the mechanisms. In practice it is extremely common for each item to pass in isolation and to fail once combined.

Reading Degradation Signals

Reliability metrics do degrade, and the degradation itself has to be read correctly. The easiest thing to misread is a metric that suddenly improves a lot — circuit breaker trip rate dropping to zero, for instance, usually does not mean the system got better; it means the threshold was temporarily raised, or health memory expiry shifted the measurement basis. Changes to reliability metrics must be reviewed alongside code changes, or you lose your basic sense of the system.

Another reading principle is look at ratios, not absolute values. Circuit breaker trips rising from ten to a hundred may just mean traffic doubled, while a retry amplification factor rising from 1.2 to 1.5 means genuine degradation. That is why the set of ratio-style metrics listed earlier matters: they are insensitive to traffic changes, so they can signal before the absolute numbers have moved at all.

Partial Failure: Continue or Roll Back

At the decision point there is exactly one core question: do the side effects that already took effect get kept, or undone? The answer depends on a combination of three dimensions — reversibility, idempotency, and whether the failure can be retried. Cross them and the decision space turns out to be quite small.

Nature of the step that already took effect	Failure point is retryable	Failure point is not retryable
Read-only, no side effects	Continue; usually needs no user intervention	Degrade: answer with the information on hand and flag the gap
Idempotent write	Continue: replaying earlier steps gets collapsed by the key	Roll back, or mark as partial success and let the user decide
Write with a compensating action	Continue; hold off on triggering compensation	Compensate: run compensating actions in reverse order and report the result
Irreversible with no compensation	Avoid if at all possible; if it already happened, alert immediately	Hand off to a human; never retry automatically

Defaults should be conservative

The table looks complicated, but in code it should collapse to a handful of branches. One workable default rule is: a read-only prefix may proceed freely; once you enter the write path, no later failure is allowed to continue silently — it must land explicitly in a partial success state. The reason is that a read-only failure does not change the world, whereas continuing after a write-path failure means the Agent keeps operating on a state it has already misread, turning a single point of failure into a cascading error.

A partial success state has to be visible to the user, and it has to be specific. What you return should contain at least three things: the list of completed actions, the failed actions with their reasons, and whether the state is recoverable. Showing a user "3 steps done, step 4 timed out, you can retry directly (the first three will not be repeated)" is far more useful than a vague failure message, and far more honest than implying that nothing happened.

Report partial success honestly

Partial success needs a fixed report structure rather than free-form improvisation by the model. At minimum it should contain four things: the completed actions and their key results, so the user knows what the system actually did; the incomplete actions and their reasons, so the user can judge whether that is acceptable; whether the current state is recoverable, meaning whether a retry can run without repeating side effects; and the recommended handling. Drop the fourth item and the most natural user reaction is to retry — and retry semantics must be identical to the previous attempt, or all the earlier work is wasted.

This structure has another role that is easy to overlook: it turns partial success from an exceptional state into a normal state that downstream code can recognize. If the return value carries an explicit status field and a machine-readable list of completed actions, the caller can react correctly on its own screen instead of guessing what happened from natural language. This matters most when an Agent is embedded in an existing business process, because downstream systems generally have no way to interpret what "most of the operation completed" actually implies.

Conversely, there is one pattern that must be explicitly avoided: wrapping partial success as success. Adding "most of the operation completed" to the return while the status field still says success. The caller advances the workflow on that basis, which just pushes a known inconsistency out to a more distant point where it is also harder to attribute. In reliability engineering, an honest failure plus a recoverable status marker is worth far more than a convincing-looking success — the problem with the latter is not that it looks bad, but that it strips the error of any shape that can be acted upon.

Checkpoints decide the granularity of recovery

Whether rollback is possible depends on whether you can return precisely to that step. That means checkpoint granularity has to line up with side-effect granularity. If a run has twenty steps and step five fails so you need to roll back, but checkpoints are only recorded at step ten, you can only start over — and starting over is a disaster whenever non-idempotent side effects are involved.

The concrete recipe is this: write a checkpoint immediately after each external side effect is persisted, and record in the checkpoint the idempotency key, the call arguments, a result summary, and the corresponding compensation identifier. On replay, use checkpoints as boundaries to skip steps that already completed, and let the idempotency key keep replay safe. Only that combination adds up to genuine "recoverable execution", and it is the direction current mainstream orchestration frameworks are filling in.

Concurrency and Rate Limiting Across Tools

Concurrency in an Agent has a property ordinary services do not: the fan-out scale is decided by the model and has no a priori upper bound. You can hard-code a maximum of eight concurrent calls, but if the model plans thirty-six sub-calls, the limit has to be enforced at the tool's exit rather than its entry.

Level of rate limiting: global beats per-tool

Rate limiting tool by tool is almost useless in an Agent setting. The same downstream service may be reached through thirty different tool names; give each tool its own token bucket with a limit of five and real concurrency becomes one hundred fifty. The correct approach is to hang the token bucket on the downstream resource identity rather than the tool name, so multiple tools that map to the same resource share one bucket.

This requires an explicit declaration mapping tools to resource groups — for example, a "search tool" and a "vector retrieval tool" both mapping to the retrieval cluster, a "database write tool" and a "database read tool" both mapping to the primary database. Any rate-limit configuration that only names tools will eventually be bypassed once the tool count grows.

Adaptive concurrency beats a fixed ceiling

Fixed concurrency ceilings have a well-known problem: set them too high and you crush the system when the downstream degrades; set them too low and you waste throughput when the downstream is healthy. Adaptive concurrency treats end-to-end latency as the feedback signal for the concurrency ceiling — rising latency means the queue has already started to back up, and admitting more work only makes it worse; flat latency means there is headroom left and you can scale up gradually. This is the classic approach derived from queueing theory, and it is the idea behind Envoy's adaptive concurrency filter and the related implementations in service meshes.

The payoff inside an Agent is even more obvious: a single run may contain a large number of calls of very different shapes, and a fixed ceiling either throttles the slow tools or waves through the fast ones, whereas adaptive concurrency contracts automatically on slow tools and opens up automatically on fast tools.

Rate limits have to be coupled to model-side quotas

An easily overlooked coupling is this: tool-side rate limiting and model-side quota consumption are two valves on the same pipeline. If one fan-out amplifies both inference cost and downstream pressure, throttling only one of them just transfers the pressure to the other. Drop tool concurrency from eight to four and, with the planning logic unchanged, token consumption on the model side barely moves — each step simply waits longer, and the end-to-end pipeline may actually end up slower because of the waiting.

The more fundamental move is to reduce fan-out at the source: turn "query thirty records one at a time" into "run one batched query", turn "update them one at a time" into "submit as a batch". These changes pay off structurally and take effect without any rate-limiting mechanism whatsoever. By contrast, tuning the concurrency ceiling only spreads the same total volume over a longer stretch of time, making the user experience worse while total work stays the same. So before you touch a rate-limit parameter, confirm that you are not using a concurrency ceiling to paper over a problem that a batch API should have solved.

Another coupling dimension is cost budget. When the remaining budget cannot support the current plan, the correct action is to narrow the scope of the task rather than queue and wait: if a "export all records" job only has budget for a third, either change it to export the last thirty days, or split it into three jobs the user triggers separately. Queueing degrades quickly into a fairness problem in multi-tenant environments — high-budget jobs fill the queue while low-budget jobs wait forever behind them, and the latter are often the more urgent interactive requests.

There is also an ordering issue between rate limiting and degradation that is easy to miss: tell the user you are degrading, then degrade. Plenty of implementations silently swap a live query for a cached one the moment degradation triggers, and the user ends up holding stale data while believing it is current. Putting the degradation marker into the return value is the same mechanism as the partial success report; these two should share one structure rather than being written out twice.

Fan-out convergence and write conflicts

Beyond rate limiting, you also need to cap fan-out scale at the planning stage. Three layers of means: constrain the number of actions returned per turn at the prompt level; set a hard ceiling on parallelism in the orchestration layer and queue the overflow; and merge multi-step operations on the same resource into a single batch tool at the execution layer. The first two are policy, the last one is engineering. It is the most effective and also the most expensive.

Write conflicts are an orthogonal problem. Writes to the same resource must be serialized, or you get the classic lost update. This is easier to hit in an Agent because the "read first, then write" reasoning pattern is naturally baked into the action sequence: the model reads the inventory, changes the price, and helpfully writes back the stale inventory value it read. There are three defenses: optimistic concurrency (write with a version number, on conflict re-read and retry), pessimistic locking, and the single-writer pattern (all writes enter one queue executed by a single consumer). For finance and inventory, single-writer plus an optimistic version number is the least-effort combination.

Implementation: resource-level token buckets and adaptive concurrency

# 并发控制挂在「下游资源」上而不是「工具名」上。
# 同一个集群被多个工具访问时，共享一个桶才拦得住扇出。

import asyncio
import time
from collections import defaultdict


class TokenBucket:
    def __init__(self, rate: float, burst: float):
        self.rate = rate
        self.burst = burst
        self.tokens = burst
        self.ts = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self, n: int = 1) -> None:
        while True:
            async with self._lock:
                now = time.monotonic()
                self.tokens = min(self.burst, self.tokens + (now - self.ts) * self.rate)
                self.ts = now
                if self.tokens >= n:
                    self.tokens -= n
                    return
                deficit = n - self.tokens
                wait = deficit / self.rate
            await asyncio.sleep(wait)          # 队内等待，而不是直接拒绝


class Governor:
    def __init__(self, resource_rates):
        self.buckets = {r: TokenBucket(rate, burst=rate)
                        for r, rate in resource_rates.items()}
        self.concurrent = defaultdict(int)
        self.limit = 8
        self.min_limit, self.max_limit = 1, 64
        self.ewma_latency = 0.05
        self._sem = asyncio.Semaphore(self.limit)

    async def run(self, resource: str, tool_fn, *args, **kw):
        await self.buckets[resource].acquire()
        async with self._sem:
            self.concurrent[resource] += 1
            t0 = time.monotonic()
            try:
                return await tool_fn(*args, **kw)
            finally:
                self.concurrent[resource] -= 1
                rtt = time.monotonic() - t0
                # 自适应：延迟抬升就收缩并发，平稳就缓慢放开
                self.ewma_latency = 0.9 * self.ewma_latency + 0.1 * rtt
                if self.ewma_latency > self.limit * 0.25 and self.limit > self.min_limit:
                    self.limit = max(self.min_limit, int(self.limit * 0.9))
                    self._sem = asyncio.Semaphore(self.limit)
                elif self.limit < self.max_limit:
                    self.limit += 1
                    self._sem = asyncio.Semaphore(self.limit)

The adaptive logic inside Governor above contains one illustrative simplification: swapping out the semaphore wholesale leaves in-flight tasks briefly unconstrained. A production implementation should decide, at release time, whether to grant additional permits according to the new ceiling instead of rebuilding the semaphore. It is kept simple here to highlight the feedback logic itself: latency is the observable of queueing pressure and the only input to adjustment.

Multi-tenant deployments need an additional fairness layer. A long job that has been running for twenty minutes while holding a large number of concurrency permits will starve other tenants. A workable approach is to bucket by tenant and add a layer of round-robin scheduling on top, or to cap the total permit count per run and allocate it by user priority. This layer is close to mandatory on multi-tenant SaaS agent platforms; for internal tooling you can skip it at first.

Chaos Testing: Proving These Mechanisms Actually Work

All the mechanisms in the five sections above share one problem: they are completely invisible on the happy path. Write an idempotency key correctly and you get no feedback at all; execute compensation logic correctly and all you get is one line of logs. Without fault injection you have no way to know whether the retry budget actually held, whether the circuit breaker actually opened, or whether compensation actually covered every write that took effect.

So chaos testing in an Agent context is not a nice-to-have; it is the only way to verify these mechanisms. The key insight is this: chaos testing asserts invariants, not pass rates. "The request succeeded" is not a useful assertion, because under heavy fault injection the success rate should not be high in the first place; "each idempotency key produced at most one side effect" is.

Four levels of injection point

Ordered from outside in, injection points fall into four layers, and each layer verifies something different.

Network layer: use a local proxy or middleware to inject timeouts, connection resets, truncated responses, and 5xx. This layer is closest to reality and can validate the timeout budget and the retry logic at the same time.

Tool adapter layer: instrument before and after the tool call to manufacture the hardest possible scenario, "execution succeeded but the call returned failure". This layer exists specifically to verify idempotency keys; there is no other way to test them.

Model gateway layer: inject rate limiting, context overflow, truncated output, and structurally incomplete JSON. Truncated output is what exposes many fragile assumptions in parsing layers, and it deserves dedicated coverage.

Orchestration and storage layer: inject process crashes, brief storage unavailability, and checkpoint write failures. This layer verifies the write-ahead ordering of the ledger and the recovery logic; it is the acceptance gate for compensation.

Invariants that must be written down

Invariants are the core asset of chaos testing, and they should be maintained centrally as an executable checklist. The following cover every mechanism discussed above:

• Each idempotency key produces at most one external side effect;
• None of the three budgets — steps, wall clock, tokens — was breached, and breaching produces an explicit failure rather than a silent truncation;
• Every side effect with a success status ultimately lands in exactly one of "compensated", "uncompensatable", or "handed to a human" — no dangling records;
• No unhandled exception escapes the run boundary; every failure has been converted into a structured error;
• After the circuit breaker opens, no further calls to that tool occur within the same run;
• Compensation actions are executed in strictly reverse order relative to the forward actions;
• Peak concurrency never exceeds the configured ceiling, and no single tenant is starved.

Implementation: an invariant checker

# 混沌测试断言的是不变量，不是「没报错」。
# 这几条就是超时/重试/补偿三层机制是否真的生效的判据。

from collections import defaultdict


class InvariantViolation(AssertionError):
    pass


def check_invariants(txn) -> None:
    # 1) 幂等：同一个键最多产生一次外部副作用
    by_key = defaultdict(int)
    for e in txn.effects:
        if e.idempotency_key:
            by_key[e.idempotency_key] += 1
    dup = {k: n for k, n in by_key.items() if n > 1}
    if dup:
        raise InvariantViolation(f'幂等失效: {dup}')

    # 2) 三重预算都没被击穿，且是显式失败
    if txn.steps > txn.limits.steps:
        raise InvariantViolation('步数超限')
    if txn.wall_clock > txn.limits.wall_clock + 0.5:
        raise InvariantViolation('墙钟超限')
    if txn.tokens > txn.limits.tokens:
        raise InvariantViolation('token 超限')
    if txn.limit_breached and not txn.degraded_explicitly:
        raise InvariantViolation('预算触顶但未显式降级')

    # 3) 没有悬空副作用：成功的写入都必须有处置结论
    succeeded = {e.step for e in txn.effects if e.status == 'succeeded'}
    settled = {e.step for e in txn.effects
               if e.status in ('compensated', 'uncompensatable', 'manual')}
    if succeeded - settled:
        raise InvariantViolation(f'悬空副作用: {sorted(succeeded - settled)}')

    # 4) 补偿逆序
    order = [e.seq for e in txn.effects if e.status == 'compensated']
    if order != sorted(order, reverse=True):
        raise InvariantViolation(f'补偿顺序错误: {order}')

    # 5) 没有未处理异常逃逸
    for e in txn.events:
        if e.kind == 'exception' and not e.handled:
            raise InvariantViolation(f'未处理异常逃逸: {e.name}')

    # 6) 熔断打开后同 run 内不再有该工具的调用
    for name, opened_at in txn.breaker_opens:
        after = [c for c in txn.calls if c.tool == name and c.t > opened_at]
        if after:
            raise InvariantViolation(f'熔断后仍调用 {name} 共 {len(after)} 次')

    # 7) 并发峰值不超上限
    if txn.peak_concurrency > txn.limits.max_concurrency:
        raise InvariantViolation('并发超限')

Running chaos in CI without going flaky

The most common failure mode of chaos testing is degenerating into a stream of randomly failing continuous integration runs, which is worse than having no chaos testing at all, because it trains the team to ignore red builds by reflex. There are three controls, and all three are quite concrete.

First, fix the fault scenario, not the random seed. The only thing that genuinely needs to be random is the choice of failure mode. The concrete parameters — how many milliseconds of timeout, how many consecutive failures, which step to inject at — should be determined by a scenario number, so that any failure can be reproduced exactly using the same number. Moving randomness from the parameter layer to the scenario layer gives you both breadth of coverage and reproducibility. Fixing only the seed hides the combinatorial explosion; fixing only the scenarios means combinations you never listed go untested.

Second, assert invariants rather than asserting success. Once you inject faults, whether something succeeded is itself uncertain, so using it as a criterion necessarily produces flakiness. Invariants always hold — no matter what fault you inject, "an idempotency key produces at most one side effect" is never violated, so it is a stable criterion. This is the same thing as the invariant list in the previous section, and nearly all the value of chaos testing is concentrated in it.

Third, separate invariant violations from ordinary failures into two distinct states. An invariant violation must block the merge immediately, because it means the mechanism itself is defective. An ordinary failure should only be recorded and counted, so you can observe mechanism activation rates. Mixing the two into the same red build leaves a team with two rational reactions — indiscriminate re-runs or indiscriminate ignoring — and both degrade chaos testing into a pure noise source.

Data isolation in a chaos environment

When you inject faults like "execution succeeded but the call returned failure" in a test environment, if the target service has real side effects, your tests produce real data. At best you pollute a development database; at worst you trigger real notifications and real charges. The injection point must therefore satisfy one of two conditions: the fault only changes what the client observes and leaves server state untouched, or the target is a behaviorally equivalent stand-in service.

The cleanest implementation is a programmable proxy layer between the tool adapter and the real service, with all injection happening inside that layer. It simultaneously solves two further problems: the injection point is decoupled from business code, so it does not silently break during refactoring; and the proxy layer records a complete call trace, which becomes the raw material for later trace replay. This is also the prerequisite for bringing chaos testing into continuous integration — without that isolation layer you cannot run it on every commit.

One more easily missed scenario deserves coverage: state convergence under sustained failure. Short faults test fast failure and circuit breaking; long faults test the recovery path — after a ten-minute outage, is system state still consistent, are there dangling side effects left behind, and does the compensation backlog pile up beyond what can be processed in reasonable time. These tests cannot run in continuous integration and need periodic drills, perhaps scheduled monthly. The drill report itself is also excellent material for aligning a team.

Execution strategy: work down from the proxy layer

The injection order should follow the principle of going from controllable to uncontrollable. First do deterministic injection at the network layer (fixed timeouts, fixed failures) to validate single-point logic. Then inject random combinations of faults to validate behavior when several mechanisms stack up. Finally inject long-running sustained faults to validate the stability of the circuit breaker and degradation paths.

Trace replay is the key to improving throughput. Take a sanitized real production trace as input and fix the random seed at the injection point, so every CI run executes the same set of fault scenarios and failures reproduce exactly. Conversely, randomness should be reserved for the choice of failure mode, not for the individual parameters of each fault; otherwise the same bug shows up today and vanishes tomorrow and nobody can locate it.

There is also one step that is easily skipped: measuring mechanism activation rates. Do not just count whether tests passed; also count what fraction of runs hit the retry budget, what fraction opened the circuit breaker, and what fraction executed compensation. If any of those rates stays at zero over a long period, the mechanisms are never being triggered at all and the tests are false safety; if some rate is absurdly high, the thresholds need adjusting. This single metric reflects the real state of the mechanisms better than a plain pass rate does.

Observability: Defining SLOs for an Agent Run

All reliability work eventually has to land on observable numbers, otherwise you cannot tell whether an improvement actually worked. Observability for an Agent has one specific difficulty: a single run is a composite process spanning several minutes and containing many external calls, and monitoring it as one request throws away almost all the useful information.

Events that must be first-class

In the event stream of every run, the following event types must be recorded structurally and must not live only inside log text: the action list produced by planning, the argument summary and idempotency key of each tool call, the reason and count of each retry, each circuit breaker state transition, the execution and result of each compensation, and the budget consumed by each step. Miss any one category and post-mortem review degrades to guessing by reading logs.

Idempotency keys absolutely must be embedded in events. A useful triage move is this: take the anomaly order number reported by the user and search backwards for every call carrying that key; if more than one shows up, the problem lies in the idempotency layer rather than in business logic. Without the key recorded, that query simply cannot be run.

Move from average success rate to tail metrics

The metrics worth watching in an Agent setting are not average success rates but a set of tail-related ratios:

• Retry amplification factor: total tool calls divided by the number of tasks the user initiated. It exposes a retry storm earlier than average cost does.
• Circuit breaker trigger rate: grouped by tool. A tool that keeps triggering is in sustained failure; a tool with zero triggers may simply have a threshold that is too loose.
• Compensation success rate: successful compensations divided by compensation attempts, plus the fraction of compensation failures handed to a human.
• No-progress step ratio: steps stopped by no-progress detection as a share of total compute consumed — a leading indicator for detecting insufficient model capability.
• Idempotency key conflict count: how often the same key was deduplicated and collapsed. A spike means upstream is submitting duplicates.
• Budget exhaustion rate: the fraction of runs that hit the wall clock or step ceiling. It reflects how well budget settings match actual demand.

Slicing these metrics by tenant, by tool category, and by task type makes them far more informative. The overall no-progress step ratio might be only a few percent, while one particular task type sits at thirty percent — and that means the planning approach for that task type should change, not the threshold.

Instrumentation and Events: Without Them None of the Above Can Be Verified

Every mechanism mentioned in the preceding sections shares one premise: on the happy path they produce no visible effect at all. Whether the idempotency key is doing anything, whether the circuit breaker actually opened, whether compensation covered every write — only the event stream can answer those questions. So the event contract is not an accessory to observability; it is part of the reliability mechanisms themselves.

A serviceable list of events

What you do not need is a long string of metrics. What you need is a set of events that can stitch the mechanisms together. The following is the minimal set, and missing any one category leaves a blind spot in triage:

• run_started and run_finished: the start and end of a run, carrying total budget, total duration, total cost, and final status.
• plan_created: a snapshot of the planner's action list, carrying the intent identifier. This is the anchor for every idempotency key that comes later.
• budget_allocated: each budget allocation, carrying the allocation basis and the amount remaining. Without it you cannot explain "why is there no quota left here".
• tool_call_started and tool_call_finished: paired calls carrying the idempotency key, the remaining time to the deadline, the attempt number, and the result classification.
• retry_scheduled: carrying the retry reason, the remaining budget, and whether it came from a server instruction. This one distinguishes "we decided to retry" from "the server told us to wait".
• breaker_state_changed: a circuit breaker state transition, carrying the triggering basis. It turns "why didn't the breaker take effect" into a queryable question.
• compensation_started and compensation_finished: carrying the ledger record identifier and the result. This is the only basis for reconciling dangling side effects.
• budget_exhausted and partial_success_returned: explicit markers for budget exhaustion and partial success, used to count degradation signals.

Mandatory fields on every event

There is one principle behind field design: make it possible to stitch together all events belonging to the same attempt after the fact. That requires at least five fields running through everything: run identifier, intent identifier, step sequence number, attempt identifier, and idempotency key. Without the intent identifier you cannot reuse idempotency keys on replay; without the attempt identifier, retry counts get double-counted; without the idempotency key, the "search backwards for duplicate side effects" query I keep emphasizing simply cannot be done.

Second, store the decision rationale alongside the event. Budget allocation events must carry the allocation rule, circuit breaker events must carry the failure count, backoff events must carry the upper bound and the actual jitter applied. Record only results and not rationale, and post-mortem analysis reduces to guessing; guess wrong once and you derive a wrong causal conclusion, then tune a parameter that should never have been touched.

The event stream is the primary material for post-mortems

With a complete event stream, reviewing a production incident changes from "read logs and guess" to "pull up a timeline". The difference is stark during incident review: with events you can see at which step the budget ran dry, which retry consumed the last of the quota, and why the breaker opened at that moment; without events you have a pile of disconnected error logs and have to infer causality from timestamps.

A Triage Path for a Production Incident

Once the mechanisms are in place, what really tests them is whether you can locate a problem quickly on the day it happens. Below is a triage path to execute in order, where each step corresponds to a mechanism from one of the earlier sections.

Triage first, dig deeper later

The first question is always scope: is it all tasks failing, some tasks failing, or just slow and expensive? This judgment shrinks the search space by an order of magnitude. Total failure points first at dependencies and configuration (gateway, auth, quotas); partial failure points first at a specific tool or a specific parameter combination; slow-but-not-failing points first at budget allocation; expensive-but-not-slow points first at retries and loops.

The second question is time alignment: when did the incident start, and does it line up with the most recent release, configuration change, or downstream change. If it does not line up, stop reading code and go look at the change log on the dependency side.

Localization actions for three high-frequency failure modes

Duplicate side effects: take the anomaly order number or target identifier the user reported and search backwards for all calls carrying the relevant idempotency key. If more than one success record shows up, the problem is in the idempotency layer. If there is only one but the user saw it twice, the problem is in the downstream notification mechanism, not in the Agent. The repair directions for the two cases are completely different, so this distinction must be settled before you touch anything.

Broad timeouts: aggregate budget-exhausted events by tool and see whether all tools hit the ceiling at once (pointing at global budget settings or slower model inference) or only a specific tool did (pointing at a distorted latency profile for that tool or downstream degradation). At the same time, check the deadline propagation chain: if the remaining timeouts across steps of the same run do not line up, the budget was reset mid-run.

Compensation did not run, or failed: go straight to the ledger and look for records with a success status that have no corresponding compensation outcome. Those records are the dangling side effects, and their count is the actual blast radius of this incident — that number should be written down explicitly in the review rather than waved past as "some tasks were affected".

Distinguish a mechanism that failed from a mechanism that never fired

The last step is judging whether the mechanism itself is broken. A breaker that did not open has three possible explanations: the failure rate genuinely never reached the threshold (not an incident), the threshold configuration is wrong (the mechanism failed), or the breaker state was overwritten by a health memory held elsewhere (a mechanism conflict). The way to tell them apart is to look at the statistics within the same time window: a high failure rate with no breaker trip is a configuration problem; a tripped breaker with tasks still failing means the breaker's feedback was not passed correctly to the model.

This distinction matters because the two problems are handled in opposite ways — a configuration problem means tuning parameters, a feedback problem means changing the error contract. And if the latter is misdiagnosed as the former, the team will spend weeks repeatedly tuning a threshold that was never broken.

Which Metrics Deserve Alerts

The easiest mistake to make with reliability alerting is to configure too much of it, until every alarm is noise that gets ignored. The effective approach is to alert only on situations that clearly require human intervention, and leave everything else to dashboards.

The four metrics that must alert

Compensation failure handed to a human: every single handover represents a user-visible inconsistency that has not been resolved. The count of such events should be close to zero, and any occurrence should have someone follow up.

Circuit breaker open for a long time: note "for a long time", not simply "open". A single open is normal protective behavior; staying open means the downstream really is broken, and it needs a clearly named owner to follow up. A reasonable threshold is "continuously open for more than N minutes".

Spike in idempotency key conflicts: a spike in deduplication collapses means upstream is submitting duplicates, or some call path is producing unintended repeats. This kind of change usually shows up before user complaints do.

Spike in budget exhaustion rate: it reflects a sudden widening of the gap between budget settings and actual demand. Spikes are usually caused by upstream slowdown or more complex planning, which makes it an excellent leading indicator.

Things that should not alert

Short-lived swings in the single-call timeout rate, occasional rises in retry counts, and momentary breaker opens — none of these three should alert on its own. They are the system's normal breathing, and alerting on them only desensitizes people to red. The reasonable handling is to give them a baseline and a deviation threshold, and escalate to an alert only once the deviation persists for a while.

It is worth emphasizing slicing by dimension. An overall no-progress step ratio of a few percent does not mean there is no problem; slicing by task type may reveal one type sitting at thirty percent — and that is the real problem, and it points at planning quality rather than reliability mechanisms. Mixing every signal into a single total is the most common source of distortion on a reliability dashboard.

How to interpret a metric once it moves

More important than the metric itself is the interpretation move. The same metric rising may mean the mechanism is working correctly, or it may mean the system is genuinely degrading, and the two demand opposite responses. Below are the combinations that are easiest to confuse.

Retry amplification factor spikes — first check whether it is concentrated in a single tool. Concentration in one tool means that dependency is flapping and it has nothing to do with Agent logic; an even distribution across all tools more likely means the budget is too tight and each call only gets one attempt. The fixes are to raise the retry budget and to raise the per-call budget respectively — exactly opposite directions.

Circuit breaker trigger rate rises — distinguish "protection is working" from "the threshold is wrong". The way to judge is to look at task success rate while the breaker is open: if success rate goes up after the breaker opens, the breaker is effective; if tasks still fail en masse while it is open, the breaker is not cutting in the right place, and what should change is the content fed back to the model, not the threshold.

Compensation success rate drops — first check whether weak compensations dominate, or whether compensations are falling outside the valid time window. If "stale compensation" starts appearing in volume, that is a design problem: it means irreversible actions are scheduled too early, or the business process itself permits execution durations that are far too long. That case needs a design change; tuning parameters will not help.

No-progress step ratio rises — this metric is frequently misread as "the model got dumber". In reality you should first rule out two technical causes: whether the state summary design actually covers the real basis for progress, and whether tool returns are so impoverished that the model cannot judge progress at all. Much of the perceived "the model got dumber" is in fact a decline in tool return quality.

Spike in idempotency key conflicts — the most common real cause is upstream submitting the same request twice, for instance a user double-clicking in the UI. Such duplicates can be collapsed, so a rising conflict count is not itself bad. What genuinely deserves alarm is deduplication failure: if one business intent produces two different keys, unstable fields have leaked into the key recipe and deduplication is quietly failing.

Budget exhaustion rate rises — first separate "individual steps got slower" from "the number of steps grew". Slower individual steps point at dependency degradation or a distorted latency profile; more steps point at declining planning quality or more no-progress loops. The repair locations are completely different: the former on the tool side, the latter in planning and circuit breaking.

The dangling side effect count is the most serious of these metrics, because it maps directly to user-visible inconsistency. Any non-zero value should have an explicit disposition, even if that disposition is "accept for now and add to a watchlist". Leaving it on a dashboard indefinitely without acting on it is worse than not counting it at all — it trains the team to become desensitized to that number.

Behind all of these interpretations runs one principle: first ask "is the mechanism working correctly, or is the system genuinely degrading", and only then decide the action. That order cannot be reversed, because misreading "the mechanism is working correctly" as "the system is degrading" most typically leads a team to tune thresholds that were already correct, and thereby genuinely break the system.

How to Layer a Dashboard

Overdoing observability creates its own problem: putting every metric on one board means no board ever gets fully read. The effective approach is to layer by the question being answered rather than by data source.

The real-time layer answers "is it healthy right now", and holds only the few criteria that require human intervention: compensation failures handed to a human, breakers open for a long time, spikes in idempotency conflicts, and the budget exhaustion rate. The design goal for this layer is to let a human decide within thirty seconds whether to act immediately; nothing else belongs here.

The trend layer answers "have we been degrading recently", and holds the ratio-type metrics: retry amplification factor, no-progress step ratio, average open/close count, compensation success rate, and dangling side effect count. This layer does not need real-time response, but it does need to be read weekly, because a lot of degradation is slow.

The capacity layer answers "how much longer can we hold", and holds cost budget consumption, quota usage, concurrency peaks, and rate-limit hit counts. This layer serves capacity planning and budget parameter tuning, and is usually sliced by tenant and by resource group.

Slice dimensions also have to be defined up front. At least three: by tenant, to locate tenant-specific problems; by tool or resource group, to locate dependency problems; and by task type, to locate planning quality problems. The last dimension is the most easily overlooked and the most valuable, because it can expose severe local problems hidden behind an "everything looks fine overall".

Migration Path: From Running Naked to Having Mechanisms

If your Agent currently has none of these mechanisms, the hard part is not technical, it is ordering. Pushing all six areas at once loses the team its patience in the first month. A sensible path is: stop the bleeding first, then constrain, then recover, then verify.

Stop the bleeding: start with the most irreversible tools

Do not start with timeouts. Start with irreversible tools. Put a human confirmation gate in front of them, or simply disable execution for now and return a message. This move has the fastest payoff and the lowest risk, and it immediately plugs the most expensive holes — duplicate charges and misdirected notifications cost far more than a failed task.

In parallel, do something that costs essentially nothing: add a unified event log to every tool call, with fields taken from the mandatory list above. Even with zero mechanisms in place, this single change makes the effect of every later improvement observable.

Then constrain: budgets and retry gates

Step two introduces a global budget object and tool tiering. This step is purely constraining, it does not change business semantics, and therefore does not need sign-off from the business side — but it does change how the product feels (some tasks will fail faster). Explain that up front and make explicit failure friendly enough that it is not read as system instability.

Introduce idempotency keys at the same time. This step should be done tool by tool rather than rolled out all at once, because each tool's key recipe and its cooperation with the server need to be confirmed individually. The typical outcome of a full rollout is that server-side deduplication does not take effect while the client has already stopped retrying, so previously masked duplicate problems start surfacing and it looks as though the change introduced an incident.

Then recover: the ledger, compensation, and resumption

Only step three is the ledger and compensation. It touches persistent storage and recovery logic with a large change surface, but its prerequisites — tool tiering, idempotency keys, the event log — are already in place from the earlier steps, so the risk is manageable.

Start in scenarios that affect only a single resource group, verify the recovery path, and only then expand. Also test the scenario "crash immediately after the side effect executes" on purpose: that is the moment most prone to error in ledger design and the hardest to cover in ordinary testing.

Run both tracks for a while

Letting the old and new mechanisms run in parallel during the transition is valuable: the new mechanism makes the decision but the outcome is only recorded, not acted on, purely for comparison. This phase answers one key question — how many tasks that would have succeeded does the new mechanism reject. If that number is high, the thresholds are too tight; if it is near zero or even negative (meaning the new mechanism rescued tasks), the thresholds are sound.

The cost of the dual-track period lies mainly in inference and implementation rather than runtime, so it does not need to run long — a handful of runs of each typical task type is enough. Before ending the dual-track period, condense the comparison results into a one-page conclusion to serve as the basis for parameter decisions and for explaining the change to the team.

How to prove the migration worked

The hardest part of reliability work is that its effect is hard to prove, because it mainly improves things that did not happen. A workable proof comes in three kinds of evidence.

The first is mechanism activation rate: how many times the breaker was triggered, how many times compensation ran, how many times the retry budget halted a retry. Curves that grow from zero show directly that the mechanisms are working rather than lying dormant in the code.

The second is invariant violation count: the number of times invariants were violated in chaos testing. It should decline monotonically as the migration proceeds, which demonstrates the point better than any improvement in success rate — because success rate itself is too heavily influenced by business fluctuation.

The third is improvement in tail metrics: P99 latency, retry amplification factor, cost per success. These numbers usually move by a modest amount, so the emphasis must be on trend rather than absolute value, and comparisons should be sliced by task type to avoid distortion from mix shift.

Express Failure as a Contract, Not a String

Every mechanism in the sections above depends on a single premise: failure has a classifiable shape. If a tool returns only a string, the orchestration layer cannot tell apart "a network hiccup, worth retrying", "the parameters were wrong, retrying ten thousand times is still wrong", "the downstream is rate limiting us, wait as long as it says", and "this operation may already have taken effect, only the response was lost". The less information the model gets, the more erratic its behavior becomes, and in the end it usually degrades into blind retrying or giving up outright.

A usable error contract has to carry at least five kinds of information: an error category used to decide between the retry path and the degradation path; whether the error is retryable; a suggested wait duration; whether it may already have taken effect, used to decide whether to trigger compensation; and a short human-readable explanation for the model. The fourth is the one most easily missed: it is the only piece of information that crosses from a transport-level error into a business-semantic error, and the only basis on which a compensating transaction can be triggered correctly. For a write call whose connection reset, is it "definitely not executed" or "may already have executed"? Only the tool adapter can answer that from the operation type; no amount of inference at an outer layer is reliable.

The other side of the contract is the feedback format aimed at the model. The error shown to a human and the error shown to the model should be separate: a human needs "the downstream service is temporarily unavailable, please try again later", while the model needs "tool X is unavailable, do not retry, choose an alternative path or tell the user". If the latter is written as a full natural language sentence, the model will readily read it as "trying again might work"; only when it is written as structured text with explicit instructions will the model actually comply. This detail looks trivial, yet it determines whether circuit breaking and degradation can truly take effect — a breaker that opens without the model knowing has not opened at all.

There is one more engineering requirement: the error category must stay consistent across the entire call chain, from the tool adapter to the orchestration layer to the model feedback, and must not degrade into a generic exception midway. In implementation you can use a typed error object plus an explicit classification enum, with conversion points in only two places — crossing a process boundary during serialization, and crossing a language boundary. Do not write a broad catch in an intermediate layer and stuff a string into it; that discards everything you just built at the most fragile position in the architecture, and such loss is very hard to notice because the functionality still appears to work.

The contract should also carry an attempt identifier. Different attempts of the same intent need to be correlatable through it, so that the retry budget, the breaker counts, and idempotent deduplication all share one statistical basis. Without this identifier, retry-related metrics can only be counted by log scraping, and you end up with numbers like "this tool probably retries a lot every day" that nobody can act on.

The Eight Most Common Anti-Patterns

Almost every item below has genuinely happened in the wild, and most of them throw no exception at all — they just quietly let reliability leak away a little at a time. Their shared trait is this: every individual piece of code looks reasonable in isolation, and the problem only appears once they are combined.

• Passing a fixed timeout value down every layer. Every layer is configured with three seconds, so a twenty-layer chain can hang for an hour before giving up. The correct approach is to pass a deadline instead, so each layer inherits the remaining budget from upstream. This one is stubborn because it is always correct in single-layer tests.
• Parallel branches each claiming the full parent budget. This looks like the safest option but in fact lets the total budget inflate with the branch count: twenty-way parallelism means twenty times the budget, and everything ends up stuck on some upstream rate limit. The fix is to split the remainder by weight and impose an absolute ceiling.
• Restarting the clock on every retry. If each retry measures its timeout from zero, that amounts to unlimited time and makes the retry budget meaningless. The fix is to have retries reuse the same deadline, and to deduct backoff time from the budget as well.
• Binding the idempotency key to the attempt. A fresh UUID per retry means the server cannot deduplicate, so "retry three times" equals "execute three times". The fix is to bind the key to the intent identifier produced during planning, and to confirm the server really stores the key-to-result mapping.
• Using the full transcript for no-progress detection. The model's redundant restating counts as progress, so the detector never fires. The fix is to hash only the set of external facts plus the to-do list, and exclude whatever the model says from the judgment.
• Writing compensation logic only in application code. Once the process dies, nobody knows what was done and recovery becomes guesswork. The fix is to introduce a write-ahead ledger and lift compensation out of the flow into an independent recovery phase.
• Rate limiting by tool name only. When several tool names point at the same cluster, per-tool limiting amounts to no limiting at all and concurrency gets multiplied by the number of tools. The fix is to rate limit by downstream resource group and have each tool declare which group it belongs to.
• Wrapping partial success as success. Adding "most of the operation completed" to the return while the status field is still success lets the caller advance the workflow, and the inconsistency is deferred to a more distant place. The fix is an explicit partial success state plus a recoverable marker.

If you want to check these eight quickly in a review, the criteria can be compressed into three questions: is this retry safe (are the side effects idempotent), is this failure explainable (does the user know what happened), and does someone own this account (can the state still be recovered after a failure). An implementation that can answer all three immediately will almost certainly not have the eight problems above; anywhere it cannot answer one, an anti-pattern is very likely already buried.

How to Set Timeout Budgets: Let Data Decide, Not Intuition

The part of the whole timeout mechanism most likely to become pure form is where the budget numbers come from. Picking "thirty seconds" and reusing it across the entire chain looks uniform, but in reality it is neither justified nor adjustable. The genuinely maintainable approach is to let the budget grow out of the latency distribution.

Start from the latency distribution

Every tool should have a latency profile containing at least p50, p95, p99, and a sample count. That profile serves two purposes: providing the basis for budget allocation, and telling you whether the tool is in an abnormal state. A p99 above twice the historical baseline usually means the downstream is degrading, and at that point a budget allocated on the baseline makes every single call time out.

There is an easily overlooked methodological point here: profiles must be computed separately per tool and per parameter scale. A full query with no filter and a precise single-row query can differ by an order of magnitude in latency, and a p99 computed over the mixture is worthless. Parameter scale can be stratified with proxy indicators such as estimated result row count or request body size; a crude stratification is still better than mixing everything together.

Quantiles Are Not Guesswork, but Do Not Copy Them Blindly

The common advice is "give each call two times p99". That number has some justification, because it leaves a factor of two of headroom for the long tail without inflating the budget so much that multi-step tasks become impossible. But it is a starting point, not a conclusion. The real criterion is what you want to happen when the downstream degrades: if you want fast failure so resources go to healthy paths, stay close to p99; if the downstream is the only source of data and there is no substitute, give it generous headroom and even permit retries.

A more practical calibration method is to watch the budget exhaustion rate. If this ratio is too low, the budget is too generous and runs are routinely dragged out for no reason; if it is too high, the budget is too tight and even normal tasks cannot finish. Setting a target of a few percentage points and adjusting weekly is more reliable than picking a number once and living with it.

The Pareto Between Budget and Success Rate

The deeper question is the trade-off between budget and success rate. Raise the budget from ten seconds to sixty and the success rate does go up, but with sharply diminishing returns: past a certain point, timeouts come almost entirely from genuinely slow tasks rather than bad luck, while the cost of waiting three times as long is real and immediate. There is a knee point on that curve, and finding it is far more valuable than chasing "higher is better".

The criterion for locating that knee is usually not a technical metric but a product constraint: if the user can tolerate waiting an extra ten seconds on a single failure, the budget should skew to the long side; if users expect an immediate response, then fail fast and return a degraded result rather than leaving them staring at a three-minute progress bar.

Turn Tool Calls into Durable Tasks

The retries and compensation discussed above all assume the call is immediate and the result arrives within that call. But in many real scenarios a tool call is naturally a long-running task that can execute asynchronously: kicking off a training job, processing a batch of documents, calling an endpoint that takes tens of minutes. If you keep applying a synchronous timeout mindset to these, you are forced to choose between waiting and giving up.

Queues Give You Free Redelivery

Delivering the call to a message queue buys you a property a synchronous design cannot have: the delivery itself is durable. If a consumer times out and exits, if a machine reboots, if the orchestration process crashes, the task is not lost and consumption resumes after restart. This effectively moves part of reliability from your application code into the infrastructure, and that part of the infrastructure is usually the more reliable half.

But queue semantics have to be understood before they can be used well. Most queues offer at-least-once delivery, not exactly-once: a message may be delivered repeatedly, and it may be redelivered because the consumer crashed after processing but before acknowledging. This semantic and idempotency are a natural pair — the queue guarantees "nothing is lost", idempotency guarantees "a repeat is harmless". Without idempotency, at-least-once means "guaranteed repeat".

Visibility Timeout Is the Timeout of This Model

Under the queue model, the meaning of timeout changes: it is no longer "give up if no result arrives" but "processing exceeded the visibility timeout, so the message will be redelivered". That distinction is critical, because redelivery happens automatically with no application-level decision, but it also means the processing logic must be safe to interrupt at any moment.

In practice, give consumers an idempotent handler and an explicit completion marker: acknowledge the message only when processing genuinely finishes, and renew the visibility timeout periodically while it runs. That way a long task can be interrupted safely without being redelivered over and over. The renewal logic must be separated from the business processing, otherwise a single exception in the business code destroys both at once.

Status Queries Beat Polling for Results

There is a common mistake in how results of async tasks are fetched: the client polls a result endpoint over and over. A more robust shape is to return a task handle on submission and pair it with status subscription or a callback. The problem with polling is that it splits one long wait into many short requests, still burns budget, and cannot distinguish "still running" from "stuck" — two entirely different states.

The recommended approach is to have the status query return structured stage information: queued, running (which step, roughly how much longer), completed, failed (with a reason). This is useful to the user, and it also lets the layer above distinguish "keep waiting" from "start compensating now" — and the basis for the latter is the failure reason, not elapsed time.

Evolving Tool Contracts: Do Not Let a Rename Kill a Running Run

Tools are alive: parameters change, return structures change, semantics get adjusted. Meanwhile a task in flight can last minutes or even tens of minutes, and any single change along the way can make a task that would have succeeded fail halfway through. This section is about the compatibility constraints on contract changes.

Running Tasks and Tool Versions

The most basic requirement is versioned reads of return values: a new version may add fields, but it must not delete existing fields or change their semantics, because old tasks are still parsing according to the old structure. The most dangerous change is not deleting a field but changing its semantics — for instance turning a status field from a string enum into a boolean, or changing the amount unit from yuan to cents. Such a change raises no error; it just makes old tasks quietly compute the wrong answer, which is worse than crashing outright.

The pragmatic approach is to tier contract changes: adding an optional field is safe; adding a required parameter is breaking; changing semantics requires a dual-write transition period; deleting requires first confirming that no old task is still running. For high-risk changes you can introduce a short-lived contract version number so the orchestration layer records the version when the task starts and verifies it when the task ends, turning an incompatibility into an explicit failure rather than a silent wrong result.

Contract Tests as Regression

A tool's reliability is also the quality of its contract. A good contract test suite should cover: structure conforming to schema, error paths returning the expected error category, idempotency keys genuinely being accepted by the server, and results being consistent under repeated calls. Hooking that suite to every tool change prevents regression better than any documentation ever will.

The piece most likely to be skipped in contract tests is the error path. People habitually test the happy return, but Agent reliability depends precisely on whether error information is well-formed. To a framework, a tool that returns 500 and a tool that returns a structured error are two completely different things, and the former can only be treated as an unknown fault.

Canary and Dual-Version Coexistence

When a semantic change is involved, the correct rhythm is to run both first, then switch: have the old and new contracts live simultaneously, split traffic to the new version by proportion, and use invariant-based observational metrics to decide whether it is safe to go to full rollout. The prerequisite for this process is that both sides can be observed independently — meaning the idempotency keys and call events described earlier must all be tagged with a version marker.

Many teams only discover the problem after full rollout, by which time the old contract is already decommissioned and the rollback window has closed. The extra cost of a dual-version coexistence period is manageable, and what it buys is exactly the option to back out at any time — for a system whose reliability mechanisms are themselves changing, that option is worth a great deal.

Change Management for the Reliability Mechanisms Themselves

A risk that is rarely discussed is this: the reliability mechanisms themselves become a source of failures. A misconfigured breaker threshold, a misallocated budget, a wrong compensation order produce outages that are worse than having no mechanisms at all, and much harder to diagnose.

Threshold Configuration Is Especially Dangerous

Numbers like breaker thresholds, retry counts, and budget seconds look like tuning parameters, but they are actually safety boundaries. When they are wrong they do not error immediately; they make the system behave in the opposite way from what was intended under specific conditions. Once such configuration is live it is very hard to attribute, because "the breaker did not open" and "the breaker opened too often" both look like nothing more than a state transition in the logs.

These settings should therefore be treated like a database isolation level: with an explicit valid range, change review, canarying, and one-click rollback. At the same time, attach observability items to the critical thresholds — breaker trigger rate, average open/close count, budget exhaustion rate — and alert as soon as any of them leaves the expected range, rather than waiting for a user complaint.

Mechanism Changes Need an Observation Window

After changing a breaker policy, watch at least one full business cycle before drawing conclusions. A common incident goes like this: some adjustment briefly raises the breaker trigger rate (because the threshold was tightened), the team concludes the change broke things and reverts it — and all the while, the real problem was a different failure that got masked. Setting a minimum observation window and making no second adjustment inside it is far more reliable than chasing a fast response.

The Rollback Path Itself Must Be Tested

The ability to "flip a mechanism off in one click" should be rehearsed regularly. Many teams only discover, at the moment they truly need a rollback, that the configuration item no longer takes effect, or that the rollback script depends on a service that has already been decommissioned. Rehearsal is cheap, and the payoff is having a button you can press immediately at the worst possible moment.

Fairness and Priority in Multi-Tenant Settings

In a single-tenant setting, rate limiting solves the problem of protecting the downstream. In a multi-tenant setting it must also solve mutual interference: one long task must not hold concurrency permits continuously and push other tenants into timing out while they queue.

Tiered Quotas

The effective approach is tiering: a global ceiling protects the downstream, a per-tenant ceiling protects fairness between tenants, and a per-run ceiling protects against the inflation of a single task. The three layers are connected by a borrow-and-return mechanism — an idle tenant's quota can be borrowed temporarily by others, and the lender drops to low priority so that it can reclaim its quota first the moment it needs it.

Interactive Requests Take Priority over Batch Work

Long-running batch jobs and second-scale interactive requests sharing one pool will disappoint someone no matter how you tune the rate limit. The correct answer is physical isolation: separate rate limits, separate queues, and where necessary separate downstream resources. Priority inside a shared pool can only improve experience, it cannot guarantee anything — and for interactive requests, latency predictability matters more than average throughput.

This also affects how you design task abortion: when a low-priority task is squeezed by a high-priority one, it should be degraded (narrower scope, lower precision) rather than interrupted. A batch job that has been squeezed in half is usually better than one that was killed, because the former still has a deliverable.

End to End: An Order Adjustment Task

Run all the mechanisms above together once and the abstractions become concrete. Consider this scenario: an Agent receives the instruction "change the delivery time for order A to tomorrow morning and notify the buyer", where the order lives in an internal system, the notification goes out by email, and the two belong to systems owned by different teams.

Planning Phase

The planner outputs two actions: look up the order, change the delivery time, then send the notification. The orchestration layer does three things here. Ordering: the irreversible email is scheduled last, with an explicit dependency on the first two steps succeeding. Budget allocation: forty seconds overall, with the order lookup getting thirty percent of the remainder and no more than eight seconds, the delivery-time change no more than ten seconds, and the rest reserved for the notification and inference. Contract validation: both tool names are in the registry and the parameter formats are valid.

At the same time it writes the ledger: two records, one for the delivery-time change (with the compensating action "restore the original delivery time") and one for the email (no compensating action, marked irreversible). This step completes before any side effect is executed.

Execution and Fault Injection

The order lookup succeeds. For the delivery-time change, the downstream returns a timeout. What happens here is where the value of the mechanisms shows most clearly:

The orchestration layer checks the remaining budget — twenty-two seconds left, while this attempt was budgeted ten seconds, so there is enough. It therefore issues a second attempt keyed by the idempotency key, which succeeds two seconds later. At the same time, the compensation budget was not eaten up by those two attempts, because budget decisions are computed on a rolling basis rather than carved up in one shot.

Next comes the email. The mail service is rate limiting at this moment and returns an explicit retry-after duration. The system adopts the wait duration given by the server rather than its own backoff, and before waiting it checks the remaining budget — only four seconds left, not enough to wait again, so it returns "delivery time updated successfully, but notifying the buyer failed" directly. The result is incomplete, but it is honest: the completed part is explicitly labeled, the incomplete part comes with a reason, and the email record sits in the ledger in a pending state so it can be reconciled and resent later.

If Compensation Fails

Switch to a worse scenario: after the delivery time is changed, every subsequent step fails, and the assessment concludes it should be undone. Compensation is triggered.

The system works in reverse order, first sending a correction notice reading "your delivery information is about to change" (if one was already sent), then restoring the original delivery time. The restore action uses the same idempotency key, so repeating it is safe. If the restore action fails three times in a row, it does not keep retrying; instead it marks the record as handed to a human and pushes a complete handover package to the on-call engineer — including the ledger record, the original order information, the compensating actions already attempted along with their failure reasons, and the subtask that may still be running.

Final Delivery

What the user receives is a structure with an explicit status: the delivery time was updated (with the new value), the buyer was not notified (with the reason and a recommendation), and the current state is recoverable such that retrying will not update the order twice. This result carries far more information than "operation failed", and it tells the user what to do if they want to retry later.

Looking back, this one task triggered seven or eight mechanisms from start to finish: budget inheritance and rolling allocation, idempotency key construction, adoption of a server-provided backoff instruction, insufficient-budget detection, write-ahead ledger persistence, partial success reporting, and a circuit breaker on another branch. Their common trait is that not one of them requires a judgment from the model — the mechanisms constrain the model, and the model makes choices within those constraints. That is the key dividing line that moves an Agent toward an operable state.

Mapping Onto Existing Frameworks: What to Adopt and What You Must Write Yourself

Some of the mechanisms in this article are already covered by frameworks and workflow engines, and there is no need to rebuild them. But some others you have to write yourself no matter which framework you use. Drawing that line clearly saves a great deal of wasted engineering effort, and it also avoids the single most dangerous misjudgment: assuming that because a framework offers checkpoints you have reliability.

What Frameworks Usually Already Solve

The three most typical cases. First, state persistence and checkpoints: mainstream graph-based orchestration frameworks all provide the ability to write each step's state into external storage and resume from any checkpoint, and deterministic replay in workflow engines is the same category of capability. They solve "where do we continue after a crash" and "a long task need not rerun from the beginning".

Second, automatic retry of a single call: including repeated execution by count or by a backoff policy, visibility timeouts, and heartbeat-based liveness detection. This layer effectively gives you at-least-once delivery semantics for free, which is especially useful for async tasks.

Third, call chain tracing: mainstream agent frameworks all integrate tracing and can produce the full call timeline, the arguments and returns of each tool call, and the parent-child relationship of nested calls. It is one of the best data sources for an event stream and can save an enormous amount of manual instrumentation.

What Frameworks Cannot Solve and You Must Write Yourself

The gaps concentrate in three places, and those three happen to be the core content of this article.

Construction and propagation of the idempotency key. A framework knows "the call failed, you can retry", but it does not know which intent this call belongs to, nor who should generate the idempotency key. Whether the key is bound to the attempt or to the intent is a business-semantic decision that no framework can infer. The root cause of the overwhelming majority of duplicate side effect incidents is right here.

Compensating transactions. Recoverable execution solves "rerun", but many side effects cannot be rerun, only undone. "Which actions are undoable, in what order to undo them, and what to do when undoing fails" is business knowledge that the tool provider has to declare; the framework does not have that information.

Cross-process resource descriptors. Budgets, idempotency keys, and ledger identifiers all need to propagate into the other process or vendor where the tool lives. Propagation across such boundaries usually has to be implemented explicitly at the application layer: adding a request header, writing a field, or injecting it uniformly inside the tool adapter.

How Far the Tool Protocol Layer Gets You

As tool calls are standardized into a cross-process protocol, request timeouts and error codes gain a uniform expression, and that genuinely reduces some cross-service implementation cost. But at the semantic level there are still several things the application layer must fill in:

• Mapping error codes to retry policy. The protocol will tell you "internal server error", but it will not tell you whether this operation may already have taken effect. That determines whether you enter the compensation flow, and it can only be inferred from the operation type.
• Allocation of the timeout budget. The protocol can carry a timeout duration, but carrying the remaining budget and distinguishing the hierarchy between a call and its sub-calls has to be done by the application layer.
• Convention for the idempotency key field. The protocol is responsible for transporting the arguments, but who constructs the idempotency key, which field it lives in, and by what rule the server deduplicates are contract terms you have to negotiate.

Once these three conventions are settled, cross-service reliability genuinely holds. Missing any one of them leaves a gray zone where "the protocol layer says the call failed, but the application layer does not know whether to retry".

One Rule of Thumb

You can split it with a single sentence: anything about business semantics, write it yourself; anything about the execution skeleton, reuse what already exists. The execution skeleton covers state storage, retry loops, call sequencing, and task delivery. Business semantics covers whether an operation can be repeated, how to undo it, and who is accountable for its correctness.

The practical value of this standard is that it blocks a very common form of over-engineering: pouring effort into building a custom distributed task executor, while what actually got skipped is working out, for each irreversible operation, how it should be undone. The latter is the main source of production incidents.

The Tool Registry: Centralize Every Declaration in One Place

Many of the mechanisms mentioned earlier in this article ultimately depend on the same metadata: which category a tool belongs to, how its idempotency key is constructed, what its compensating action is, its latency profile, its resource type, and who owns it. If that information is scattered across code, configuration, and documentation, any missing item silently disables the corresponding mechanism. Centralizing it into a registry is the precondition for keeping the whole system maintainable.

What a Registry Entry Should Contain

At a minimum, seven items: idempotency category (one of three classes, deciding whether automatic retry is allowed); idempotency key recipe (describing which parts compose the key, who generates it, and which field it is propagated into); compensating action (present or not, plus the strength tier of the compensation); latency profile (p50/p95/p99 and sample count); resource ownership (which downstream resource group it points at, used for rate limiting); side effect tier (how visible it is externally, deciding whether extra gates are needed); and owner (who to look for when something breaks).

The "idempotency key recipe" item is the one most often written as a vague description. A qualified version should be able to drive implementation directly: the key is a concatenation of tenant identifier, intent identifier, tool name, and a digest of the normalized arguments; the intent identifier is generated during planning and propagated downward; and the fields to strip during normalization are listed in configuration. Writing only "use the request ID for idempotency" does not qualify, because the request ID changes on every retry.

Treat the Registry as Code

It is a runtime dependency, not documentation. So it should be versioned, reviewed, contract-tested, and shipped through the same release process as the rest of the code. Every change has to answer one question: how will an old task that is already running read this entry. Adding fields is usually safe, while changing field semantics generally requires a dual-write transition period.

A more valuable step is to generate the tool documentation in the prompt from the registry. Earlier I mentioned that the root cause of tool hallucination is inconsistency between the description and the registry; generating both from the same data eliminates that inconsistency physically. This is a small investment with immediately visible payoff, and it should be scheduled at the very front of your reliability work.

Use the Registry for Resource Governance

With a unified declaration of resource ownership, many governance actions can be automated: rate limits are configured per resource group, monitoring aggregates per resource group, capacity assessment counts call volume per resource group, and fault drills select injection targets per resource group. There is no need to maintain a second mapping table, and therefore no configuration drift.

One last practical suggestion: give the registry an integrity check that runs at startup, verifying that every entry has all required fields and that the idempotency category and the compensation declaration are self-consistent (for example, an entry declared irreversible with no compensation declaration and not marked as avoided-by-prerequisite). This kind of check catches a batch of configuration errors at deploy time, whereas otherwise they typically are not discovered until the first incident after go-live.

Rollout Order: Advance by Input-Output Ratio

Finally, here is an executable order of advance. The first two phases cost almost nothing and pay off enormously, and should not be skipped.

Week one: a global deadline and hard timeouts

Establish a single budget object at the run entry point, have every tool call inherit the remaining budget from it instead of picking its own timeout, and give each tool both a time-to-first-byte timeout and an idle timeout. This step alone usually eliminates most "long periods of no response" tickets.

The acceptance criterion for this phase is unambiguous: for any tool call you can look up its deadline in the event stream, and that deadline has been inherited all the way from the run entry point. If you cannot, some layer in between reinitialized it. While you are at it, stand up latency profile collection for tools as well — every later budget decision depends on it, and starting one week earlier gets you trustworthy quantiles one week earlier.

Week two: tool tiering and retry gates

Tier the tool registry into three classes and explicitly mark irreversible tools; bind idempotency keys to intent; disable automatic retry for irreversible tools; and add a retry budget ratio. This step will surface a batch of tools that "have actually been executing repeatedly all along", which is a benefit in the form of discovery.

The way to verify this step is to check the idempotency key conflict count: once tiering is in place, tools that have never collapsed a duplicate request should start showing collapse counts. A long-term zero usually indicates a problem with key construction rather than a business reality in which duplicates just happen not to occur. Also spot-check the call chains of irreversible tools to confirm they genuinely have no automatic retry path.

Month one: the ledger, compensation, and the three budgets

Introduce a side effect ledger with write-ahead persistence, complete the compensation metadata for every tool with external effects, add the three budgets of steps, wall clock, and tokens along with explicit degradation, and schedule irreversible actions at the tail of the orchestration. This is the largest single change, so advance it category by category rather than all at once.

Advance it by tool category: start with the combination of read-only and idempotent writes, which carries almost no risk; then add writes that have compensating actions; and only touch irreversible actions last, because only at that point are you genuinely changing the external world. In the reverse order, the very first change could cause an irreversible error, and at that moment the team's compensation chain has usually not yet been verified.

A quarter: chaos engineering and SLOs

Turn the invariant list into executable checks in CI; build the tail metrics dashboard; put rate limiting and adaptive concurrency in place per resource group; and only then consider multi-tenant fairness. The deliverable of this phase is not "no incidents", but rather that when incidents occur the damage is bounded and the aftermath is diagnosable.

Do not wait until this phase to start building metrics — the observability instrumentation has to be done in parallel with the first three steps. Otherwise, by the final quarter you will find that you cannot tell whether the first three steps had any effect at all, and can only report based on a feeling.

Boundaries and Costs

Finally, the boundaries of applicability of this whole approach need to be stated plainly, because reliability engineering itself has a cost.

If your Agent is a single round of Q&A with only read-only tools and no external side effects, then ninety percent of the mechanisms above are over-engineering. Only two things are worth doing in that scenario: a timeout on the inference call, and an overall step ceiling. Conversely, if your Agent sends messages, moves money, or changes production configuration, then every single item in this article is on your risk list.

There is a genuine tension between reliability and autonomy, and it should be acknowledged honestly. A circuit breaker lowers single-call success rate, since many calls that would have self-healed now fail fast. Compensation introduces new points of failure. Conservative default policies hand control back more often. All of these are trading peak experience for a tail floor; in scenarios where the far tail decides survival, that trade is usually a good one, but on short-lived, low-risk internal tooling it may not be.

Looking at where this is heading, three things are worth watching continuously. The first is recoverable execution becoming a default capability of orchestration frameworks, pushing checkpoints, ledgers, and compensation down out of application code and thereby lowering the duplicated build-out cost for every team. The second is standardization of deadline and error semantics at the tool protocol layer, making deadline propagation and structured errors a universal convention across languages and processes instead of something each framework defines on its own. The third is turning the budget into a static constraint, so the planner is budget-constrained at the moment it generates actions, rather than being cut off by a circuit breaker halfway through execution.

There is another frequently overlooked cost: the latency the mechanisms themselves add. Budget checks, idempotency key computation, ledger writes, and circuit breaker state lookups all sit on the critical path, and added together they are not necessarily zero. The practical approach is to make them "triggered on demand": read-only tools skip the ledger, the fast path skips some validation, and breaker state uses a local cache with periodic refresh instead of querying shared storage on every call. The risk of this route is that the fast path and the slow path behave inconsistently, so you must periodically force a trip down the slow path as a control, to stop the fast path from silently masking problems.

Another cost is procedural. An open breaker needs someone to follow up, a failed compensation needs someone to handle it, a threshold change needs someone to review it — all of which imply an on-call rotation and an explicit division of responsibility. Without procedural support, the mechanisms decay into a state of "we alerted and nobody looked", and at that point they are not merely valueless but actively harmful, because the noise they keep generating erodes the team's overall trust in alerting. So before shipping a mechanism, confirm who responds. That matters more than writing the code first.

It is equally worth stating clearly when not to do this at all. The criterion is not complexity but the product of "cost of a single failure" and "probability of failure". If the cost of a single failure is extremely low — say an internal read-only query tool that, if it fails, simply lets the user ask again — then the entire mechanism set is net negative. Conversely, as long as a single failure involves an externally visible irreversible action, or would contaminate the state of every subsequent step, the return on the investment is unambiguous.

There is one intermediate state worth calling out separately: low frequency but high cost scenarios, such as an annual settlement process that runs once a quarter. Its failure probability is not high, but the cost of a single incident may exceed the sum of all daily work. For these scenarios, the direction of investment should be the inverse of everyday traffic — the goal is not to lower the failure rate but to guarantee that after a failure someone can discover it, replay it safely, and trace it completely. Observability and recovery capability matter far more here than circuit breaking and rate limiting.

The final judgment, and the sentence this whole article wants to leave you with: Agent reliability will not come from a smarter model; it will come only from those mechanisms that sit outside the model and treat failure as inevitable. It is when your system assumes that tools will time out, downstreams will crash, models will loop, and processes will die that it finally starts to behave like a service that belongs in production.