Connecting Agents to Real Business Systems: SaaS Integration and Transactional Consistency

Introduction: The First Time an Agent Writes Real Data, the Nature of the Problem Changes

For most of the past two years, the conversation about Agents has been about what they can do: read documents, query databases, write reports, generate SQL, call a few APIs. What these scenarios share is that when an Agent fails, the worst outcome is a worse answer — the user asks again or fixes it by hand, and everything recovers. Let's call this class of system a "suggestion-type Agent" for now; its correctness criterion is whether the output is worth referencing.

The moment an Agent has write permissions, the nature of the problem changes. It can flip an order's status from "awaiting shipment" to "shipped", move a customer's money from account A to account B, send twenty thousand individual emails, leave a number on a contract. A wrong output can be rewritten; a wrong status means one more shipment, one more transfer, one more promise made to the outside world actually happened. The system turns from "a tool that generates content" into "an executor with side effects on the external world", and the questions you have to answer are no longer about how smart the model is, but about how permissions are divided, how transactions are managed, and how accountability works when something goes wrong.

This is also the rhythm at which many teams fall into trouble: the read-side integrations come out beautifully, the first version with write integrations runs, the second version starts having incidents, and once incidents appear you discover there is no rollback at all — because downstream is a third-party SaaS that offers neither transactions nor undo APIs. An Agent that has only read data and gets something wrong wastes time at worst. An Agent that has written data and gets something wrong produces an external fact that cannot be taken back.

1. Start With Danger Tiers: Not All Writes Are Equally Irreversible

Many teams get stuck on "should we give the Agent write access at all" because they treat it as a yes-or-no question. The right question is actually: how hard is it to undo this write? Sort tools into four tiers by reversibility, and the governance measures differ completely for each. Mixing them together in one discussion can only produce useless conclusions like "ban everything or allow everything".

Tier	Typical action	Consequence when wrong	Viable governance
In-place rewritable	Editing a note, a tag, a draft title	New value overwrites old; bounded loss	Pre-write snapshot + rollback audit
Logically reversible	Cancelling an order, revoking an authorization, issuing a refund	A reverse operation exists, but downstream may already have observed it	Compensating transaction + reversal audit
Financially irreversible	Payment, internal transfer, refund to an external card	The money has already moved; only reconciliation or dispute remains	Mandatory human confirmation + amount threshold + two-person review
Externally irreversible	Sending email/SMS, webhook notification, submitting a regulatory filing	The outside world already received it; you cannot pretend it did not happen	Outbound gateway + pre-send content review + full audit trail

The value of this table is that it turns "dangerous" from an adjective into a categorical variable you can put into code. That variable determines three things: which approval path the action takes, which compensation path it takes after a failure, and what granularity of evidence the log has to keep. The idempotency key, the human confirmation points, and the audit fields discussed later all unfold around this table.

One thing has to be emphasized: irreversibility is not a static property. A tool that "sends a notification email to a customer" looks like an external notice, but the business meaning embedded in it may be "your order has shipped" — a sentence that in an e-commerce context sits right on the boundary between offer and commitment, and a mis-send will trigger disputes and platform penalties. So the real approach is to tier by business semantics rather than by HTTP verb: both are POST, yet POST /draft and POST /shipment/confirm differ in risk by two orders of magnitude.

2. Why "Calling an API" Suddenly Gets Hard in the Agent Scenario

Traditional integration code has mature patterns for dealing with third-party APIs: you know the request can fail, you know it can time out, you know the other side retries, you know the transaction boundary is on your side. You write a service layer, handle timeouts and retries, and the task is done. In a developer's mental model, an API call is a synchronous function call with a clear success or failure.

The Agent smashes that assumption, in three specific ways:

• The call is initiated by the model, and the parameters are generated by the model. In traditional integration the parameters come from validated business code; on the Agent side they are the product of parsing a piece of natural language. Wrong fields, wrong magnitudes, mismatched units — the failure rate is inherently higher, and nothing static can catch it before the failure happens.
• The call is part of a loop. In traditional integration, "this step failed" raises an exception and terminates. In an Agent, "this step failed" may trigger a retry, different parameters, or an entirely different tool. One business intent expressed as several physical calls is the normal case, not the exception.
• A human is in the loop, but not on the critical path. In a traditional flow, approval is a node on the flowchart. In an Agent flow, approval may be scheduled somewhere in the middle, or the model may "believe it has already been approved". Where to place the human confirmation point therefore becomes a real problem, not something you solve by routing through an approval workflow.

Each of these three shifts maps onto a group of designs later in the article: the parameter problem needs business-level validation before execution, with validation failures returned to the model as information rather than as exceptions; multiple physical calls require idempotency or deduplication; and human intervention has to be designed as a signed credential rather than a single exchange of dialogue.

3. The Route This Article Takes

What follows is ordered as "draw the boundaries first, then the mechanisms, then accountability":

1. Why permission boundaries have to be cut along the three dimensions of read / write / cross-tenant rather than by feature, and what to do when OAuth scope granularity is not yours to decide.
2. What alternatives exist when an idempotency key does not work on someone else's API, and which alternatives introduce new problems of their own.
3. Why long transactions across SaaS systems leave only the saga as a path, how to grade compensation by strength, and how to handle the in-doubt state of "called but never confirmed".
4. The three forms two-phase commit can actually take in the Agent scenario, and the conditions under which each one fails.
5. Where human confirmation points belong, what a human should be shown at each one, and how to avoid confirmation fatigue and TOCTOU.
6. What audit trails have to record for later accountability, and why a pre-write snapshot and "the system state at the time" matter more than the operation's outcome.

The core judgment of the whole article is this: the hard part of connecting to real business systems is not "whether the call can be made", it is "when something goes wrong, can you explain it, undo it, and trace it to a person". These three capabilities decide whether an Agent integration moves from a toy into production.

4. Permission Boundaries: Cut Along Read, Write, and Cross-Tenant, Not Along Features

Why Authorizing by Feature Is Not Enough

The most common approach is to authorize by feature: give the Agent an orders tool that can both query and modify orders, give it a refunds tool that can initiate refunds. This seems perfectly reasonable on the surface, but it has three holes, and all three of them show up in production.

The first hole is blast radius. A tool that "modifies an order" may actually be able to modify the shipping address, the recipient, the phone number, the notes, the status, and the quantity. If permissions only distinguish "can modify orders" from "cannot modify orders", then a single model hallucination can ship the goods to the wrong address, and nothing in the system stops it. To stop it you have to split the tool finely enough that "change the address" and "change the status" are two different tools with two different sets of authorizations.

The second hole is mixing reads and writes. In the OAuth integrations of most SaaS products, reading an endpoint and writing an endpoint often share the same scope on the server side, sometimes even the same set of object-level permissions. That means once you grant write access, the Agent's bulk-read capability is opened up along with it — and in most security assessments a read action is no lower in severity than a write, because it can walk away with the entire customer database.

The third hole is cross-tenant. In a multi-customer deployment the permission is named "can operate orders", but the real boundary is "can operate orders under which tenant". These two dimensions have to be checked together; missing either one causes an incident. The check belongs after server-side credential resolution and before tool execution, rather than letting the model fill in tenant_id in the parameters and hoping it got it right.

The Three-Dimensional Authorization Model

Combining those three holes, a workable authorization model needs at least three orthogonal dimensions: action (read / write / delete / export), object scope (which resource types, which fields, which tenants), and risk level (corresponding to the irreversibility tier table in section 1). Only the intersection of all three is let through.

The key design judgment is this: these three dimensions must be evaluated on the server side or in a separate authorization layer, and must never be delegated to the model or expressed by the model through parameters. The model can propose the request "I want to change the status of order 12345 to shipped", but it cannot decide "I have the right to modify order 12345". The first is intent, the second is authorization, and the moment the two are mixed together the permission system has already failed.

The engineering approach is to introduce a layer of capability tickets: what the Agent's planning phase produces is not a direct API call but a restricted, signed ticket that can do only one class of thing and touch only one set of objects. The tool execution layer accepts tickets only, never bare parameters. The ticket itself can be inspected, expired, and revoked, and a human can glance at it and know exactly what this operation is allowed to do.

# 能力票据：把"意图"和"授权"拆成两件事
# 模型只能产出 grant 里的 action / resource / constraints，
# 至于"能不能"由这里的 policy 层说了算。

import time
import hmac
import hashlib
import json
from dataclasses import dataclass, field

# 不可逆性分级，与文章第一节的表格对应
RISK = {
    "draft.write": 1,        # 可原地改写
    "order.status.write": 2, # 逻辑可撤销
    "refund.create": 3,      # 财务不可逆
    "notice.send": 3,        # 对外不可逆
}

# 超过这个等级必须人工确认
CONFIRM_THRESHOLD = 3


@dataclass
class Grant:
    action: str                       # 动作，如 order.status.write
    resource_type: str                # 资源类型，如 order
    resource_ids: list                # 允许触碰的具体对象
    fields: list                      # 允许修改的字段白名单
    tenant_id: str                    # 租户，由凭证解析得出，不由模型提供
    max_calls: int = 1                # 票面额度，用完即焚
    constraints: dict = field(default_factory=dict)
    issued_at: float = 0.0
    expires_at: float = 0.0
    nonce: str = ""


class PolicyError(Exception):
    pass


class CapabilityBroker:
    """唯一的授权出口。工具层只认票据，不认模型给的参数。"""

    def __init__(self, secret: bytes):
        self.secret = secret
        self.used = set()

    def issue(self, *, action, resource_type, resource_ids, fields,
              tenant_id, ttl=120, max_calls=1, constraints=None):
        if action not in RISK:
            raise PolicyError("未登记的动作不允许发放票据: " + action)
        now = time.time()
        grant = Grant(
            action=action,
            resource_type=resource_type,
            resource_ids=list(resource_ids),
            fields=list(fields),
            tenant_id=tenant_id,
            max_calls=max_calls,
            constraints=constraints or {},
            issued_at=now,
            expires_at=now + ttl,
            nonce=hashlib.sha256(
                (str(now) + resource_type).encode()).hexdigest()[:16],
        )
        return grant, self._sign(grant)

    def redeem(self, grant, signature, *, args, caller_tenant):
        self._verify(grant, signature)
        if time.time() > grant.expires_at:
            raise PolicyError("票据已过期")
        if grant.nonce in self.used:
            raise PolicyError("票据已被使用，拒绝重放")
        if caller_tenant != grant.tenant_id:
            raise PolicyError("跨租户访问被拒绝")
        if grant.action not in args:
            raise PolicyError("请求动作与票面不符")
        extra = set(args[grant.action].keys()) - set(grant.fields)
        if extra:
            raise PolicyError("超出字段白名单: " + ",".join(sorted(extra)))
        if self.used_count(grant) >= grant.max_calls:
            raise PolicyError("票面额度耗尽")
        self.used.add(grant.nonce)
        return grant.action, args[grant.action]

    def needs_human(self, grant):
        return RISK.get(grant.action, 99) >= CONFIRM_THRESHOLD

    def _sign(self, grant):
        raw = json.dumps(grant.__dict__, sort_keys=True, default=str)
        return hmac.new(self.secret, raw.encode(), hashlib.sha256).hexdigest()

    def _verify(self, grant, signature):
        if not hmac.compare_digest(self._sign(grant), signature):
            raise PolicyError("票据签名校验失败")

    def used_count(self, grant):
        return 1 if grant.nonce in self.used else 0

OAuth Scope Granularity Is Not Yours to Decide

The model above is clean, but reality has a hard constraint: the granularity of the credential is set by the third-party SaaS. You want to split permissions by action and by field, and what you get back may be a single orders.write that draws no line between read and write, none between fields, none between tenants. There are three pragmatic approaches.

• Distinguish by credential type instead of attempting fine-grained authorization. The simplest effective method is to hold two sets of tokens, a read token and a write token, and let the tool layer pick the matching one per action. That way even if their scope is coarse, you have separated reads from writes inside your own process, and at minimum you guarantee that when the read path is compromised the write credential is not in hand.
• Do a shadow write on your own side. Land the write first in a local shadow table, and after the third-party call succeeds let a single component "publish" the state. This local table is yours: you can obviously validate by field, record before and after values, and withdraw unpublished writes. The cost is one extra hop of latency and one state synchronization.
• Write down the risk of coarse-grained authorization explicitly and put gates on it. Record clearly that "invoking this tool is equivalent to granting full write permission", then configure confirmation points, rate limits, and alerts specifically for that tool. This is the least elegant but most honest approach, and a lot of production systems stop right here.

What you should not do is pretend the scope is fine-grained enough and then write "we enforce least privilege" in the documentation. The real boundary of a permission model is written in the code, not in the documentation, and the boundary in the code is that coarse scope the other side handed you.

Read Permissions Are an Attack Surface Too

Putting all the attention on write operations is another common bias. Agent reads carry two risks of their own: first, bulk extraction — the model may call read endpoints back to back inside a loop and drag off an entire table, while every individual call looks legitimate on its own; second, sensitive data read into the context, and that context gets sent to the model provider, so once sensitive data has left your boundary it is beyond your control.

For the first risk, the gate belongs on the read side: a cap on how much a single task may read, a cap on rows per object, and "more than N consecutive reads of the same resource type requires an explanation". For the second risk, the reasonable approach is field-level redaction of what tools return, so the model sees "phone number ending 4321" instead of the full number, with the complete value available only through a separate, approval-gated fetch action. Redaction belongs in the tool layer, not in a prompt that says "please do not output phone numbers" — the latter hands access control to a probabilistic model. Any data you "do not want to appear in the model's context" should be cut off in the tool layer rather than hoping the model complies. The model is a passive receiver; it has no ability to enforce a data protection policy.

5. When the Idempotency Key Does Not Work on Someone Else's API

The Three Sources of Idempotency

Let us clarify one thing first: idempotency is not your client-side behaviour, it is a server-side contract. You only have idempotency if the other side supports "sending the same Idempotency-Key takes effect once"; if it does not recognize that header, sending it a hundred times accomplishes nothing. There are only three sources of idempotency you can rely on in engineering practice, and in practice most third-party SaaS products provide only half of the first two.

Source	Mechanism	Typical availability
Client-supplied	The request carries an idempotency key, the server holds a uniqueness constraint	Only a few payment-class APIs offer this, and the key's scope and expiry differ from one to the next
Natural business key	Guaranteed by business uniqueness, e.g. only one shipment record can exist for an order	The most common, but it is up to you to discover it
Your shadow layer	Enforce uniqueness in your own storage, write locally before sending anything out	Always available, at the cost of one extra write

Without the first source, the default reaction of most teams is to resend the same request three times and hope for the best. That is the worst option on the table, because in the best case it wastes three units of quota and in the worst case it produces three real records. So the real task is this: how do you make retries safe when the server offers no idempotency support.

Approach One: A Client-Side Idempotency Ticket, Building Uniqueness on Your Own Side

This is the most general option, and the one worth implementing first. Its core is not adding a parameter to the request but first establishing, on your own side, a unique record saying "this is what I intend to do", and only then calling out. It does not matter that the external call happens more than once, because whatever the outside returns is absorbed by your ledger.

There are three implementation points. First, the ledger must hit the disk before the call goes out; reversing that order opens a replay window in which the call succeeded but the ledger never recorded it. Second, the uniqueness constraint must live in the database layer, not as an application-level check-then-insert — under concurrency, two retries will both observe "not present". Third, the state must be an explicit three-valued state rather than a boolean: pending / succeeded / unknown. Storing only a boolean is the most common mistake of all, because you cannot tell "it never ran" apart from "it ran but the result was lost", and those two cases call for exactly opposite handling.

# 幂等台账：外部 API 不认幂等键时，把去重责任收到自己这边。
# 关键点是三态而不是布尔 —— unknown 是最容易被忽略的一态。

import sqlite3
import time
import json
import uuid

PENDING, SUCCEEDED, UNKNOWN, FAILED = "pending", "succeeded", "unknown", "failed"


class DedupeLedger:
    def __init__(self, path):
        self.db = sqlite3.connect(path, isolation_level=None)
        self.db.execute("PRAGMA journal_mode=WAL")
        self.db.execute("""
            CREATE TABLE IF NOT EXISTS op (
                key         TEXT PRIMARY KEY,
                op_type     TEXT NOT NULL,
                natural_key TEXT NOT NULL,
                tenant_id   TEXT NOT NULL,
                state       TEXT NOT NULL,
                request     TEXT NOT NULL,
                response    TEXT,
                external_id TEXT,
                created_at  REAL NOT NULL,
                updated_at  REAL NOT NULL
            )
        """)
        # 天然键兜底：即使外部调用重复发生，业务上也不该出现第二条
        self.db.execute("""
            CREATE UNIQUE INDEX IF NOT EXISTS op_natural
            ON op (op_type, tenant_id, natural_key)
        """)

    def reserve(self, key, op_type, natural_key, tenant_id, request):
        """先落盘再外发。冲突时返回已存在的记录，而不是报错。"""
        now = time.time()
        try:
            self.db.execute(
                "INSERT INTO op (key, op_type, natural_key, tenant_id, state,"
                " request, created_at, updated_at) VALUES (?,?,?,?,?,?,?,?)",
                (key, op_type, natural_key, tenant_id, PENDING,
                 json.dumps(request, sort_keys=True), now, now))
            return {"state": PENDING, "key": key, "external_id": None}
        except sqlite3.IntegrityError:
            row = self.db.execute(
                "SELECT state, response, external_id FROM op WHERE key=?",
                (key,)).fetchone()
            # 已有记录：重放是安全的，把上次的结果交回去
            return {"state": row[0], "key": key, "external_id": row[2],
                    "response": row[1]}

    def mark(self, key, state, response=None, external_id=None):
        now = time.time()
        if state == UNKNOWN:
            # 超时或连接中断：既不能判成功也不能判失败，留给对账
            self.db.execute(
                "UPDATE op SET state=?, response=?, updated_at=? WHERE key=?",
                (state, json.dumps(response or {}) if response else None, now, key))
            return
        self.db.execute(
            "UPDATE op SET state=?, response=?, external_id=?, updated_at=?"
            " WHERE key=?",
            (state, json.dumps(response or {}, ensure_ascii=False),
             external_id, now, key))

    def recheck_after_timeout(self, key, probe):
        """对账用的探测：用一个只读接口确认那次调用到底生效没有。"""
        row = self.db.execute(
            "SELECT state, external_id, request FROM op WHERE key=?",
            (key,)).fetchone()
        if row[0] == UNKNOWN:
            found = probe(json.loads(row[2]))
            if found:
                self.mark(key, SUCCEEDED, external_id=found)
                return SUCCEEDED
        return row[0]

    def pending_older_than(self, seconds):
        """悬挂扫描：超过窗口还没结论的，必须有人工介入或对账。"""
        cutoff = time.time() - seconds
        return self.db.execute(
            "SELECT key, op_type, natural_key, state FROM op"
            " WHERE state IN (?,?) AND updated_at < ?",
            (PENDING, UNKNOWN, cutoff)).fetchall()

Approach Two: Deduplicate on the Natural Key, Expressing "the Same Thing" as a Single Fact

Plenty of business processes never need an idempotency key at all, because the business itself is unique: one order can only have one shipment record, one settlement period can only have one reconciliation statement, one phone number can only receive one verification code per day. Find that natural key and deduplication can be done on the query side, which is more robust than doing it on the write side.

The way to discover the natural key is to ask yourself one question: if this write really did execute twice, what implausible data would end up in the database? The answer is usually the natural key. This framing works better than digging through the API docs looking for Idempotency-Key, because the latter frequently does not exist in the first place.

Approach Three: The Pre-Check, and Why It Bites You

The most intuitive approach is "query first, and if it already exists do not write". This works in many scenarios, but there are three traps worth knowing about in advance.

• TOCTOU. There is a time window between the query and the write, and two concurrent executions (the user retried once, or another path happens to be processing the same thing) can both pass the check and then both write. The pre-check lowers the probability of a duplicate; it does not eliminate it.
• Doubled cost. Every write operation is preceded by an extra read, so the number of API calls doubles outright. Against a third-party API with rate limits, that can make you hit throttling constantly during batch jobs, which is worse than the occasional duplicate.
• It bakes an "insert if absent" assumption into your business logic. If that "absent" is itself part of a business judgement (whether this customer already has a refund record decides whether a refund is permitted at all, not merely whether to duplicate it), then the pre-check is really introducing a state machine into the business layer, and the complexity is no lower than doing idempotency properly.

So the criterion is this: a pre-check is suitable for validating invariants of the form "if this were duplicated it would become a business error", and unsuitable as the sole implementation of idempotency. The first is a business requirement, the second is a reliability requirement. The two can coexist, but do not share the same piece of code.

When to Give Up on Idempotency and Cap the Rate Instead

There is a class of operations that is hard to make idempotent (each call genuinely has to happen once, such as incrementing a counter or sending a random verification code) and not worth building a ledger for. The right handling is not to grind away at idempotency but to acknowledge that it is inherently non-idempotent and then put it inside a hard "at most once" quota: within each task cycle the number of successes for this class of operation is strictly capped, and hitting the cap means stopping and waiting for a human rather than letting the model decide whether to try again.

Operations of this kind have to be marked separately in the capability ticket, for example at_most_once=true, with the ticket allowance set to 1; once consumed it is never resent and only enters the human queue. The benefit is honest semantics: your system explicitly records "this operation is not guaranteed idempotent, we rely on a quota and a human backstop", instead of leaving behind a comment that reads "in theory this will not duplicate". Writing "this should not duplicate" in the system and writing "runs at most once, a duplicate goes to a human" are two completely different engineering states. The first is a wish, the second is a verifiable contract.

6. Long Transactions and Partial Failure: Across SaaS, the Saga Is the Only Road Left

Why XA Is Not Even an Option Here

The classic form of two-phase commit requires every participant to implement two protocol methods, prepare and commit, before a coordinator is entitled to propose a global transaction. But when the participants are the REST APIs of third-party SaaS products, that premise does not hold: the other side will not implement prepare for your transaction, and you cannot force it to hold a lock indefinitely after a crash while it waits for your decision.

So in the scenario of an Agent connecting to real business systems, "should we use two-phase commit" is not a technology-selection question but a question of whether the premise holds at all. Once you cross a process boundary you do not control, a distributed transaction has only two possible shapes: either it degrades into compensation (eventual consistency), or it gives up global atomicity and guarantees only the atomicity of each local step. There is no third road in between.

Worth adding: local step atomicity is often missing too. A third-party API's "modify order status" may internally be three actions — update the main table, write an operation log, trigger a notification. Your single HTTP call cannot see inside that, and what you get back is their final state. At that point the only things you can depend on are the response they gave you and subsequent queries, not their internal transaction.

Compensation Is Not Rollback: Grade Its Strength First

The most common mistake is treating saga compensation as if it were a database rollback. A database rollback has atomicity and isolation guarantees; compensation does not. Compensation is a process of cancelling an earlier forward operation with a new forward operation. It can fail, it can fail partially, and downstream may even observe the intermediate state. So compensation has to be graded by strength before you decide whether to rely on it.

Compensation strength	Shape	Example	Required precondition
Strong	The undo API has no side effects downstream	Revoking a coupon that has not been used	Downstream guarantees the revocation triggers no notification and leaves no visible trace
Weak	The revocation leaves a trace, or is itself observed asynchronously	Cancelling an order emits an "order cancelled" notification	Accept the trace, or account for it as part of the expectation beforehand
None	No reverse operation exists at all	A bank transfer already settled, an outbound email already sent	Can only be intercepted beforehand, never handled afterwards
Fake	Looks like it was undone, but only changed state inside your own system	Locally marking an order cancelled while downstream still believes it shipped	Must have a reconciliation backstop, and must be stated plainly to the user

The last row is the most dangerous, because inside the system it looks exactly like a successful compensation. The test for this judgment is simple: after the compensating action, query downstream once — did the state actually change? If it did not, it is fake compensation. Fake compensation is not forbidden, but the user has to be told "our side shows cancelled, their system may still show shipped", rather than being handed one uniform success state in the UI.

Called but Never Confirmed: In-Doubt Is the Normal Case, Not the Exception

Long transactions in the Agent scenario raise a problem that is not especially prominent in traditional distributed transactions: the call may already have reached the other side while the response was lost in the network layer. In a traditional system this usually shows up as the user seeing a timeout and then retrying, and the volume is small; in the Agent scenario, multiple retries inside a single loop amplify this kind of "result unknown" call, and the model will often change the parameters and try again the moment it sees a timeout.

This produces a class of state that has to be expressed explicitly in the data model: in-doubt. When the result of a write operation is unknown, it can be recorded neither as success nor as failure; it has to stay in the in-doubt state and enter a periodic reconciliation process. Ignoring the in-doubt state leads directly to one of two outcomes: either you treat unknown as success and the user sees "completed" while the other side has no data, or you treat unknown as failure and retry, and the other side produces duplicate records.

The handling order for an in-doubt case should be: first attempt a read-only probe (does the other side offer a query endpoint); if the probe cannot resolve it, escalate to a human; never blindly resend. What the human step needs is the full set of call parameters, the time it was issued, the error characteristics returned at the time, and the fragment of the response body that was received. If those were not recorded, a human cannot adjudicate either.

Business-Level Compensation vs Technical-Level Compensation

Another distinction that has to be made clearly is the layer at which compensation happens. Technical-level compensation means "undo the write I just performed"; the action is fixed and mechanically executable. Business-level compensation means "return the business to a legal state"; the action depends on business context and may have nothing to do with the original operation.

For example, an Agent ships an extra box to a customer. Technical-level compensation is "cancel that order line"; business-level compensation might be "cancel the order line and trigger a customer-service follow-up based on the customer's tier". The former is mechanically reliable but can leave an awkward business state behind; the latter fits the real need more closely but requires the model or a rule set to generate it — and that is precisely where the danger lies: letting the model generate the compensating action amounts to handing an irreversible operation to a probabilistic component.

The pragmatic design is this: technical-level compensation is executed by deterministic code, the model is responsible only for "proposing a compensation intent" (which item to cancel, and for what reason), and the compensation that actually lands is determined by a rule-based mapping table. Only when the model cannot get into that layer is responsibility clear.

Implementation: The Saga Ledger and the In-Doubt Sweep

# Saga 编排器：只负责"记录做了什么"和"决定下一步"，
# 不负责回滚 —— 跨系统没有回滚，只有补偿。

import time
import json
from dataclasses import dataclass, field, asdict


class StepState:
    PENDING = "pending"
    DONE = "done"
    FAILED = "failed"
    COMPENSATED = "compensated"
    COMPENSATE_FAILED = "compensate_failed"
    IN_DOUBT = "in_doubt"


# 每个步骤必须预先声明补偿能力，声明不了就不该进 Saga
STEP_CONTRACT = {
    "create_ticket":   {"compensate": "cancel_ticket",   "strength": "strong"},
    "reserve_stock":   {"compensate": "release_stock",   "strength": "strong"},
    "charge_card":     {"compensate": None,              "strength": "none"},
    "send_notice":     {"compensate": None,              "strength": "none"},
    "assign_courier":  {"compensate": "unassign_courier","strength": "weak"},
}


@dataclass
class Step:
    name: str
    state: str = StepState.PENDING
    request: dict = field(default_factory=dict)
    response: dict = field(default_factory=dict)
    attempts: int = 0
    last_error: str = ""
    updated_at: float = 0.0


class Saga:
    def __init__(self, saga_id, steps):
        self.id = saga_id
        self.steps = [Step(name=n) for n in steps]
        self.created_at = time.time()

    def snapshot(self):
        return {
            "saga_id": self.id,
            "created_at": self.created_at,
            "steps": [asdict(s) for s in self.steps],
        }

    def plan_compensation(self):
        """逆序补偿 —— 已完成的步骤按倒序一个个撤，撤一个确认一个。"""
        plan = []
        for s in reversed(self.steps):
            if s.state != StepState.DONE:
                continue
            contract = STEP_CONTRACT[s.name]
            if contract["compensate"] is None:
                # 无补偿：不能悄悄跳过，必须显式记录为"需要人工"
                plan.append({"step": s.name, "action": None,
                             "action_required": "manual",
                             "reason": "该动作无反向接口"})
                continue
            plan.append({"step": s.name,
                         "action": contract["compensate"],
                         "action_required": "auto",
                         "strength": contract["strength"]})
        return plan

    def record(self, name, state, response=None, error=""):
        s = self._get(name)
        s.state = state
        s.attempts += 1
        s.updated_at = time.time()
        if response is not None:
            s.response = response
        s.last_error = error
        return s

    def in_doubt(self):
        return [s.name for s in self.steps
                if s.state in (StepState.IN_DOUBT, StepState.PENDING)
                and time.time() - s.created_at > 300]

    def _get(self, name):
        for s in self.steps:
            if s.name == name:
                return s
        raise KeyError(name)


def classify_outcome(response, transport_error):
    """把一次调用的结果归成三态。这一步是整个 Saga 的地基。"""
    if transport_error is None:
        return StepState.DONE, response
    if getattr(transport_error, "definitely_not_sent", False):
        # 连接都没建起来，对方一定没收到，可以安全重试
        return StepState.PENDING, None
    if getattr(transport_error, "server_replied", False):
        # 拿到了错误响应：这是确定性的失败
        return StepState.FAILED, getattr(transport_error, "body", None)
    # 发出去了但没等到回音 —— 最难处理的一类
    return StepState.IN_DOUBT, None

There is one design judgment in this implementation that is easy to skip past: plan_compensation does not return a skip for steps that have no reverse interface — it returns an explicit "requires a human" record. Plenty of implementations just continue here, and the result is that a failed transaction shows up in the system as "partially complete", while the steps that cannot be compensated were neither rolled back nor noticed by anyone. Listing them explicitly is what gets someone to go and handle them.

7. The Three Forms Two-Phase Commit Can Actually Take in the Agent Scenario

Earlier I said there is no classic 2PC across systems. But the intent behind "prepare first, commit second" is extremely valuable — it eliminates an entire category of "half-committed" problems. So in the scenario of an Agent connecting to real systems, what is worth doing is to rebuild the semantics of 2PC in degraded form into three implementable shapes, and to be clear about the failure conditions of each.

Shape One: Two-Phase Confirmation (Prepared State + Committed State)

The premise is that the other side's API itself offers two actions, "create draft" and "confirm into effect". Financial and ticketing SaaS products frequently have this design: first POST something that returns a draft number, then PUT to confirm. If that is available, it is the cleanest option, because during the prepare phase the other side has already reserved the resource.

The key engineering point of this shape is crash recovery for the coordinator. The coordinator must first persist the record "I am about to commit X, the other side's draft number is D", and only then issue the confirm. If the order is reversed — confirm first, persist second — and the process dies in between, you will never know which draft was supposed to be confirmed.

Shape Two: Shadow Table + Atomic Publish

When the other side has no draft mechanism, you can build one on your own side. All writes land in a local shadow table first, in the state staged, and users can view, edit, and withdraw them. After confirmation, a publisher translates the shadow row into the other side's API call and, on success, marks the shadow row published and stores the ID the other side returned.

This shape pulls the decision of "whether to commit" entirely into your own hands, so the confirmation point can be any UI action at all, and it can even support a "dry-run publish" (compute the parameters without actually calling). Its cost is the dual-write problem and consistency lag: the shadow table says success while the other side failed. The way to handle that is to give shadow rows both a published state and a sync_error state, plus a scheduled job that reconciles the two.

Shape Three: Pre-Write Snapshot + One-Way Commit

The weakest but most general shape. Before any write operation, capture and store the complete current state of the target object; the write operation itself is a single, indivisible call. If a problem shows up afterwards, restore from the snapshot. This shape does not chase atomicity, it chases recoverability.

Its applicability is much broader than the first two, because both of those require cooperation from the other side. The drawback is that the recovery action is itself a write operation and can in principle fail again; so the snapshot has to contain enough information to make recovery deterministic (store the amount, the status, and the version number — not just a diff).

Failure Conditions: When the Two Phases Fall Apart

Looking at the failure conditions of the three shapes together, they all converge on the same set of root causes.

Failure condition	Symptom	Root cause	Mitigation
Coordinator crashes after prepare	The other side's resource is held permanently and the business never sees the draft	Dangling transaction	prepare must carry a TTL and the other side auto-releases on expiry; the coordinator runs a recovery sweep
The commit request is lost	The other side committed while you recorded a failure, so you commit a second time	An uncertain result treated as a failure	The commit phase must query before retrying, and the ledger must distinguish the unknown state
Human confirmation spans both phases	The user confirmed against data from five minutes ago, and by execution time the data has changed	TOCTOU	The confirmation ticket binds the object version, revalidates before execution, and is voided on mismatch
Cross-tenant data enters the same saga	Tenant A's confirmation lets tenant B's commit through	The ticket is not bound to a tenant	Capability tickets are forced to carry tenant_id, and the signature covers it
Draft semantics are not guaranteed by the other side	Creating the draft already triggered a downstream notification	"Prepared" is not actually prepared	Test it once during integration; do not trust the word "draft" in the docs
The shadow table grows without bound	The meaning of historical staged rows conflicts with current rows	The shadow table has no lifecycle management	Clean staged rows by TTL, and only after they are published or withdrawn
The publisher is a single point	The publisher dies and all staged rows pile up	No queueing	Publishing goes through a durable, replayable queue, with the idempotency key bound to the shadow row ID
The version number is supplied by the model	Optimistic locking becomes meaningless	The concurrency control parameter is untrustworthy	The version number is returned by the read tool and passed back verbatim; the tool layer validates its source

The two rows most worth dwelling on in this table are the second and the third. The second is the root of "duplicate commits", and the absurdity of it is that you had already got half of it right (the ledger has an unknown state), yet the whole design was undone because the code says if error: retry. The third is specific to the Agent scenario, because in a traditional flow confirmation and execution are milliseconds apart, whereas in the Agent scenario they may be minutes apart or separated by a queue waiting on a human.

When Not to Use Two Phases

Two phases cost something: one more round trip, one more piece of state, one more set of recovery logic, one more invariant to maintain. If the operation itself is low-risk, compensable, and has a small blast radius when it fails, adding two phases is a net loss.

One actionable criterion is this: if this write fails, can you accept it ending up in a "partially effective" state, with the user able to understand that state in the UI? If you can accept it, do not use two phases — do it directly, simply and reliably. If you cannot, especially when money or an external commitment is involved, then consider two phases or an equivalent strong-confirmation mechanism. Two phases solve the problem of "half-committed", not the problem of "committed wrongly". Committed wrongly is solved by validation, confirmation, and compensation. Conflating the two usually means you spent your effort on the wrong layer.

8. Who Should Own the Timeout of a Long Transaction

A multi-step operation across SaaS products is naturally a long transaction, and a long transaction inevitably has to face the question of "when do we give up". There is one concrete engineering judgment here: the timeout budget should be allocated by the Saga coordinator according to the number of remaining steps, rather than each step setting its own default.

The reason is straightforward: for a five-step saga where every step has a 30-second timeout, the total budget is 150 seconds and the waiting time in between is never counted; conversely, if each step is computed at 30 seconds, the user waits 29 seconds only to fail on the first step, and the remaining four never get a chance to run. The right approach is to give the whole saga a total budget, then maintain a remaining budget inside the coordinator: before issuing each step's call, check whether what is left is still enough to cover that step's p99 latency, and if not, go straight into compensation instead of sending it out.

There is one more easily overlooked point: time spent waiting for human confirmation should not count against the technical timeout, but it must count against business timeliness. If an operation has to wait three days for a person to approve it, then any 30-minute technical timeout is meaningless, while "this reservation is still holding inventory three days later" is a real incident. The right approach is to set the TTL of the reserved resource explicitly longer than the expected duration of the human confirmation, and to instrument and alert on that duration.

9. Where Human Confirmation Points Belong

The design of confirmation points is the hardest trade-off in this kind of system. Insert too few and the risk exposure is large; insert too many and people form a muscle memory of "click confirm", which is more dangerous than having no confirmation at all. The failure mode of most systems is the latter: nobody reads the content of the confirmation dialog, and the approval workflow is more formal than substantive.

So the right question is not "whether to confirm" but at which positions to confirm, what a person should be shown at each one, and whether the act of confirming can itself be trusted.

Three Criteria That Decide Whether Confirmation Is Necessary

Put the irreversibility tiers from section 1 to use, and whether a write operation needs confirmation can be decided by three criteria together. The first is degree of irreversibility, meaning "after undoing it, can the outside world still pretend it never happened". The second is blast radius: how many objects an operation touches, how much money, how many people. The third is detectability: if this operation goes wrong, can it be found and corrected quickly — an operation that can be detected has lower confirmation value, because correcting it afterwards is not expensive.

The combination of these three criteria yields a practical conclusion: operations that are irreversible, wide-reaching and hard to notice must be confirmed; irreversible but narrow operations go through confirmation plus batching; reversible operations should not be confirmed. Adding confirmation to a reversible operation is pure cost — the user has to click every time, but a mis-click can still be undone.

Six Candidate Positions, and What Each Should Display

Take a typical cross-system task apart and there are six positions where a confirmation can be inserted. They are not mutually exclusive, and which ones you choose depends on the business, but the information each position has to display is different.

• Confirmation of the execution plan after planning. Confirm at the moment when the Agent has produced its plan but has not touched any data yet. What it should display is a list of "what is about to happen": which objects are involved, which fields, the possible amounts, which steps are irreversible. This layer catches "misunderstanding" — what the Agent wants to do is not what the user wanted. This is the most valuable confirmation point of all, because it happens before every side effect.
• Confirmation before an irreversible action. At the moment just before calling the third party to perform an irreversible operation. What it displays must include the complete verbatim parameters that are about to be sent, not a summary. A summary drops the details, and the details are precisely what lets a user judge right from wrong. For a notification email, a summary saying "a notification email will be sent" gives the user nothing to judge by; the verbatim text is "【Order】Your parcel has shipped, tracking number SF1234, dispatched today", and the user knows at a glance whether it should go out.
• Tiered confirmation for batch operations. When one class of operation affects N objects, do not pop N dialogs. Tier by blast radius: 1 to 3 objects are confirmed one by one; 4 to 50 objects show a categorical summary (for example, "change notifications will be sent to 32 customers, involving 3 order types") plus a sampled detail view; more than 50 go through a separate batch approval flow where an authorized role picks them off one by one in a list.
• Human adjudication of cross-system state inconsistency. When reconciliation finds "you said success but it actually failed" or "you said failure but it actually succeeded", this position is not a confirmation but an adjudication. It has to display a comparison of the two sides' states together with your recommended action, and no side should be preselected by default.
• Human decision after a compensation failure. When saga compensation fails halfway through, a person has to decide "keep undoing, or accept the current state". It shows what has been undone and what could not be, and the part that could not be undone must state its consequences plainly.
• Temporary authorization for privilege escalation. When an operation needs permission beyond the current ticket's scope, this is a "temporary elevation" rather than a "confirm execution"; what it shows is "what extra permission this needs, which objects it covers, and for how long it is valid".

Confirmation Fatigue and Degradation Strategies

The biggest failure of a confirmation mechanism is degrading it into a rubber stamp between human and machine. There are three signals that this has happened: the dwell time on the confirmation dialog keeps getting shorter; the approval rate approaches 100%; users start complaining "why do I always have to click this". The moment any one signal appears you should intervene, rather than continuing to add explanatory text.

The effective approach is to decouple confirmation from execution: let the user confirm "the policy for this class of operations" once, instead of confirming each individual operation. The policy might be "allow the Agent to adjust shipping fees on orders under 200 yuan", or "allow adding internal notes to tickets but not changing their status". At execution time the system automatically lets through whatever the policy covers, and only what falls outside the policy is confirmed one by one. That way the object a human reviews drops from "tens of thousands of operations" to "a handful of rules" — and rules are objects that can be reviewed, changed and traced.

Another approach is adjusting the information density of the confirmation according to past behaviour. If an Agent has submitted this class of operation for three consecutive months with zero rework, confirmation can be degraded to "a batch page plus a single approve button", but a traceable record must be retained. Conversely, if there was recently an operation that had to be undone, confirmations for that class should be upgraded back to item-by-item display. Both degradation and upgrade should follow explicit rules rather than being decided ad hoc by one person.

TOCTOU: The Time Window Between Confirmation and Execution

This is a problem specific to the Agent scenario, and it is very easy to miss. The data the user saw when the confirmation dialog appeared may already differ from the data at the moment the system actually executes. The gap may be seconds (the time the user spends thinking) or hours (the user approved it, but the request sat in a queue).

The solution is to make confirmation a credential that carries a version number. At confirmation time, record the version number or content hash of the key objects; before execution, re-read and compare, and on mismatch void the confirmation and run through the process again. This sounds cumbersome, but it avoids an entire category of "the user clearly approved A and the system changed it to B" incidents, which are extremely hard to explain afterwards.

At the same time the confirmation credential must carry a time limit. An operation approved ten minutes ago should not be executed two hours later, because the business conditions may have changed, while what the user approved was the context at that moment. How to set the time limit can follow the timeliness of the business itself: if the amount changes within a one-hour window, the credential should be shorter than that.

Confirmation Is Not Authorization

The last easy mistake is to treat "the user clicked confirm" directly as "the user authorized this operation", and then write "executed with user authorization" in the audit log. These two things differ in nature: confirmation is informed consent to one specific execution, while authorization is permission for a class of operations. Mixing them causes trouble during accountability — the user can say "I only saw that dialog, I never said it could change things like that from then on".

So the system should contain two separate things: a confirmation credential (for this execution, carrying the verbatim parameters, a data snapshot, a version number and a time limit) recorded in the execution record; and an authorization policy (for a class of operations, carrying scope, quota, validity period and grantor) recorded in the policy store. The execution record references the version of the policy it relied on, so that afterwards you can answer two different questions: "why was this permitted" (look up the policy) and "was this person aware" (look up the confirmation credential). Many teams only have one of the two, and end up either unable to explain where a permission came from or unable to explain whether the person involved was aware. In audit terms these are two independent questions, and they need two independent pieces of evidence.

10. Audit Trails: What You Have to Record to Be Accountable Later

The first six sections were all about how to get things right. This one is about how to explain yourself when you got them wrong. The criteria for these two things are completely different: a system can be entirely correct with no audit log at all, and it can log every single line and still be impossible to hold accountable afterwards — because what it logged is the wrong thing.

Agent scenarios make this sharper. In a traditional system, the context of a single write is well defined: the button the user clicked, the values in the form, the branch in the code. The context of a single write by an Agent includes the model version, the prompt, which documents happened to be in the context at that moment, the implementation version of the tool, and the intermediate decisions the model reasoned its way to. Reproducing "why did it do that" afterwards breaks the chain if any single link is missing.

The Four Questions Accountability Has to Answer

When you design audit fields, the starting point is not "which fields should we record" but "which questions do we need to answer afterwards". A record that can support accountability has to answer at least these:

1. Who triggered it? A human acting directly, a scheduled job, a callback from an upstream system, or the Agent continuing on its own (a retry after the previous round timed out). This field decides the first layer of responsibility attribution.
2. On what basis did it act? What the authorization policy was at the time, what the confirmation credential was, and what the previous step produced. Without this you cannot distinguish "the system works this way by design" from "someone changed the configuration on the fly".
3. What was the state of the system at the time? The full content of the modified object before the change, the relevant business parameters, and the position in the inventory/credit/state machine at that moment. Without this you cannot reconstruct the blast radius afterwards, and you cannot judge whether the operation made sense at the time.
4. What was the outcome? What the remote side returned, how long it took, how many retries happened, and what the final state was: succeeded/failed/unknown.

Of these four, the third is the one most often left out, and it is exactly the one that neither post-incident accountability nor post-incident repair can do without. Plenty of systems record only the operation and the result, never the pre-write state, and when something goes wrong they are left with recollection and guesswork. A pre-write snapshot is the highest-value-per-cost field in this kind of system; all it costs is serializing the object once.

The Event Model and Its Required Fields

Audit trails should be an event stream, not log lines. Log lines get sampled, rotated, and filtered by severity in the logging system; audit events must do none of those things. Each event category has its own structure, but a few groups of fields have to ride along on every event.

Field group	Fields	Purpose
Identity	event_id, run_id, step_id, actor (person or system identifier), on_behalf_of (acting on whose behalf)	Attach the event to a specific execution chain and tell autonomous execution apart from delegated execution
Causality	parent_event_id, caused_by (user instruction / upstream event / retry)	Rebuild the decision chain and locate where a given parameter came from
Model	model_id, prompt_hash, tool_version, context_manifest	Reproduce the decision basis at the time; a prompt_hash is far cheaper than storing the full prompt text
Intent	action, target, requested_fields, rationale	State what was intended, as distinct from what actually happened
Facts	before_snapshot, after_snapshot, external_request, external_response	The basis for blast-radius assessment and repair
Authorization	policy_id, policy_version, approval_ticket_id, granted_scope	Answer "why was this permitted" and "was a human aware of it"
Outcome	state (succeeded/failed/unknown), attempts, duration, error_class	Whether the outcome is known; the unknown state has to exist explicitly

Pay attention to the rationale field. What it records is the reason the model claims, not the truth, so it should not be treated as evidence — only as a supporting clue. What can actually serve as evidence is external_request: the verbatim parameters that were sent. The distinction matters: afterwards, seeing the model say "I confirmed the amount was 100 yuan" when what actually went out was 1005 yuan gives you two completely different starting points for the investigation.

Tamper resistance: making the record itself credible

If audit records can be modified or deleted, they are worthless in an accountability scenario. There are three levels of approach, in increasing order of effort:

• The bottom level: append-only, no modification. Store it in an append-only table or object store, allow only INSERT, and use database permissions so that the application account has no UPDATE and no DELETE rights. This step blocks application-layer mistakes and most conventional attacks.
• The middle level: a hash chain. Every record contains the hash of the previous record, forming a chain. If any single record is modified or deleted, the chain breaks, and detection works by recomputing the hashes. This needs no blockchain — one extra column in the database is enough — but the hash has to be computed synchronously at write time.
• The top level: periodically anchor the chain's root hash externally. If you do not even trust the DBA on your own database, push the root hash periodically to a channel you do not control (storage under a different account, or an append-only external service). The value of this step is not technical but psychological: it turns "modify the data" from "a technical action" into "an action that leaves an external trace", and the effect of that change on internal behavior is often larger than the technical guarantee itself.

Implementation: an append-only event chain

# 只追加事件链：每条事件带前一条的哈希，任何篡改都会在重算时暴露。
# 注意这里没有 UPDATE 路径 —— 表结构上就不提供修改入口。

import time
import json
import hmac
import hashlib
from dataclasses import dataclass, asdict, field


# 哈希链的锚定目标与普通记录是分开的存储，签名密钥也分开
ANCHOR_INTERVAL = 300


@dataclass
class AuditEvent:
    event_id: str
    run_id: str
    step_id: str
    actor: str                 # 人 / 定时任务 / 上游系统
    on_behalf_of: str          # 代表哪个租户或哪个用户
    action: str                # 想做的事
    state: str                 # succeeded / failed / unknown
    intent: dict = field(default_factory=dict)    # 意图与理由（辅助线索）
    before_snapshot: dict = field(default_factory=dict)
    after_snapshot: dict = field(default_factory=dict)
    external_request: dict = field(default_factory=dict)
    external_response: dict = field(default_factory=dict)
    authorization: dict = field(default_factory=dict)  # policy + approval ticket
    model_context: dict = field(default_factory=dict)  # model/prompt hash/tool version
    attempts: int = 1
    duration_ms: int = 0
    prev_hash: str = ""
    hash: str = ""
    ts: float = 0.0


class AppendOnlyAudit:
    def __init__(self, store, chain_key: bytes, sign_key: bytes):
        self.store = store
        self.chain_key = chain_key
        self.sign_key = sign_key
        self.last_hash = store.tail_hash() or ("0" * 64)

    def append(self, event: AuditEvent):
        # 关键顺序：先算哈希再落盘，落盘后推进 last_hash。
        # 如果进程在两步之间挂掉，这条事件丢失但链未断；
        # 如果反过来先推进后落盘，链会指向一条不存在的记录。
        event.prev_hash = self.last_hash
        event.ts = time.time()
        payload = json.dumps(asdict(event), sort_keys=True,
                             ensure_ascii=False, default=str)
        event.hash = hmac.new(self.chain_key, payload.encode(),
                              hashlib.sha256).hexdigest()
        self.store.insert(event)          # 只 INSERT
        self.last_hash = event.hash
        return event

    def verify_chain(self, limit=100000):
        """重算校验。返回断裂位置；返回 None 表示链完整。"""
        prev = "0" * 64
        for e in self.store.iter_from_head(limit):
            if e.prev_hash != prev:
                return {"broken_at": e.event_id, "reason": "prev_hash 不匹配"}
            body = dict(asdict(e))
            claimed = body.pop("hash")
            body["hash"] = ""
            payload = json.dumps(body, sort_keys=True,
                                 ensure_ascii=False, default=str)
            # 落盘时 hash 字段为空参与计算，复算时保持一致
            recomputed = hmac.new(self.chain_key, payload.encode(),
                                  hashlib.sha256).hexdigest()
            if recomputed != claimed:
                return {"broken_at": e.event_id, "reason": "内容与哈希不符"}
            prev = e.hash
        return None

    def anchor(self, anchor_store):
        """把链的当前根哈希 + 签名推到外部存储，外部拿不到内容只能验证。"""
        body = json.dumps({
            "head": self.last_hash,
            "count": self.store.count(),
            "ts": time.time(),
        }, sort_keys=True).encode()
        sig = hmac.new(self.sign_key, body, hashlib.sha256).hexdigest()
        anchor_store.put({"head": self.last_hash, "count": self.store.count(),
                          "sig": sig, "ts": time.time()})
        return sig

    def replay_for_incident(self, run_id):
        """事故复盘的主查询：按 run 拉全链路，缺失的因果关系要显式标出。"""
        events = [asdict(e) for e in self.store.by_run(run_id)]
        by_id = {e["event_id"]: e for e in events}
        chain = []
        for e in sorted(events, key=lambda x: x["ts"]):
            e["causal_parent_present"] = (
                not e.get("parent_event_id")
                or e["parent_event_id"] in by_id)
            chain.append(e)
        # unknown 状态必须单独挑出来，这些是需要人工跟进的对象
        unknown = [e for e in chain if e["state"] == "unknown"]
        return {"run_id": run_id, "events": chain, "needs_followup": unknown}

Retention, redaction, and access control

Retention policy for audit data should be tiered by content, not set uniformly. Events for irreversible operations (payments, outbound sends, permission changes) should be retained long enough to cover the relevant dispute window and regulatory requirements — usually on the order of years — and when they expire, expiry should be "archival" rather than "erasure": the archived portion stays queryable as a summary, while the original payload is handled according to compliance requirements.

There is an easy-to-miss tension here: auditing requires recording "the verbatim parameters that were sent out", and those parameters may contain personal information; the retention period is also far longer than that of business data. The resolution is structured redaction at record time for sensitive fields — record the field name, a hash, and a prefix and suffix, instead of the full plaintext. The full plaintext goes into a narrow store with its own permissions and its own encryption, access requires a second approval, and every access is itself audited (otherwise the question "who looked at this customer's payment details" can never be answered).

One last thing: the audit data itself has to be searchable enough to produce a useful conclusion. When you design the fields, work out "what will I query next time there is an incident", which normally means indexing on at least five dimensions: tenant, time range, target object ID, operation type, and final state. Especially "filter for unknown by final state" — that is the main query path for inspecting suspended operations, and without the index your inspection job degrades into a full table scan, and then nobody runs it.

11. One End-to-End Run: A Cross-System Consolidated Shipment Task

The first ten sections looked at mechanisms one at a time. This one strings them together and runs them once, because a lot of the design trade-offs only reveal their cost inside a complete flow. The scenario is a typical support ticket that nearly every integration team eventually hits: a customer says in the conversation, "Can the four orders I placed last week be merged into one shipment? Please don't send me four separate shipping notifications." The request looks like a single sentence, but in the system it touches four systems and six categories of state, two of which are irreversible outbound actions.

From One Sentence to a Set of Constrained Actions

After reading that sentence, the Agent first has to translate it into business semantics rather than directly into API calls. Some knowledge here must come from the system and the model must not invent it: which orders can be merged, what the merge constraints are, how shipping cost is computed after a merge, whether invoices can be combined across orders. These rules live in the merchant's commercial agreements, are written across several operational documents that contradict each other, and were revised as recently as three months ago.

The right split is to let the model handle "extract the intent and parameters from natural language" and let the rules engine handle "decide whether these parameters constitute a legal operation". The first is what the model is good at; the second is what the model is unreliable at. If you hand legality to the model as well, the likely conclusion is: these four orders can be merged, because from the text they do look like the same batch — while in reality one of them used a coupon, one has already shipped partially, and one is a pre-order item. The model cannot know any of that, because that information is neither in the user's sentence nor in its context.

So the first step of the flow is not planning, it is data collection. The system has to read out, in one shot, the current state of all four orders along with addresses, payment methods, available coupons, reserved inventory, and invoice status, and use that as the input snapshot for this task. Every judgment after this point is based on that snapshot rather than on query results the model might refresh at any moment. This is what guarantees that everything shown at every later confirmation point is self-consistent: what the user sees, what is validated at execution time, and what is recorded in the audit log all come from the same data.

The First Confirmation Point: the Plan, Not the Outcome

Once the rules engine has run, what it produces is a structured list of actions, not an executable script. This is where the model earns its place: translating that list into language a human can read and pointing out which steps carry extra risk the user needs to hear about — for example, "one of these orders used a coupon; after merging, the coupon will not be retained and the discount will not be refunded". The model's role in this step is expression, not decision-making.

The confirmation page the user sees should contain these pieces: the list of orders to be merged with each one's key details, every category of state change that will occur, which changes are irreversible, what happens if it fails midway, and whether there are choices the user needs to make. What should not appear on the page is any description of the model's "confidence level" for the operation — that is a number a user cannot act on.

The value of this confirmation point is that it intercepts misunderstanding, and misunderstanding is the hardest class of incident to repair afterwards. If the orders have already been merged and the notifications have already gone out, telling the user "you actually just wanted to ask whether merging was possible" means nothing. Correcting the misunderstanding while there are still zero side effects costs almost nothing. So whenever a task involves an irreversible action, the first confirmation point belongs before any side effect, even if it only covers part of the scenarios.

Halfway Through: the Invoicing Call Times Out

After the plan is confirmed, execution begins. The first two steps go smoothly: the merge request is submitted successfully, a tracking number is generated, and the waybill status on the logistics side moves to "awaiting pickup". The third step calls the e-invoicing service, and midway through processing the remote side resets the connection, so the client gets no response at all.

This is the hardest state this article keeps coming back to: not knowing whether the remote side actually issued the invoice. Note the retry trap here — if you follow the conventional pattern of catching the timeout exception and retrying, you get one of two bad outcomes. Either the remote side did issue the invoice and the retry creates a second invoice for the same order; or it did not, but the first request is still sitting in their queue and the retry creates the second one. The difference is that the first case is invisible until reconciliation, while the second exposes itself a few seconds later as a "duplicate invoice" — which is the lucky variant.

The correct handling has three actions, none of them optional. First, mark this step as suspended rather than as failed, which means the status field in the idempotency ledger has to be able to express that third state. Second, probe with a read-only interface: query the invoicing service by business order number to see whether an invoice was already generated. This is idempotent, because a query produces no side effects. Third, hand the decision to a human — if the probe cannot give a definitive answer either, generate a manual work order with the evidence from both sides attached.

There is a pragmatic judgment here: query interfaces are much rarer than people assume. Plenty of third-party services offer only the write endpoint and no lookup by business key, or their lookup lags by minutes. So routing to a human after a failed probe is not a design flaw — it is the normal path. A significant share of automation systems in production actually take the shape of "handle the certain parts automatically, queue the rest for people", not "fully autonomous". Setting that expectation with the business side at design time is far better than explaining it after launch.

The Compensation Gets Stuck Halfway Through

Suppose one of the four orders used a non-refundable coupon. The rules engine concludes that this order cannot be merged and must ship separately. The task discovers that constraint halfway through execution, which makes it a partial success. The system has to decide whether to roll back the first two steps or push forward.

Judge by compensation strength: the tracking number has been generated but not yet picked up, so cancelling it is weak compensation — once a tracking number exists it is already in the logistics system's queue, and even though nobody has collected it, a record of it already exists in their business system. Un-merging the orders is strong compensation, because it is essentially a data structure transformation that leaves no visible trace when undone. The customer notification has not been sent, so it is not in the compensation plan.

The compensation plan is therefore: undo the merge first, then cancel the waybill. Undoing the merge is deterministic and idempotent, so it can just be called. Cancelling the waybill calls the logistics cancellation API, and that API has its own business rules: a generated but uncollected waybill can usually be cancelled, but if the carrier has already assigned a branch and pushed tracking information, the cancellation fails. When the call fails, the system must not simply treat the whole compensation as failed and retry the entire flow. It has to mark that step individually, at compensation granularity, as "compensation failed" and move it to the human queue.

The interface finally handed to the user has to reflect this state truthfully: the merge has been undone, the waybill cancellation failed because the carrier has already assigned a branch, and the suggested resolution is to contact human support to cancel the original waybill, or to wait for the customer to sign and then request a refund. Rendering this as "operation failed, please retry" is information loss: the user retries, and ends up with a pile of duplicate waybills.

What You Can Produce Afterwards

After a task like this finishes, the audit system should hold this string of records: the user's original request verbatim together with the parsed intent parameters, the complete snapshot of all four orders at task start, the rules engine's verdict and the rule version it hit, the plan confirmation credential (including a hash of the page content the user saw and a timestamp), the verbatim external request and response for every step, the three-state marker and probe result for the suspended step, every step of the compensation plan and its outcome, and the explanatory text finally delivered to the user.

The concrete questions this string of records can answer include: why was this order judged unmergeable; did the user know the coupon would be clawed back; was this order processed twice; and if the business side later claims the page never showed that warning, you can compare the page snapshot against the template version that was rendered at the time.

Things you could not do should be marked honestly too. If you did not record the rule version, you cannot explain why that verdict was reached; if you did not record the pre-write snapshot, you cannot assess how much those four orders were subsequently affected. The value of an audit trail depends on its weakest link, not its most complete one.

12. The Unsexy Engineering Details in the Integration Layer That Decide Success or Failure

The first twelve sections were all about control flow: when to stop, when to roll back, when to ask a human. But in real projects, what trips up Agent teams is usually that the control flow is entirely correct while the data is a complete mess. This class of problem has a shared signature: it never raises an error, it only produces wrong results — and the wrong results look entirely reasonable, which makes them harder to spot than an outright crash.

Same Field Name, Two Sets of Semantics

The most common source of error in cross-system integration is a field with the same name and a different meaning. A field called "amount" comes back from the e-commerce platform as an integer in cents, from the payment gateway as a float in yuan, and from the finance system as either tax-inclusive or tax-exclusive depending on which endpoint you called. The Agent extracts one amount from a single user sentence; it passes through four layers — model, tool, gateway, external system — and at every layer it is reasonable to assume nobody converted it.

The key judgment here: unit conversion must happen at the boundary, not in business code. In other words, every numeric value read from an external system must be converted to the internal canonical unit before it enters an internal data structure, and that conversion should be enforced in the tool layer, so business code never gets the option of saying "what I received was cents". The moment one endpoint forgets to convert, the error shows up in the final result multiplied by one hundred — and both a times-one-hundred error and a divided-by-one-hundred error tend to look "about right" on a report.

Timezone problems are sneakier than unit problems, because they involve semantic judgment rather than numeric conversion. A timestamp from a third-party system may be local time without a timezone, Unix time in seconds, or milliseconds. Mixing milliseconds and seconds gives every duration calculation a factor-of-one-thousand deviation, and such deviations show up in logs as "this took a very long time" rather than as "the data is wrong", which makes them extremely expensive to debug. Daylight saving is worse: in some timezones local time happens twice in a year, and a naive time comparison concludes that the order is reversed.

The pragmatic approach is to standardize on timezone-aware timestamps internally, and to parse and assert over external times explicitly in the tool layer: reject immediately if parsing fails or the result falls outside a plausible range (an order creation time more than seventy-two hours in the future, say), so the problem surfaces at the integration layer instead of flowing into business logic and being discovered there.

Amount Precision and Rounding Direction

Amounts are trickier than quantities because they carry precision requirements and rounding rules, and unless those rules are stated explicitly two computations differ by one cent — and one cent is one entire unmatched line during reconciliation. Scenarios that split an amount across systems are worse: a shipping fee has to be apportioned across three orders by their amounts, the shares do not sum to exactly one, and in the end one of them has to absorb the rounding error.

State one rule explicitly here: floor every share except the last, and give the entire remainder to the last one. The value of this rule is not that it is mathematically superior but that it is deterministic, recomputable, and independent of order. Any implementation of the "round each share to cents and then check whether the total adds up" variety turns a reconciliation problem into a random one.

Another question that has to be settled in advance is currency. In a cross-currency scenario: which exchange rate converts "100 yuan" when it is written to an external system, at which point in time the rate is taken, and who absorbs the resulting exchange gains and losses. Unless these are settled, the Agent will reach for whatever the current rate happens to be when it executes an instruction like "convert this amount to dollars", while the correct move is to use the rate locked in when the order was created. Exchange rates, like business rules, belong on the list of things "the model may not invent on its own".

The Three Meanings of Missing Fields, null, and "No Value"

In plenty of interfaces, "the field does not exist", "the field value is empty", and "the field is zero" are three different things — but to the model they may look identical. If a tool represents "no shipping address exists" as an empty string, the model may read that as "the user asked to clear the address", then call the set-address endpoint with an empty value, and the customer's address gets wiped out.

The defense is to model the three states of every field explicitly in the tool contract, and to specify the default behavior for each. The most practical rule is: anything missing from an external response means "unknown" and must never be interpreted as "please clear this". Clearing a field has to be triggered by an explicit user instruction, through a separate path. The rule looks like a small thing, but what it blocks is a class of data destruction that happens very easily and cannot be undone.

Enum Drift and State Machine Drift

Third-party SaaS enums change, and they do not warn you in advance. An order status goes from five values to six, the casing of a shipping status string changes, a status is deprecated but still shows up in historical data. None of these makes your code throw — it just means some branch is now unreachable, or some condition now lands on the wrong branch.

There are three ways to cope. First, map external enums to internal enums at the integration layer, and instead of silently dropping unrecognized values, file them under an "unknown" bucket and raise a metric and an alert, so you know immediately when the other side adds a status. Second, probe actively on a schedule: run through a sandbox environment or a test tenant periodically and take the set difference between the enum values you have actually seen and the ones you know about. Third, run state machine validation before the write, because an illegal transition like "pending payment straight to signed for" may return success from the remote API while your own business absolutely forbids it.

Tenant isolation is computed on the server, not passed as a parameter

The worst accident in a multi-tenant system is a cross-tenant read or write. Technically it usually comes from a very natural mistake: treating the tenant identifier as an ordinary parameter passed into the query method, filled in by the caller. This mistake is especially easy to make in Agent scenarios, because the model sees the field in the tool description and fills it in itself — from whatever was in the context, which may be a sentence from the user, may be a document retrieval, and may just as well be a hallucination.

The right approach is to resolve the tenant identifier from the credential context and inject it uniformly in the data access layer, where neither the model nor business code can reach it. At the same time, make an explicit assertion in the authorization layer: every resource identifier involved in this operation must belong to the current tenant, otherwise reject outright. The assertion costs almost nothing and it blocks the single most severe class of accident — and that class tends to be discovered only long afterwards, because customers do not report each other.

Rate limiting is a quota problem, not a reliability problem

Third-party SaaS rate limits behave worse in Agent scenarios, because Agents naturally produce bursts: a single task might plan a dozen tool calls and fire them all at the same endpoint within a few seconds. In traditional integration you would smooth that with a token bucket, but in an Agent scenario smoothing means the task hangs for a long time and the user sees a spinner.

What is needed here is to let the model participate in scheduling. One workable mechanism is to put "this endpoint is currently congested" into each tool's description and let the model choose an alternative path or batch the calls. That is more flexible than hard queueing at the gateway, because the model has semantic understanding and can tell which calls can be merged and which can be deferred. The cost is that this mechanism depends on the model actually reading that description, so the more reliable approach is to make it a hard constraint in the tool layer: check before the call and, if congested, return a structured "retry later" result, giving the model a definite signal instead of a timeout.

Every mechanism that relies on "let the model weigh the trade-off" needs a hard backstop constraint. The first provides flexibility, the second provides determinism, and only the second guarantees that the system does not fail open when something goes wrong.

Webhook and Asynchronous Callback Confirmation Problems

For some SaaS providers, write operations are asynchronous: the call returns acceptance immediately, and the real result arrives by callback. That pushes the "outcome known-ness" problem from the previous section even further out — you now have to handle a callback that never arrives, a callback that arrives twice, and a callback that arrives out of order.

For interfaces like this, the correct modeling is to make the operation an object with a lifecycle: accepted, in progress, succeeded, failed, timed out. The synchronous return can only advance the state to "in progress"; after that the only things that can advance it to a terminal state are the callback, or a fallback polling query. Callback handling must be idempotent, because the remote side does not promise to send only once; it must verify the signature, because a callback is external input; and it must tolerate out-of-order delivery, because an early failure callback may overwrite a later success callback, so state transitions must be judged by business semantics rather than blindly overwritten.

If an interface is asynchronous, the Agent should not poll and wait for it. The system should own a task object, the Agent's responsibility ends the moment it submits, and the status shown in the interface is updated by the system's callbacks and polling. This avoids both exhausting your quota with the Agent querying in a loop and losing state to a dropped context while waiting.

13. Rollout Order: What to Build First

We have covered more than a dozen mechanisms, and in practice you do not need to build them all at once. There is one ordering judgment that saves a lot of rework: start from your most irreversible action, not from your most frequently called endpoint. There are two reasons. First, an irreversible action has the shortest tolerance window — once something goes wrong there is no second attempt, so you are forced to get it right the first time. Second, irreversible actions are usually infrequent, so mechanisms built on top of them have a manageable maintenance cost, whereas piling mechanisms onto a high-frequency endpoint tends to get bypassed by the team because the friction is too high.

Concretely, the first step is to produce an explicit risk list: enumerate every write operation you are going to expose to the Agent, annotate each one's degree of irreversibility and blast radius, and make the rationale concrete in business terms rather than API paths. This step needs no code, but it determines the order of everything that follows, and it is the only document that will still be able to answer "why did we decide this way" six months from now.

The second step is to issue a capability ticket for the two or three most irreversible actions on that list. This is the foundation every later mechanism rests on, because confirmation points, audit, and the idempotency ledger all hang identifiers off the ticket. Building the ticket first forces you to work out which objects and which fields this operation is allowed to touch — and unless you settle that question, every later mechanism invents its own parameter validation.

The third step is to add pre-write snapshots for these categories of action. The payoff is immediate: it does not prevent the error, but it turns the cost of an error from "unhandleable" into "reversible", and that reduction is enough to make the business side willing to open up write access at the current stage. Their most common worry is "what happens if something goes wrong", and a snapshot plus a human check is an honest answer to it.

The fourth step is the audit trail. Auditing has to go in alongside the earlier steps, because arriving one step late means a batch of unaudited operations has already happened without pre-write snapshots or confirmation credentials, and that batch of data is unaccountable from that point on. The audit trail itself does not have to be complete, but the event skeleton has to be right the first time, because fields backfilled later usually do not line up with the fields recorded at the time, and the value of historical data decays badly during backfill.

The fifth step is the idempotency ledger and compensation. These two can come later, because they mainly solve "duplication" and "partial success", and while you keep traffic limited to a small scope the frequency of those problems does not yet justify the build cost. Wait until the confirmation points and audit are in place, you have started ramping traffic, and you have actually observed duplicate executions — then build the ledger, and you will still know what to build it against.

The sixth step is the mechanism related to two-phase operations. That category belongs last, because it is the most expensive and the most complex, and it is only needed after every other measure has failed. Plenty of teams skip the first five steps and go straight to two-phase, then discover that the real problem was never "the commit was half-done" but "nothing ever checked whether it should have been committed at all".

14. How to Verify That an Agent Wired to Real Systems Is Correct

Testing systems like these is notoriously hard: nondeterministic output, side effects that cannot be replayed, no isolation between environments. But a few techniques are dramatically better value than the others, and are worth building first.

Replaying Recorded Production Traffic

Record real conversations together with the corresponding tool calls, redact them, and use them as regression cases. This solves two problems at once: it prevents changes from introducing regressions, and it turns "how was this task actually handled back then" into a baseline you can compare against. During replay, replace the external system with a behaviorally equivalent fake service — the crucial part being that the fake must be able to produce realistic failure modes: timeouts, throttling, missing fields, duplicate submissions, not just successes. If the fake only returns success, the test covers the happy path, and the happy path is exactly the path with the fewest problems.

Shadow Traffic and Data Reconciliation

Attach the new logic to the read-only path of real traffic, let it compute "what would happen if this were executed" without actually executing, then reconcile that against the real result. The value of this method is that it covers the distribution of real data, and real data distributions are exactly what unit tests can never construct: the empty, the extreme, the oddly formatted, the semantically contradictory — those only exist in the production database.

What you reconcile is not "the results are the same" but "the key fields agree". Systems are full of legitimate nondeterminism — timestamps, generated identifiers, model wording — which was never supposed to match. What has to agree are business fields: amounts, quantities, statuses, target object identifiers. If you do not draw that line clearly, the reconciliation report is all noise, and then nobody reads it.

Reconciliation Jobs as a Source of Test Cases

Every time a reconciliation job running in production finds a discrepancy, it should be turned into a fixed regression case. This is the highest-value technique in this article's view, because it converts operational cost into a quality asset, and the cases come from reality: verified, with no human invention in them. Plenty of teams have test suites made entirely of hand-crafted edge cases, and none of the nasty scenarios that actually exploded in the real world.

Contract Testing Has to Run in Both Directions

Your understanding of a third-party interface will be wrong somewhere, and this class of error shows up as a quiet wrong value rather than an exception, so you have to verify proactively. Do it by periodically running every read endpoint against a sandbox tenant and taking the set difference between the returned field set, enum values, and type distribution and the model you have built in; newly appearing fields and values need manual confirmation of their meaning. Then verify in reverse: construct a request according to your understanding of the contract, send it, and see whether the other side accepts it. This bidirectional check catches two typical errors early: the other side quietly added a field you do not handle, or your understanding of some required parameter is wrong.

Fault Injection Has to Hit the Right Spot

The suspended, timed-out, throttled, and duplicated states were covered earlier; the purpose of fault injection is to verify that the system's classification logic really can handle them. But the injection point has to be chosen carefully. The most effective injection point is not a random timeout but the window after the response arrives and before the state is persisted, because that window corresponds exactly to the worst case of "they executed it, we think it failed". The second effective injection point is after the confirmation credential is generated and before execution, verifying that version validation really does catch a change.

These two injection points share a property: they test the order-sensitive invariants in the system, and order-sensitive logic is precisely the part of Agent integration that is easiest to get wrong and hardest to spot by reading code. Random timeouts are actually mediocre, because most retry logic handles random timeouts correctly anyway — the real problems appear in the cases that "look like a failure but are not a failure".

15. Eight Practices That Trap You

Here are the practices most often criticized in the field. They share one trait: taken individually each looks entirely reasonable, and together they add up to a system nobody can be held accountable for.

First, making confirmation a verbal question instead of a credential. Then you cannot prove afterwards that the person in the loop knew, and you cannot refuse to execute after the data changed. Second, automatically retrying the whole flow after a compensation fails, instead of retrying only the step that failed. Third, classifying every uncertain outcome as a failure, which is the origin of all duplication problems. Fourth, writing permission checks into the tool description and hoping the model complies. Fifth, using retry in place of idempotency. Sixth, stuffing the idempotency key into a custom field of the request body and assuming the other side will check it. Seventh, recording only the outcome of an operation and never the pre-write state. Eighth, setting the timeout for long transactions to a global constant, so it is either far too long or far too short.

Five of those eight share a root cause: treating "unknown" as "failed" or "succeeded". The state a system most needs to model explicitly is the unknown one, and it is the state most easily swallowed by a default. The other three are ordering problems: execute then record, trust then verify, retry then judge. For any code that involves an external side effect, it is worth asking line by line "can this operation be reordered, and what happens if it is?" — for many incidents the answer is sitting in those three questions.

16. Identity and Credentials: Who Is Actually Executing This

Once you are wired into real systems, one question is unavoidable: who actually issued this write? It looks like an audit question, but it is really a permission design question, because it directly determines four things: whether the request is refused, whether quota is consumed, who is accountable, and who the callback belongs to. One vague default value in the integration architecture can cause four problems in four different directions at once.

Acting on Behalf of a User, or on Behalf of the System

There are two modes with completely different semantics. In the on-behalf-of-a-user mode, every call the Agent makes carries the end user's identity, and externally it reads as "this user changed it themselves". In the on-behalf-of-the-system mode, the Agent calls with a service account, externally it reads as "the system changed it automatically", and the user's identity is only recorded in internal logs. The difference matters enormously to the business side: the first mode has to answer "does this user have permission", the second does not — but everything hangs off a single service account, and when something goes wrong only internal audit can trace it back to a person.

Getting the choice wrong has knock-on effects. Support scenarios usually suit on-behalf-of-user, because the business side's existing permission model is already per-user, reusing it is the cheapest option, and the audit lines up naturally. Batch reconciliation, report generation and similar tasks can usually only act on behalf of the system, because the initiator is a scheduled job rather than a person. The place errors happen most often is the mixed scenario: the Agent performs an operation for a user, but that operation triggers a background system task, and when that background task calls another service the identity is lost — the other side's records end up with no user information at all. This kind of identity break is fatal for post-incident accountability, because the record on the other side contains no clue pointing to a specific person.

The pragmatic approach is to make identity travel explicitly with the task instead of relying on the implicit propagation of the call stack: record the initiator's identity in the task context, and take it from the task context for every outbound call rather than from the global session. Then identity survives even when the flow passes through a queue, through another service, and through many rounds of tool calls by the model. The cost is that every service has to explicitly implement "read identity from the context" rather than "read identity from the thread", which requires some discipline and some checking, but is far cheaper than refactoring the other side's system.

Token Lifecycle Has to Be Independent of the Task

When a task might run longer than the lifetime of its token (a queue that sat for hours waiting on human confirmation, a batch task that waited a long time under throttling), token expiry fails the flow midway — and the failure point is often right before or right after an irreversible action, which is messier to handle than a plain failure.

The steadier approach is to make token refresh an automatic behavior of the gateway, handled centrally, and to record in the audit trail "which version of which token this call used" and "whether a refresh happened". That record has real value during an investigation: if an incident involves a permission change, you need to know the user's permission state at the time, and today a lot of systems can only answer "the user's permissions were changed at some point", not "what were his permissions when this call happened".

Temporary Privilege Escalation Must Be Recorded and Must Expire on Its Own

"Temporary privilege escalation" came up earlier as a confirmation point. In implementation it has to be a time-boxed authorization record, not a secret extra permission smuggled into the token and left to expire on its own. The difference is that every single use of the former can be recorded and reviewed, while the latter has only two timestamps — effective and expired — and every call in between leaves no trace.

Keep the time box short. The principle is that the validity period of an escalation should not exceed the time the task needs, plus some margin — not a plausible-looking round number of days. Because the real risk of a temporary permission is not that it persists, but that while it persisted it was used for something you never intended to do: that is, a gap opened between the scope of the escalation and its actual use. Binding the escalation to a specific set of resources and specific actions shrinks that gap dramatically.

17. Cost Gates on Write Operations

The resource and cost issues around reads have been covered thoroughly enough elsewhere. There is one distinctive cost of writes that is routinely ignored: it is amplifiable. If a user's natural language request is read by the model as "update all of these orders", and "these" contains two thousand matching objects inside a context window of ten thousand, then both the cost of the conversation and the scale of the side effects are out of control.

There are three levels of control. The first is intent clarification: when the number of matched objects exceeds a threshold, the correct behavior is to go back and ask the user rather than continue, because the user almost certainly did not realize that the literal meaning of their sentence covers such a wide range. The threshold has to be set small enough that users are not annoyed. The second is an execution quota: within a single task, the number of successful writes of the same kind is hard-capped, and once the cap is exceeded you stop and hand the remainder to a human. The third is a rate limit: the total number of writes allowed per unit of time.

The one most often left out of the three is the first. The second and third are technical measures: they stop the system from being overwhelmed, but they cannot stop the system from correctly doing something the user never wanted. That distinction is critical during post-incident accountability: if a user complains "I never told it to change these", your quota and rate limiting are both working exactly as designed, the responsibility lies at the semantic layer, and the only defense at the semantic layer is the clarifying question inside the conversation.

The other cost judgment concerns the cost weight of a retry. A retry after a failed write is treated as cheap in a lot of systems, but from a business perspective it can be very expensive — if the first attempt actually succeeded, the duplicate records the retry produced have to be cleaned up by hand, and cleanup costs far more than the retry itself. So writes should use a more conservative backoff policy: better to fail fast and route to a human than to keep probing an uncertain outcome.

18. Deduplication Across Tasks

The idempotency discussed so far is all within a single task, with the key derived from the business identifier of the step. But there is a more insidious kind of duplication: the user expresses the same intent in two separate conversations, producing two independent tasks.

The typical scenario goes like this. The user says "change this order to ship tomorrow"; partway through execution it fails because of an external problem; the interface says "operation did not complete, retry?" and the user clicks retry — but the system creates a new task instead of resuming the old one. The two tasks each have an independent idempotency key, so the first call in the second task is not blocked by its own ledger, and a duplicate is produced.

The key to fixing this is to make "the user's intent" a deduplication dimension, not merely "this step". Concretely, generate an intent fingerprint for each task: the task type, the set of target objects, and a normalized representation of the key parameters. Tasks with the same fingerprint may not exist in parallel within a short window — either resume the original task, or explicitly declare the previous task terminated and run its compensation.

The cost of this mechanism is that you have to handle fingerprint collisions — two genuinely different tasks misjudged as duplicates. The standard should be to err toward calling something a duplicate rather than calling it a new task, because the former only costs efficiency while the latter costs data correctness. The fingerprint should include enough distinguishing dimensions to make that trade-off real. A related detail is how the "retry" button is implemented: if it genuinely creates a new task, the interface should not call it a retry but "start over", and that difference in wording noticeably reduces how badly users misread the system's behavior.

19. The Boundaries, the Costs, and the Option of Not Granting Write Access at All

Taken together, the first eighteen sections amount to a fairly heavy body of engineering. Its actual cost includes: maintaining issuance and validation for the capability ticket layer; building tables for the idempotency ledger, handling three states, and having someone write reconciliation jobs; making the saga recoverable and compensable; designing confirmation pages and handling version validation; making the audit append-only, hashing, and periodically anchoring; and normalizing units, timezones, and amounts in the integration layer. All of that is not a small project, and every item is a permanent maintenance burden rather than something you write once and finish.

So there is one question that has to be answered honestly: is this worth it? The inputs to the judgment are not "can it be done technically" but "what is the worst-case loss from an Agent performing one erroneous write in this business scenario" and "how often does this scenario occur". When both are low, the best solution is usually not to build the whole stack above but simply not to grant write access at all.

Three Alternatives to Write Access

Option one is read plus proposal. The Agent reads all the data it needs and produces a structured proposal to be executed; the user reviews it on a separate screen and a human clicks execute. The payoff is that the irreversibility of write operations stops being a problem at all, because the executor is a human and a deterministic system while the model only reads and computes. The cost is a low automation rate and a worse experience than full automation for long flows. Its applicability is broader than people assume — many scenarios that look like they need full automation are really cases where the business side would rather have something that "finishes this step for me" than something that "does this whole thing automatically".

Option two is to split permissions by operation type. Open only the reversible writes and keep the genuinely irreversible part manual. Concretely, let the Agent create drafts, edit internal notes, adjust non-critical internal state, and open tickets it has permission for, while giving it no authority to pay, ship, or send anything outbound. The system still automates a large amount of work, and the small number of high-risk actions go through confirmation. For most businesses the reality is that high-risk actions are only a few percent by frequency while low-risk actions are the overwhelming majority, so this option captures most of the value.

Option three is deferred execution plus batch confirmation. The Agent accumulates everything it wants to do, waits for a batch to fill up or for a time window to close, then presents the whole batch as a checklist for a single confirmation. The value is that it shifts the granularity of confirmation from "per operation" to "per batch", which dramatically reduces confirmation fatigue while keeping human control over irreversible actions. It applies when the operations are mutually independent; if there are dependencies among them, batching makes partial-failure handling far more complicated.

These three options are not mutually exclusive, and the common best combination is: option one covering all irreversible operations, option three carrying bulk execution of low-risk operations, and the middle ground left to full automation. In that combination the fully automated part shrinks down to only those operations that, even when they go wrong, can be detected and repaired automatically.

When This Machinery Is Not Worth It

There are a few classes of scenario where you can say plainly that it is not worth it. The first class is single-tenant internal tools where operations are low-frequency and reversible, such as an ops assistant used inside one team where a misconfigured setting can be restored in five minutes. The full build cost far exceeds the benefit there, and a pre-write snapshot plus an audit trail is enough. The second class is scenarios where writes are worth far less than reads: if the Agent's core value is helping users find and understand information, then all the engineering investment in write operations can be saved, and you can focus on doing reads well.

The third class is subtler: scenarios where the business side cannot accept the latency a confirmation flow introduces. If users expect immediate completion and adding confirmation means waiting several seconds to tens of seconds, the product gets rejected. The right response is not to quietly remove the confirmation but to go back to the three options above and find a path that keeps both control and speed — and the answer is usually batching in option three.

The Real Cost of This Machinery

The engineering cost was covered above; what is underestimated more often is the process and cognitive cost. First, the team gets drained continuously by questions like "why does this one need confirmation and that one does not", and without a clear rules document those questions get discussed over and over. Second, incident handling becomes more ceremonial: the person on call has to learn to read the suspended ledger and the reconciliation report, which requires training. Third, every code change touching a write operation has to consider three questions — is this operation reversible, what do we do when the outcome is unknown, and can it be explained afterwards — and building that way of thinking takes longer than building any one specific mechanism.

What you buy for this cost is a capability: when the system goes wrong, you can state exactly what it did, on what basis, who approved it, and what the data looked like at the time. The value of that capability is invisible when the system is small, and becomes very hard to substitute once the system starts carrying real business volume and has genuinely had an incident. So the right question is not "should we adopt this machinery" but "does the expected lifetime and business criticality of this system justify paying for accountability".

20. The Minimal Sufficient Conditions for Granting Write Access

If this article had to be compressed into a checklist you could verify item by item, the following are the ones this article considers indispensable. Their order is also the recommended build order.

On permission boundaries:

• Writes and reads use different credentials, and neither the model nor business code can obtain the write credential.
• The tenant identifier is derived from the credential and injected uniformly; no layer anywhere accepts a tenant parameter supplied by the model.
• Every write carries a signed capability ticket, and the ticket contains the action, the resource list, the field whitelist, the quota, and the validity period.
• The tool layer accepts only the ticket, never bare parameters; a parameter outside the field whitelist is rejected outright rather than returned as a correctable model error.
• Business legality is decided by the rules engine, not by the model; the model only translates natural language into intent parameters.

On idempotency and duplication:

• Every write has an idempotency ledger record whose state is tri-state rather than boolean, with unknown as an independent state.
• The ledger record is persisted before anything goes outbound, and the uniqueness constraint lives in the database layer rather than in a check-then-insert in application code.
• Non-idempotent operations are explicitly marked, and the quota limits them to at most one attempt, with the remainder routed to a human.
• A task retry reuses the task rather than creating a new one, or at minimum the intent fingerprint blocks duplicate tasks from existing in parallel.

On transactions and failure:

• Every step that participates in a saga predeclares its compensation capability and compensation strength; a step that cannot declare one is explicitly marked as requiring a human.
• The suspended state has its own inspection job, probing uses a read-only interface, and a probe that comes back inconclusive routes to a human instead of resending.
• A failed compensation is marked individually at step granularity and does not trigger a retry of the whole flow.
• The whole saga has a total budget; before each call the remaining budget is checked, and running out goes straight into compensation.

On human intervention:

• The confirmation point for an irreversible action sits before the side effect, and the confirmation credential carries the verbatim parameters rather than a digest.
• The confirmation credential is bound to an object version number or content hash, revalidated before execution, and voided for a redo if they disagree.
• The confirmation credential has a time limit, and confirmation and authorization are two independent records, each able to answer "why was this permitted" and "was a human aware of it" on its own.
• A step with no compensation capability produces an explicit manual work order when it fails, rather than being silently skipped.

On auditing and accountability:

• The audit trail is an append-only event stream where each event carries the hash of the previous one, and the chain's root hash is periodically anchored to external storage.
• Every write event carries a complete pre-write snapshot, and sensitive fields store a hash plus partial plaintext rather than the full plaintext.
• Events record the verbatim parameters actually sent, and the reason the model claims is kept out of the evidence fields as a supporting clue only.
• All unknown operations can be filtered out by final state, the index exists, and the inspection job genuinely runs every day.

21. Availability Is Not What You Should Watch at Runtime

Once you are wired into real systems, availability metrics largely lose their warning value. A system can have a very high interface success rate and very low latency while continuously making wrong decisions — it is simply wrong, very stably. What this kind of system actually needs to watch is a set of criteria about "are writes still worth trusting".

The most important of these criteria is not any absolute value but the trend of the confirmation pass rate. In a healthy system this number should sit at a moderately low level — if it hovers near one hundred percent for a long time, confirmation has degenerated into a rubber stamp that nobody reads any more, and the protection it offers is close to zero. The real signal is a sudden rise over a short window: a jump from sixty percent to ninety-five percent almost certainly means the semantics of some class of operation changed, or that the system is trying to route around confirmation. Turning this into an alerting metric is far more effective than discovering it later while reviewing an incident.

The second is the human intervention rate, but it has to be read in layers. A rise in the overall rate may just be a rise in business volume and not a problem at all; what is meaningful is how the composition by intervention reason shifts. If the share of "suspended, awaiting adjudication" is climbing, the stability of the external interface or your timeout settings has a problem; if the share of "compensation failed" is climbing, the compensation strategy you chose conflicts with the other side's business rules; and if the share of "semantic mismatch, needs clarification" is climbing, that is the signal most worth watching, because it means the model's understanding of user intent is degrading — and that normally happens before the incident does.

The third is the number of duplications intercepted. How many duplicate executions the idempotency ledger has blocked is an indicator very few people look at and a very valuable one. A high number means the system's retry logic, or the retry affordance in the interface, is triggering duplication frequently; a number close to zero should instead make you suspect the ledger is not working at all rather than conclude that the system is unusually clean. A healthy system's value here is relatively high at the start, declines gradually as the frontend interaction and model behavior are tuned, but never falls to zero.

The fourth is the distribution of the time from confirmation to execution. Its use is to detect a subtle kind of degradation: confirmation credentials starting to expire in bulk. If the share of expiring credentials is rising, the user's decision speed no longer matches how fast the system's state changes, and the fix is to shorten what the confirmation shows or to adjust the credential lifetime — not simply to extend the lifetime, which hides a process problem behind a technical one.

22. Permission Tiers Should Be Adjustable

A lot of systems fix the permission tier at launch and then adjust it by feel. A more workable approach is to treat the tier itself as a dynamically adjustable quantity, with adjustment rules explicit enough to execute automatically.

The basic logic is positive feedback toward tightening: when the automatic execution ratio for a class of operation keeps falling, or the human intervention rate keeps rising, or the confirmation pass rate keeps falling below a threshold, automatically degrade that class to per-item confirmation. Loosening has to be more conservative in the other direction: only when a class of operation has had zero human intervention, zero compensation failures, and zero suspensions for a sustained period do you consider moving it from "requires confirmation" to "may execute automatically". The bar for loosening should be markedly higher than the bar for tightening, because the cost of getting these two decisions wrong is completely asymmetric — tightening costs a few extra seconds of human time, loosening costs an incident that may be impossible to take back.

The implicit requirement of this mechanism is that tier changes are audited too, and that the audit can answer "why was this downgraded" and "who approved the restoration". A configuration that records only the current tier and not the history of changes cannot be reviewed afterwards: you will see that the operation was automatic at the time of the incident, but not why it was downgraded earlier and later restored.

Conclusion

Back to the judgment I opened with: the hard part of connecting an Agent to a real business system is not whether that one call can go through, but whether after something goes wrong you can take it back, explain it clearly, and trace it to a person. Everything this article has covered is, in the end, serving those three goals.

If you break those three down a little further, you find the article repeating one principle: model "unknown" explicitly. Why the idempotency ledger uses three states instead of a boolean, why suspension has to be independent of success and failure, why two-phase submission is the one most prone to duplicate submissions, why human confirmation has to be bound to a version number, why the audit trail has to distinguish intent from fact — these are all the same principle at different layers. Anywhere "unknown" can occur, give it a name, a state, and a handling procedure instead of letting some default swallow it.

The second recurring principle is separation of powers. The model handles understanding and expression, not legality and the decision to execute; the system handles deterministic rules instead of relying on probabilistic reasoning; compensation actions are decided by a rule-based mapping table rather than generated on the spot by the model. Once those three lines are drawn clearly, a lot of the risk disappears at the architectural level instead of being patched by defensive code at runtime.

One last point is worth making: the payoff of all this is invisible before something goes wrong. It makes the process a little slower, development a little more complex, and debugging a little more verbose, while the incidents it avoids are pure accounting costs while they are not happening. So the real question is not "is this worth the investment" but "is this system important enough to deserve a system that can explain itself". When the answer is yes, the complexity above turns out to be the cheapest path, because it does not ask you to modify a third-party system, nor to persuade the business side to accept unpredictable behavior — it only asks your own system to be honest enough.

Honest here is meant literally: record the state as it is, mark the parts you do not know as unknown, tell the user plainly the parts you cannot do. A system that is vague about these things looks smoother in the short term, but every one of its failures then requires someone to spend more effort figuring out what happened, and the longer that takes, the larger the loss.

23. Concurrent Writes to the Same Object: A New Race Condition Brought by Agents

The concurrency pressure Agents introduce differs in kind from that of traditional integration. In traditional integration the things that modify the same object at the same time are usually a limited number of fixed paths, and their ordering is something a developer can reason about. In Agent scenarios the things modifying the same object at the same time may be several tasks from one user, tasks from several different users, and a scheduled job colliding with a user task — and the ordering among them is an outcome of model decisions that no developer can know in advance.

The most direct manifestation is lost update. Two tasks both read the current state of the order, both generate a modification plan based on that state, both submit, and the later submission overwrites the earlier one — while the earlier task goes on believing its modification took effect. This class of problem exists in any concurrent system, but Agents make it much more likely, because the model's planning takes an unpredictable amount of time and the probability of collision rises sharply.

The fix is optimistic locking, and it must come with a real conflict-handling strategy. The easy mistake is to add version checking only and then fail and retry on conflict. That is barely usable in a traditional system and produces a very bad experience in an Agent system: the task fails repeatedly, the model keeps changing parameters and retrying, and eventually it may find a way around that looks reasonable but is wrong for the business. The correct approach is to treat a conflict as information that requires replanning rather than as an error: return "this object was modified by someone else" together with the other party's changes to the model and let it re-decide on the new state whether to continue, with a cap on the number of rounds — once the cap is exceeded, route to a human.

A more fundamental judgment is that some conflicts should be solved by narrowing the write, not by retrying. If two tasks modify different fields of the same object they should never conflict in the first place, provided your write is not "read it, modify it, write the whole thing back" but "write only the fields I changed". The difference between the two is subtle in implementation but enormous in effect: the semantics of the former is "the state of this object at that moment", the semantics of the latter is "the new values of these fields". Agent scenarios should enforce the latter.

The Cost of Locking Is Usually Higher Than the Cost of Conflict

Some people reach for pessimistic locking, which is simple and direct. But pessimistic locks are expensive in Agent scenarios for two reasons: how long the lock is held is unpredictable (it depends on how long the model's inference takes), and third-party systems generally do not provide cross-system locks. The usual compromise is to put a very short lock around a single write, protecting only the commit itself rather than the whole read-think-write cycle — that is, to accept that read-think-write is not atomic and to detect the conflict with a version check at commit time.

That is in fact the right choice, because the compromise acknowledges a fact: on a process that spans model inference and multiple external systems, you cannot hold a meaningful lock. Rather than pretending to have atomicity, let conflicts happen explicitly and be handled.

24. Time, Ordering, and Late-arriving Facts

Another hidden source of error across systems is time. In a multi-system integration it is normal for events to arrive out of order and be processed out of order: a webhook may arrive late, a task in the retry queue may complete before a newer task, and two systems' clocks may differ by tens of milliseconds. If the system resolves conflicts with "last writer wins", then an operation that happened earlier may be overwritten by one that happened later, even though the latter was physically written first.

Handling this requires distinguishing two times: event time (when it actually happened in business terms, carried by the operation itself, such as the actual shipment time of an order) and processing time (when your system wrote it). Conflict resolution should be based on event time; processing time is only for monitoring and ordering. Resolving conflicts by processing time works fine on a single machine and is wrong across systems.

Concretely, in an Agent scenario you will hit this case for real: the user says "change this order to ship tomorrow", the Agent changes the status; a few minutes later the user says "make it the day after tomorrow", and the Agent changes it again. The two conversations may be handled by different tasks, and the second task may finish first. If you resolve by processing time that looks correct; but if the second task happened to finish first and the first finished afterwards, resolving by processing time rolls the status back to "ships tomorrow", while the user's last explicit instruction was "the day after tomorrow".

The fix is to have every write carry a monotonically increasing version identifier whose source is a business fact rather than processing order. That business fact can be the ordinal of the instruction within the conversation (monotonic inside one session), or an explicit user-side version. A write carrying this identifier is validated at commit time: if the submitted version is older than the currently recorded one, reject it and return "a newer instruction already exists" instead of overwriting.

Principles for Handling Out-of-Order Callbacks

As mentioned earlier, asynchronous callbacks can arrive out of order. The principle is that state transitions must be idempotent and monotonic. Monotonic means the state may only advance in the business-defined direction and never roll back, and an operation that has reached a terminal state cannot be rewritten by a later callback. In implementation, the callback carries a monotonic ordinal for the business version: any ordinal greater than or equal to the current record is ignored, while an ordinal smaller than the current one whose state is "further along" needs human judgment — because it means something happened that the system's record has already passed.

That last case is the one this article keeps returning to: an inconsistency was detected, but there is no way to automatically decide who is right. The correct handling is to record it explicitly and route it to a human rather than guessing by "the latest arrival wins". Guessing wrong bakes a wrong state into the record — and a wrong state that gets locked in is far harder to deal with than a brief inconsistency.

25. What the System Should Look Like When a Third Party Is Down

Every one of these mechanisms ultimately has to face one question: what do you do when the external service you depend on is itself down. A common but wrong approach here is "retry until it succeeds". Its problem is that while the other side is down for a long time, retrying keeps consuming your quota and connections, while the user can do nothing during the wait and the interface offers nothing but a spinner.

A more sensible design is to make unavailability an explicit state rather than a stretch of silent waiting. Specifically, when a critical dependency fails past a threshold, the system should move the related tasks into a pending-retry queue and tell the user plainly: "the other side's service is temporarily unavailable, your operation is queued, we will retry automatically once it recovers, and we can also notify you of the result now." The value of this is that it translates a technical state into a promise the user can understand — and a promise can be kept or held accountable for.

Another question to prepare for in advance is the degraded path: when writes are unavailable, can reads still work? The answer depends on the business, but what matters more is having a plan rather than deciding on the spot during an incident. A common and effective degraded design is to downgrade writes to "generate a pending change set" and store it, then apply it in bulk once the other side recovers, either by the system or by a human. That design is worth nothing in normal times, but at the critical moment it turns "the business has stopped" into "the business is delayed" — and delay is acceptable, a standstill is not.

The design goal when a dependency is unavailable is not "recover as fast as possible" but "turn the standstill into a delay". A delay has an end and can be communicated; a standstill has no end and can only be discovered by users themselves.

26. Human Adjudication Is Not a Dialog Box, It Is a Workbench

Several earlier sections mentioned "routing to a human", but what the system should hand to that person deserves to be spelled out on its own. Because a great many systems handle this extremely thinly: they forward a raw error message to the user and tell the user to contact support themselves. In practice that outsources the judgment the system should be making to the person with the least information.

A qualified human-adjudication ticket has to contain at least five categories of information, and missing any one of them leaves the handler unable to decide within a reasonable time. The first is what the system is doing and how far it got, described in language a human can understand rather than as a list of step names. The second is what has already happened: which steps succeeded, what observable external results they produced, and what the record identifiers are in the other side's system. The third is where it is stuck: the specific failure signature, the remediation attempts already tried, and why those attempts did not resolve it.

The fourth is a side-by-side comparison of both states, the most critical item and the one most often omitted. If it is stuck in suspension, the interface has to show "the state we recorded" and "the state detected on the other side" at the same time, so the divergence is obvious at a glance. The fifth is the available remediation actions, each annotated with its impact: keep waiting, retry once, correct the local record to match the other side's actual state, initiate compensation, or contact their support. Each action should be executable with one click, rather than leaving the handler to guess which endpoint to call.

There is one design judgment here worth emphasizing: every decision the handler makes has to become a learning for the system. Concretely, if the handler decides "the other side's probe result is unreliable, go with our own record", then that decision itself has to be written into the audit trail with its reason attached, and the distribution of such decisions should be reviewed periodically. If the "go with ours" ratio is unusually high for some class of suspension, it means the probe interface simply does not apply to that class of operation and should be removed from the automatic probing path. This closed loop is one of the few places in this kind of system where improvement actually compounds.

27. A Staged Path from Read-Only to Writable

A lot of teams take this path in practice: first build an Agent that can read and answer, and once that works hand it write access all at once. The first step is low risk and gives fast feedback; the second concentrates risk and surfaces it late. A better path is to widen authorization stage by stage, giving every step a chance to expose problems.

Stage one is pure read access plus proposal output. The Agent reads all the data and produces structured proposals without executing them. This stage surfaces problems in the model's intent understanding and in rule coverage, at very low cost, because every action is reversible. What you watch here is the ratio of proposals the user accepts versus modifies; how those move is the source of confidence for everything that comes after.

Stage two opens up reversible writes. Typical actions are creating tickets, editing internal notes, and updating non-critical statuses. Failure here is cheap enough to run at volume, and the focus is on seeing how the ledger, the snapshot, and the audit trail behave under real traffic. The problems most likely to surface at this stage are concurrency conflicts and enum drift, because real data is far dirtier than test data.

Stage three opens up irreversible writes that require per-item confirmation — payments and outbound sends. The focus is no longer whether it can be done but how information-dense the confirmation point is and how confirmation fatigue is managed. This is also where you start dynamically adjusting permission tiers by operation type, because the misoperation rate differs sharply between operation types.

Stage four is where low-risk types get full automation. Only types that ran long enough in stage three with zero human intervention and zero compensation failures reach this tier. Most teams never get to stage four, but capturing most of the business value already in stage three is itself the point of designing the path this way — it makes "not going all the way" an acceptable end state.

One easily overlooked obstacle to progress deserves a final note: every stage requires the business side to re-confirm the risk they are willing to accept, and that process is often slower than the engineering. So from day one you should write down "which tier we will reach and what the risk of each tier is" as a document jointly maintained with the business side, rather than explaining it once the system is built. That document is also the most useful thing you will have after an incident.

28. Two Closing Techniques That Are Easy to Skip

Constructing the Idempotency Key Is Harder Than the Idempotency Mechanism Itself

In a lot of implementations the idempotency key is a random number thrown together on the spot, for instance by calling a random generator once when the task starts. That only solves deduplication for "retries within the same task"; it does nothing about the cross-task duplication mentioned in section 2, and its cost is that as soon as the task runs again, the same user intent produces a brand new key that the ledger cannot catch.

The far more useful construction is to derive a stable key from business facts. Building it takes three steps: first determine which objects this operation targets and concatenate their stable identifiers in sorted order; then determine the key parameters of the operation, such as action type, target state, and the quantities involved; finally combine the two into a string and take its hash as the idempotency key. The property of this key is that repeating the same user intent produces the same key, while different intents produce different keys.

The cost of this construction is that you have to work out which parameters belong to the intent and which are implementation details. Implementation details must absolutely stay out of the key, otherwise the same intent produces different keys before and after an upgrade and deduplication breaks; everything that belongs to the intent must go into the key, otherwise two genuinely different operations get misjudged as duplicates. Those judgments require business knowledge and cannot be derived from a general rule, so this layer should be written together with the specific operation types rather than built as a generic function.

One more detail is worth noting: the stability of the key depends on the object identifiers being concatenated being stable themselves. If the object identifier is versioned or timestamped on every query, the key changes on the second run. So when you take an identifier from an object, specify exactly which field you are taking, and forbid any field that varies over time.

Dry Run as the Last Line of Defense

Two-phase confirmation and the shadow table were both covered earlier, and each of them implies a capability: computing what this execution would do without producing any external side effect. That capability is worth pulling out on its own, because it can be used independently of any transaction mechanism as a universal guard before every irreversible action.

A dry run lets the execution path run through once but replaces the final external write call with a local no-op: parameters are assembled as usual, validations run as usual, permissions are checked as usual, and only the last step is intercepted to record "these are the parameters I am about to send". The criterion for a passing dry run has to be complete: every validation passed, the permission check passed, the rule engine deemed it legal, and the parameters assembled successfully. A failure at any step surfaces during the dry run, at zero cost.

The limits of the dry run should be stated as well: it cannot guarantee that the other side's state at real execution time matches what it was during the dry run, and it does not cover the other side's own business validation failures. So it is a pre-check that catches a great deal, not a mechanism that can replace confirmation. The sensible use is to place the dry run before the confirmation — generating the content of the confirmation page from the dry run result, so that what the user sees under "what will happen" is genuinely computed rather than described by the model. As noted earlier, the most valuable information on the confirmation page is the verbatim parameters, and a dry run happens to produce a trustworthy copy of them.

There is one easily missed implementation detail: the dry run must travel the same code path as the real execution, differing only in the last step. If the dry run has its own separate validation logic, its divergence from the real execution becomes a new and extremely hard-to-detect class of error — the system shows everything as normal during the dry run and fails during actual execution, and investigation of such a failure usually gets misdirected toward "is this dry run stale?".

29. Which Layer These Things Belong In, and Who Maintains Them

Every mechanism discussed so far faces the same organizational problem in implementation: they belong neither to business logic nor to infrastructure, and put in the wrong place nobody maintains them. Placed too close to the business side, every business team implements its own copy, and the quality depends on who happens to be on that team; placed too close to infrastructure, the business side feels blocked and ends up routing around it to call the API directly.

A workable division has three tiers. The first is the integration layer, responsible for normalizing units, timezones, and amounts, for enum mapping, for tenant injection, for token refresh, and for semantic guards such as "a missing field does not mean cleared". This tier should be one adapter per external system, with business code facing only a unified internal interface. One test of whether it is in place is that you should be unable to find a single external system's field name or API path anywhere in business code; if the search comes up empty, the boundary is clean.

The second is the execution layer, responsible for issuing and validating capability tickets, the idempotency ledger, saga orchestration, confirmation credentials, the three-state classification of call outcomes, and persisting audit events. This tier is generic — every write passes through it — so it has the strictest stability requirements and is the least suited to each team writing its own copy. One test of whether it is in place is whether you can find one place in the code that answers "which idempotency key did this call use, what authorization did it pass through, and what was the final state"; if the answer is scattered across several places, this tier has not converged enough.

The third is the business rule layer, responsible for the things only the business side knows: which orders can be merged, in which cases a human is mandatory, which business action compensation should take. This tier should be owned and modified by the business side, with the engineering team providing the execution mechanism. One test of whether it is in place is whether a business rule can be changed without a code release, for example by configuring it in the rules engine or through a configuration screen. If changing one business constraint requires a release, the rules are coupled too tightly to the code, and the business side will go around the system and edit the data itself.

The tier most often skipped in this division is the third. The team's inertia is to write business rules into engineering code because it is faster and because engineers are more comfortable with it. The consequence is that the business side loses its sense of ownership of the rules, and ends up either filing a request every time or bypassing the system to edit the database by hand — and the latter is the hardest source to trace of all consistency problems.

30. The Contract Maintained Jointly with the Business Side

Last, the risk list and tier document mentioned repeatedly above deserve a word of their own. Their importance lies in being the only artifact that crosses team boundaries: the engineering team uses it to guide implementation, the business side uses it to decide what to expose, and both use it to align expectations. It has to be maintained jointly, not written by the engineering team and handed over.

There are several things worth writing down clearly in this contract: which operations are already open to the Agent, how far open they are, the degree of irreversibility and blast radius of each, which confirmation path each takes, the current human intervention rate of each, and the conclusions of the most recent incident reviews. That last one is especially useful — writing an incident's root cause back onto the corresponding line, so that the line slowly becomes the most valuable operations knowledge you have, and someone new to the team can read it once and understand why each class of operation is the way it is.

This contract also has a side effect worth looking forward to: it will naturally generate recommendations for tier adjustments. If one line on the list has had a zero compensation failure rate and a zero human intervention rate for three consecutive months, it is a candidate for the next tier of automation; conversely, if a line's confirmation pass rate is one hundred percent, its confirmation content needs to be redesigned. The contract therefore becomes a living document rather than a one-off review artifact.