Long-Running Agent Engineering: Task Graphs, Resumption, and Cost Control

Why an Agent Cannot Run for a Long Time

Hand an Agent a task of the shape "search, filter, write a report," and you can comfortably let it run for ten minutes. Swap in something like "scan the whole repository for circular dependencies, produce a fix patch for each one, and run the test suite against every patch," and the same Agent will most likely start breaking down around minute thirty: the model begins repeating files it has already read, empty tool returns get treated as valid input, the context snowballs until the critical constraints no longer fit, and what finally comes out is a summary that reads smoothly while having forgotten every conclusion accumulated along the way.

These failures rarely show up in the form of an error. More often the process is still alive, the exit code is still 0, the log even contains the words "task complete," and yet the thing that came out the other end is worthless. The root cause is that the mental model behind most Agent runtimes is still "one session equals one task": state lives in memory, the model has to rebuild the world out of conversation history at every single step, and the moment that history gets truncated or compacted, the world has been silently rewritten.

What a long-running task actually needs is not a model that can keep working for longer stretches of time, but the task lifecycle decoupled from the process lifecycle. The process is allowed to die; the task is not. The machine is allowed to change; the state is not allowed to be lost. That sentence sounds almost flat, but it decides nearly every architectural choice that follows: why there has to be a task graph that is independent of the conversation, why checkpoint granularity should be set by "the density of irreversible operations" rather than by "time intervals," and why, when a user changes requirements halfway through, you cannot simply truncate the context back to some point and start a fresh round.

Three Time Scales: Do Not Mix Them Together

When people discuss long-running tasks, the easiest mistake to make is to shove three completely different time scales into a single "timeout/retry" mechanism, and end up with nothing clean at any of the three layers. At a minimum they include:

• The single-step time scale: one model call, one tool execution, one batch of subtasks. The order of magnitude is seconds to minutes, and the failure modes are timeouts, rate limiting, and malformed returns.
• The stage time scale: a node or a sub-chain in the task graph, something like "classify all candidate files." The order of magnitude is minutes to tens of minutes, and the failure modes are getting stuck in a loop, settling into a local optimum, or repeatedly second-guessing itself.
• The whole-task time scale: the entire task from start to delivery, potentially spanning hours or days, with human waiting and human intervention interleaved. The failure modes are budget exhaustion, requirement drift, and changes in the outside world (a dependency upgrade, someone else having edited the file).

These three layers need three different sets of mechanisms. Retry and timeout on the single-step layer are pure engineering problems, and a client library handles them perfectly well. The stage layer needs progress estimation and stall detection -- and note that this is not the same thing as a timeout. "This subtask has been running for twelve minutes" does not mean it is stuck; what matters is judging whether its output per unit of time is still growing. The whole-task layer needs budgets, pausing, resumption, and a channel through which a human can intervene.

• The simplest signal for deciding whether a long-running task is still worth watching: take the tool output from the last N steps, deduplicate it, and measure how much genuinely new information it contains. If the new-information count is zero across several consecutive steps, then no matter how long it keeps running, it should be treated as stalled rather than as "working hard."

The rest of this article follows six directions: persistence, idempotency, human intervention, budget, observability, and rollback. But these are not six parallel topics. Persistence is the foundation, idempotency is the load-bearing wall built on that foundation, and the remaining four all rest on the premise that "the run can be resumed from any point at any time."

Task Graph and State: What to Persist and What Not To

The easiest dead end in state persistence for long-running tasks is the belief that "you just store the conversation history in a database and you're done." That path is good enough for tasks under two hours, because it reuses abstractions the Agent framework already has. But the moment you have to cross processes, cross machines, or cross days, it runs into three concrete problems:

1. Conversation history is a narrative, not state. History is "a sentence ordered by time." It does not tell you which steps already completed, which side effects already landed, or which conclusions have since been overturned by later evidence. Re-deriving state from history means asking a model that may well be wrong to perform bookkeeping that deterministic code should have done.
2. History balloons. A few hours of tool calls can easily pile up several hundred thousand tokens. Storing it is not hard, but loading all of it on every recovery is both slow and expensive, and it mixes a large amount of low-value content (retried stack traces, repeated directory listings) into the model's field of view.
3. History cannot express parallelism. Three subtasks run at the same time and one of them fails. History is a linear sequence, so there is no way to read the structured fact "A and C are already done, B is still running" out of it.

The correct approach is therefore to introduce a independent task graph structure: nodes are task units, edges are dependencies and data flow, and each node records its own status, an input digest, output references, and the effect log needed for idempotency. Conversation history is demoted to one kind of "artifact produced inside a node" -- the prompt and the return value for a particular step -- and is no longer the only basis on which state can be recovered.

A Task Graph Representation That Is Good Enough

What follows is a deliberately trimmed version. In a real system nodes carry more fields, but these are the minimum set required for recovery:

from dataclasses import dataclass, field
from enum import Enum
from typing import Any


class NodeStatus(str, Enum):
    """节点状态机。恢复逻辑只允许从这里规定的几个终态往外走。"""
    PENDING = "pending"        # 未开始
    RUNNING = "running"        # 已认领，进程可能已经死了
    BLOCKED = "blocked"        # 依赖未满足或等待人工决策
    SUCCEEDED = "succeeded"    # 完成且副作用已确认落地
    FAILED = "failed"          # 失败，是否可续跑由 retryable 决定
    ABANDONED = "abandoned"    # 被外部判定放弃，保留用于审计


@dataclass
class Node:
    """任务图中的一个节点。

    key 必须是稳定且可重算的，不能用随机 UUID——
    否则重放同一步骤会生成一个新节点，幂等就无从谈起。
    """
    key: str
    kind: str                              # 节点类型：llm_call / tool / human / subgraph
    status: NodeStatus = NodeStatus.PENDING
    deps: list[str] = field(default_factory=list)

    #产出与输入分离：产出存引用（比如对象存储路径），输入存摘要
    input_digest: str | None = None         # 输入指纹，用于检测「依赖变了但我没重跑」
    output_ref: str | None = None           # 产出落盘位置

    #幂等三件套
    effect_key: str | None = None           # 本节点所有外部副作用的唯一标识
    effect_log: list[str] = field(default_factory=list)  # 已确认落地的副作用 id
    compensations: list[str] = field(default_factory=list)# 补偿动作 id，按逆序执行

    attempt: int = 0
    retryable: bool = False                # 失败后能否原地重试
    resume_from: str | None = None          # 断点续跑时指向的上一个成功节点 key
    cost_used: float = 0.0                  # 已消耗预算（token 折算）
    wall_used: float = 0.0                  # 已消耗墙钟时间（秒）
    meta: dict[str, Any] = field(default_factory=dict)


@dataclass
class TaskGraph:
    nodes: dict[str, Node] = field(default_factory=dict)
    root: str | None = None

    def add(self, node: Node) -> Node:
        self.nodes[node.key] = node
        return node

    def ready(self) -> list[Node]:
        """返回所有依赖已满足、可立即执行的节点。

        注意：RUNNING 状态不算 ready。进程重启后残留的 RUNNING
        必须先被显式回收成 PENDING，否则既不会被执行也不会被监控。
        """
        out = []
        for n in self.nodes.values():
            if n.status != NodeStatus.PENDING:
                continue
            if all(self.nodes[d].status == NodeStatus.SUCCEEDED for d in n.deps):
                out.append(n)
        return out

    def reclaim_orphans(self, live_run_ids: set[str]) -> list[str]:
        """回收孤儿节点：状态是 RUNNING 但持有它的进程已经不在了。

        这是「进程重启后状态还在，但活儿没人干」这一类问题的唯一解法。
        返回被回收的节点 key 列表。
        """
        reclaimed = []
        for n in self.nodes.values():
            if n.status != NodeStatus.RUNNING:
                continue
            if str(n.meta.get("run_id")) in live_run_ids:
                continue
            n.status = NodeStatus.PENDING
            n.meta["reclaimed_from_run"] = n.meta.get("run_id")
            n.meta["reclaim_count"] = n.meta.get("reclaim_count", 0) + 1
            reclaimed.append(n.key)
        return reclaimed

There are two points in the snippet above worth emphasizing on their own, because a great many implementations miss them.

First, the node key must be recomputable. A lot of teams use a UUID as the node id, which turns "replaying the same step" into "inserting a new node" at the data layer. Idempotency becomes unachievable, and the state machine splits into two parallel branches. The correct approach is to derive the key from the input content, for example hash(node kind + input digest + dependency output digests). That way a replay lands on the same key, and only then is the recovery logic justified in doing "skip what is already done and reuse its output."

Second, RUNNING is not a safe state. It only means "some process claimed at some moment that it was working on this." Once the process is kill -9'd, that state sits there forever as a zombie that is neither failed nor completed. The first step of any recovery flow has to be lease-based reclamation: every RUNNING node carries a run_id and a heartbeat timestamp, and when a new process starts it reclaims "RUNNING nodes whose heartbeat has expired" back into PENDING (or declares them FAILED outright, depending on whether that node is re-entrant).

Checkpoint Granularity: Cut Along the Density of Irreversible Operations

Checkpoint granularity is the part of this section that genuinely requires a decision. Too coarse and one failure wastes hours of work. Too fine and the bookkeeping overhead and state volume explode, and "observable progress" degenerates into a pile of noise. There are three common ways to cut:

• Cut by conversation turn: save once every N model calls. The easiest to implement, at the cost that every side effect between two checkpoints lives only in memory, so if the process dies, all of it has to be redone.
• Cut by step: save once before and once after each tool call. The finest recovery granularity, but the state write volume equals the number of tool calls, and many steps are themselves extremely cheap (reading one file, say), so giving them a dedicated checkpoint is not a good trade.
• Cut by side-effect boundary: merge consecutive read-only steps into one segment, and place a checkpoint only before and after "a step with an irreversible external write."

The third is the only default I would recommend for real long-running tasks, and the reason is simple: the cost of replaying a read-only step is close to zero (read the file one more time, make one more search call, waste a few tokens at most), whereas the cost of replaying a step that writes externally can be incalculable. Aligning checkpoints with side-effect boundaries means that recovery only ever has to replay read-only work between two side effects, which is a bounded and controllable cost. Put a checkpoint in the middle of two side effects instead, and what you are left with is "I may already have written it, but I have no way of knowing whether I did" -- which is the hardest problem there is.

When you get down to implementation, one option is to tag each node with a risk label:

Node kind	External side effect	Replay cost	Recommended checkpoint
Read a file / list a directory / query an index	None	Tokens only	Mergeable; save once at the end of the segment
Call a read-only API (query style)	None	Tokens plus rate-limit quota	Mergeable; watch the rate limit
Write a local file (draft, overwritable)	Local, idempotently overwritable	Low	Save immediately after the write
Commit to git / open a PR / send a message	Remote, visible, hard to retract	High, and it interrupts people	Save the intent before the write, the confirmation after it
Place an order / make a payment / delete a resource	Irreversible	Very high	Force a human-confirmation node

Pay attention to the last two rows. For actions such as "commit a PR" or "send a message," saving a single checkpoint after the write is not enough -- if the process dies between the write and the save, recovery will replay the action, and you will end up with a duplicate PR sitting in the repository. Actions of this kind have to use the three-phase pattern: write the intent first, then execute, then confirm:

async def run_node_with_intent(graph: TaskGraph, node: Node, store) -> None:
    """带意图日志的副作用执行。

    核心不变式：任何远端可见的副作用，在写入外部系统之前，
    必须先在本地持久化一条「我打算做这件事」的记录。
    """
    if node.effect_key and node.effect_key in node.effect_log:
        #已经确认落地过，直接复用，不重放
        node.output_ref = node.meta.get("output_ref_from_first_run")
        node.status = NodeStatus.SUCCEEDED
        await store.save(graph)
        return

    #第一段：写意图。必须在调用外部系统之前落盘。
    intent = {
        "effect_key": node.effect_key,
        "node_key": node.key,
        "payload_digest": node.input_digest,
        "ts": now(),
    }
    await store.append_intent(node.key, intent)

    #第二段：执行。副作用 ID 用 effect_key 派生，保证幂等键稳定。
    ext_id = await external_api.create(
        idempotency_key=node.effect_key,   # 外部系统支持的话，这是最省事的一层
        **node.meta["request"],
    )

    #第三段：写确认。哪怕这一步之前进程死了，
    #恢复时也能靠 idempotency_key 或外部查询把这次操作认领回来。
    node.effect_log.append(node.effect_key)
    node.output_ref = ext_id
    node.status = NodeStatus.SUCCEEDED
    await store.save(graph)

How exactly effect_key should be constructed is worth its own paragraph. It has to satisfy two conditions: the same logical side effect gets the same key under any retry, and different side effects must never collide. The most robust practice in the field is to make it a combination of "target location + action type + digest of the payload," for example pr:repo-a/branch-x:create:<payload_hash>. If the same PR has to be revised twice because the requirements changed, the two keys are naturally different and both revisions will execute -- which is exactly the behavior we want.

The Idempotency Premise of Resumption: What Replaying a Step Actually Repeats

The phrase "make the operation idempotent" appears everywhere in documentation and almost never survives contact with real code, because everyone quietly assumes that "replaying a step means executing the same operation again" -- when the real risk is that the step being replayed is usually not an atomic operation at all. It is composed of several sub-actions, some of which cost almost nothing to replay and others of which cost a great deal.

Worse, the semantics of a replay drift over time. Replaying the step "fix the circular dependencies" may simply no longer be reasonable two hours later: someone else has edited the files, the dependency versions have been upgraded, the previous step's conclusion rested on a stale directory snapshot. So the correct posture for resumption is not to replay blindly, but to first determine whether the output of the last successful node is still valid, and only then decide whether to replay or to skip.

Three Kinds of Idempotency: More Than "Calling the Same Function Twice"

If you take the actions inside a single node apart, you will normally find three completely different kinds of side effects, and they need completely different handling:

• Read-class effects: naturally idempotent. The only cost of a replay is tokens and rate-limit quota. You can replay them freely -- but remember that what you read may have changed in the meantime, so a read result has to enter state carrying "the timestamp at which it was read." Otherwise, after recovery you have no way of knowing how stale the data in your hands is.
• Local-write effects: as long as it is "overwrite the same path," it is basically idempotent. The most dangerous variant is "append," because if the recovery flow replays two appends to the same log file, you get duplicate lines. Either convert these operations into overwrites, or put the effect_key inside the file content and deduplicate by key when reading.
• Remote-write effects: sending a message, opening a PR, calling a payment API, triggering CI. This is the part that genuinely requires cooperation from the external system.

For the third category there are three levels of implementation strength, from weakest to strongest:

1. Lean on the external system's idempotency key. A great many modern APIs accept an idempotency key and guarantee on the server side that repeated requests carrying the same key take effect only once. This is the least effortful option, so look for it first.
2. Query before you write. Before executing, query the external system once and check whether the side effect already exists, matched on some unique marker (for a PR, the title plus the branch; for a message, a hash of the content). Fragile, but the only option available when there is no idempotency-key support.
3. Reconcile and repair afterwards. Allow the operation to execute more than once, but pair it with a periodic reconciliation job that cleans up or flags whatever it finds duplicated. This approach cannot be used on its own; it can only act as a backstop for the first two, because the cleanup action has its own window of exposure.

• A pragmatic judgment: if your action cannot express "I intend to do this" in any form, and cannot be looked up afterwards, then it should not be automated at all -- it should be demoted to a human-confirmation node. In a long-running task system, a human node is not a failure. Mixing unobservable automation into the system is.

There is an idempotency trap here that tends to be overlooked: an LLM node is almost never idempotent. Replay the same node twice and the text the model produces may differ (and even with a temperature of zero, the model service's implementation makes no guarantee of bit-level consistency). So idempotency for an LLM node does not come from "replaying it does not produce a different result," but from "replay it exactly once, then freeze the result": as soon as the node's output has been durably written, recovery reads it from disk and never calls the model again. Get this one wrong and a task that recovered three times will have produced three reports worded differently enough that their conclusions may contradict one another, leaving a human with no way to decide which one to believe.

Working alongside that is the "input fingerprint" mechanism. Compute an input_digest for every node, derived jointly from the output digests of its direct dependencies. During recovery, if you find a node that is already SUCCEEDED whose dependency output digests no longer match what was recorded at the time, that means something upstream changed -- and at that point you cannot simply reuse its output. You have to mark it as invalidated and, following the invalidation-propagation rules, void the downstream nodes along with it. This chain is the technical precondition for "the user changed requirements halfway through" being handled correctly at all.

Interrupting and Resuming the Human-in-the-Loop: How to Continue From the Middle After a Requirement Change

Everything so far has handled "the machine died unexpectedly." This section handles "a human deliberately called a halt and then came back with a different requirement." This is where long-running task systems most easily produce a bad experience, because a lot of implementations reduce it to "save the context, and when the user comes back they continue the conversation" -- which is precisely the wrong thing to do.

Here is why it is wrong. A user leaves, comes back two hours later, and says "forget those 20 legacy modules for now, just handle the batch we added last week." That sentence changes the set of goals of the task, not the wording of the task. If you treat it as a new user message appended to the history, the model is left facing a self-contradictory context: the reasoning of the preceding dozens of steps was all built on the premise of "process every module," and the newest message negates that premise. The model usually will not error out. It will try to reconcile the two and then produce a result that blends both sets of assumptions.

The correct approach is to model a "requirement change" as an invalidation event on the task graph, and let the propagation rules decide which completed work is void and which still stands. To do that you have to take one preliminary step: group the nodes by the "task premises" they depend on.

Premise Fingerprints: Making the Invalidation Scope Computable

When each node executes, it should record, in addition to input_digest, a set of precondition tags: which task-level assumptions it depends on. The precondition tags of a node like "filter the modules to be processed" might be {"scope": "all_modules"}. Once the user changes the scope to {"scope": "new_modules"}, you do not need to understand a single piece of business semantics; a set difference over the tag values is enough to compute which nodes had their output invalidated.

def invalidate_by_precondition(graph: TaskGraph, changed: dict[str, str]) -> list[str]:
    """前提变化后，算出必须作废的节点集合。

    changed 形如 {"scope": "new_modules"}，表示某个维度的
    任务前提被改成了新值。只影响依赖该维度的节点。
    """
    stale = []
    for node in graph.nodes.values():
        pre = node.meta.get("preconditions", {})
        if any(pre.get(k) != v for k, v in changed.items()):
            stale.append(node.key)
            #连带：任何依赖了 stale 节点的下游，其产出也不可信了
            for other in graph.nodes.values():
                if node.key in other.deps and other.key not in stale:
                    stale.append(other.key)
    return stale

The real payoff of this function is not the amount of code, but that it turns "continuing from the middle" from a vague product problem into a determinate data problem. Run it once and you know: 14 nodes need to be re-run, 9 can be reused, and 3 of them have to be re-confirmed with the user first. Without this capability the product side can only produce the answer "sorry, the requirements changed too much, we recommend starting over" -- which is precisely the answer users can least accept.

The Semantics of Pausing Have to Be Distinguished

A lot of systems have a single "paused" state, which leaves them unable to express different intentions. There are at least four in practice, and their recovery paths are completely different:

• Park: the task itself is fine, it is just waiting on an external signal -- a CI result, or a reply from the user. It can be resumed in place, but while it waits it should not keep consuming budget, and stall detection should not judge it as hung.
• Narrow: the scope got smaller, such as doing only the newly added modules. The 9 nodes already completed remain valid, and only the excluded portion is invalidated.
• Rewind one step: a completed node is proven to have been computed wrongly, and it plus its downstream have to be recomputed. This is the path most often triggered in recovery logic, and the hardest one to get right.
• Abort: the task no longer makes sense. Keep the state for auditing, but produce no further action.

Collapsing these four into a single paused boolean saves a handful of enum values in the short run and costs a great deal in the long run. The hardest case is precisely "narrow," because it requires preserving completed outputs while invalidating an exact subset, and the other three never need that kind of fine-grained surgery. Once the categories are clear, the amount of code actually goes down.

• A rule of thumb for judging whether a pause design is sound: while a task is parked, timeout monitoring and stall detection must both be disabled. Otherwise your system will spend the two hours of a user's lunch break repeatedly alerting on a node that is merely waiting, and the resulting alert fatigue means that a genuine hang goes unnoticed by everyone.

The Granularity of Re-confirmation

When the user comes back, the Agent should proactively restate "what is kept, what is void, and what runs next," and should ask questions only about divergences that cannot be inferred. The criterion is simple: if the new requirement can be derived from the old one plus a small number of constraints, do not ask; if it cannot, do ask -- and ask everything at once. This has a far bigger effect on experience than any difference in model capability. What users complain about is usually not "the result was wrong," but "it could have asked me directly, and instead it guessed a direction and ran for twenty minutes."

Cost Ceilings and Time Budgets: A Budget Has to Be Enforced, Not Suggested

A long-running task has two sources of cost with completely different natures, and they need to be governed separately: token cost, which grows with the number of steps, and attention cost, which is the time users are forced to spend waiting and reviewing. The latter is often more painful than the former, yet it is not measured at all, which makes it very hard to even enter the optimization conversation.

On the subject of budgets there is one piece of experience worth emphasizing on its own: writing the budget into the prompt as a line that says "please watch the cost" has almost no effect. The reason is not hard to see -- the model sees after the fact how much it has already spent, while a constraint in the prompt is an upfront intention, and there is no enforced link between the two. What actually works is enforcing it at the scheduling layer: before dispatching the next node, always ask "if I dispatch this node, does the budget still cover it?"

Three Tiers of Budget Gate

The practical approach is to set up three gates, each corresponding to a different degradation action:

Gate	Trigger condition	Degradation action	Why it works
Soft threshold	70% of the budget used, or remaining time below 70% of the total duration	Compact the context, trim history, disable low-value tools	What it saves is the unit price, not the planned amount of work
Hard threshold	100% of the budget used	Only allow executing the wrap-up nodes on the whitelist	Guarantees there is always a deliverable rather than a half-finished job
Emergency stop	Either budget or time exceeds 100%	Freeze the task immediately and emit the current best result plus a list of what is unfinished	Avoids losing everything to a process killed midway

That "wrap-up node whitelist" inside the hard threshold is what keeps the experience of a long-running task under control. If a task is stopped at 95% and produces a half-written document, that is worth zero. But if, at the scheduling layer, you have pre-marked nodes such as "generate the report," "generate the list of unfinished items," and "notify the user" as allowed to execute regardless of budget, then when the budget runs out you can still hand over "18 of 24 items completed; the remaining 6 were not executed because of the budget cap, here is the suggested next step." In a long-running task, this is the single highest-leverage design detail.

Time Budgets Need More Active Management Than Token Budgets

Token consumption is roughly proportional to the step count and is therefore predictable, whereas wall-clock time is often stretched by factors you do not control: rate-limit backoff, network jitter, and subtasks that unexpectedly have a long tail. If you do not distinguish the two, a task with "40% of the budget left but only 8 minutes remaining" will wrap up in a rush at the very last moment, and quality will fall off a cliff.

class BudgetGovernor:
    """在派发节点前做准入判断，而不是在超限后才报错。"""

    def __init__(self, token_cap: float, wall_cap: float, soft_ratio: float = 0.7):
        self.token_cap = token_cap
        self.wall_cap = wall_cap
        self.soft_ratio = soft_ratio

    def admit(self, node: Node, used_token: float, elapsed: float) -> str:
        #白名单节点永远放行：收尾产出比预算纪律更重要
        if node.meta.get("always_admit"):
            return "admit"

        token_left = self.token_cap - used_token
        time_left = self.wall_cap - elapsed

        #估算这一步的开销，留出 20% 余量给不确定性
        need = max(node.meta.get("est_token", 0) * 1.2, 1)
        if token_left < need:
            return "deny_hard"
        #剩余时间已经不够再跑一个常规节点
        if time_left < node.meta.get("est_wall", 30) * 1.5:
            return "deny_hard"

        if token_left < self.token_cap * (1 - self.soft_ratio):
            return "admit_degraded"   # 允许，但要求压缩上下文
        return "admit"

Pay attention to the two estimates est_token and est_wall. They do not need to be very accurate -- in practice the historical mean per node kind is enough, and even the median across nodes of the same kind will work. What matters is that the field exists at all: a scheduler with no estimates can only discover the problem after the money is gone, whereas a scheduler with rough estimates can allocate the remaining budget to the handful of nodes that matter most in advance. This "budget-aware task ordering" is itself a quality improvement: it makes the system do valuable work first under resource pressure instead of bumping into a random order.

While we are here, the return value admit_degraded deserves to be unpacked. Compacting the context is not the same as simply truncating history -- truncation is in fact the approach most likely to lose the critical constraints. Correct compaction preserves four things: the core constraints of the task, the key conclusions of the earlier steps, the stage the run is currently in, and the questions still left open, while discarding the search noise in between. The criterion for deciding whether a stretch of history is "noise" is whether it will still affect subsequent decisions, not "whether it is long."

Progress Observability for Long-Running Tasks: Making People Willing to Leave It Running

A task that runs for four hours, with those four hours a complete black box in the middle, will never actually be trusted by anyone. This is not a psychological problem but an engineering one: at any moment the user has to be able to answer, in one sentence, "what is it doing, how much is left, and is it worth continuing to wait for." A system that cannot do that leaves the user with exactly one rational option, which is to refresh every five minutes and then give up.

Heartbeats Have to Carry Information, Not Just a Timestamp

The most primitive approach is to have each node update a heartbeat timestamp, and that only answers "is it alive," never "what is it doing." A useful heartbeat has to contain at least three things: the key of the node currently executing, the number of work units completed within that node and the total, and the cumulative cost of this run. Only the three of them together support even a rough estimate of how much remains.

There is one critical detail when estimating the time remaining: use a quantile, not an average. In a four-hour task the first few nodes are usually fast (they just read a few files), and later nodes get slower and slower (they have to run tests, they have to wait on CI). Extrapolating linearly from the average systematically underestimates the remaining time, and that directly causes budget-aware scheduling to wrap up in a rush at the end. Estimating with P80 or P90 is on the conservative side, but what the user sees is a slightly pessimistic number, which is almost always better than an optimistic number that gets proven wrong.

More important than that is expressing uncertainty honestly. "Estimated 40 minutes remaining, low confidence" is far more useful than a precise-looking fake number down to the minute. This field keeps rediscovering the same lesson: give a person a fake precise number and they will organize their waiting around it, and once they discover it was wrong, trust in the whole system goes down the drain with it.

Stall Detection: New-Information Volume Is a Better Signal Than Elapsed Time

Timeouts can detect "stuck," but they cannot detect "spinning." An Agent repeatedly calls the same search API with the same parameters, repeatedly opens the same file, repeatedly emits nearly identical reasoning, and every call returns successfully within two seconds -- from a timeout perspective everything looks fine, while in reality it has just wasted twenty minutes.

The effective detection method is to measure new-information volume: normalize and deduplicate the tool output of the last few steps, and count how much genuinely new content appeared. When this metric falls to near zero, the Agent is going in circles. At that point the right move is not simply to kill it off, but to do something more useful: push the recent duplicated content and the current goal back into the context and tell it plainly, "you have already tried these approaches without progress, please take a different path." A good portion of tasks that look stuck are really the model lacking any sense of "what I have already tried."

Let the State Itself Become the User Interface

Observability for long-running tasks has one excellent property: its state is structured and persisted, so it can be rendered directly into a user interface without any extra instrumentation. This is fundamentally different from "running a black-box process," because you can generate a snapshot at any moment: what is done, what is running, what is void, what is waiting on a human, how much budget has been consumed, and what the key conclusions are.

That snapshot is at the same time the recovery mechanism itself. If a user can click on a completed node in the interface and say "this conclusion is wrong," the system knows where to rewind and where to resume, without the user having to hand-write "go back to step 47 and redo it." Observability and recoverability here are not two features; they are two read directions over the same state structure -- this is the structural conclusion most worth remembering when designing a long-running task runtime.

Traces Should Be Organized for Readability, Not for Completeness

The last layer is traceability. If the log of a long-running task is nothing but a raw event stream, no human can get through it; but if you only keep summaries, you lose the detail needed to debug problems. The practical compromise is two layers: the decision layer records "why this action was chosen, which alternatives were dropped, and on what basis," and this layer is written for humans, so it has to be organized in natural language; the execution layer records the raw requests and responses, the complete tool inputs and outputs, and precise timestamps, and this layer exists for machines and for post-hoc debugging, so it can be arbitrarily verbose.

The decision layer is specific to long-running tasks; short tasks do not need it, because in a short task the developer can just read the whole thing, whereas nobody can finish reading the whole process of a four-hour task. The value of this layer is most obvious during post-mortems: when the Agent reached a wrong conclusion, you can distinguish "it did not have sufficient evidence at the time" from "it had sufficient evidence but reasoned incorrectly." The first calls for better information gathering and the second for better reasoning, and these two directions are so different that mixing them together leaves you with no way to start.

Rollback or Resume: An Honest Taxonomy of Failures

The last question, and the hardest one to decide: after a task fails, should you re-run it from the beginning or resume from the point of failure? In practice a lot of systems answer "it depends," while the code usually says "uniformly re-run from the last checkpoint," and for long-running tasks that default is typically neither safe nor economical.

Classify by "What Can Still Be Trusted"

The basis for the classification should not be the type of failure (timeout, exception, assertion failure), but the scope of the invalidation:

• A purely computational failure with no external write: just retry. The only cost is tokens, and you can even raise the temperature to increase diversity.
• An external write happened, but the write itself is idempotent: just retry. Repeating the operation does not change the result, it only wastes a little time.
• An external write happened and cannot be retracted: do not retry. This needs human judgment -- inspect what actually happened, then decide whether to compensate or to accept. This is exactly why the earlier effect_log and intent log exist; without them you cannot even answer "did the write actually happen?"
• A conclusion failure: an intermediate conclusion produced by the Agent is proven wrong (tests failed, a later step discovered the premise does not hold). Retry cannot fix this; retrying only gets it wrong again in the same way. What is needed is to rewind to the point of divergence and fork again, and to explicitly mark in the context that "the previous path has been rejected; the failure reason was X."
• A specification failure: the task itself is defined incorrectly, or the user changed requirements midway. Retrying this class of failure is pure waste of resources; the correct action is to go back to the human.

The fifth category is the one most easily misdiagnosed as the first. Consider a task that ran for two hours and delivered a document in the wrong format: the diagnosis the system typically emits is "output parsing failed, retrying" -- so a task that was never going to improve on a retry burns another two hours. The correct premise for retrying is "the failure cause is random, and retrying can change the distribution of outcomes"; if the failure is deterministic, the expected value of a retry is zero and the expected cost is the entire budget.

Compensating Transactions: Why Rollback Is Not Simple in Agent Scenarios

Databases have mature transaction models we can learn from, but the outside world of an Agent scenario has no transactions. Your Agent opened three PRs, sent two messages, and wrote a configuration file, and those actions are scattered across places you do not control.

The workable alternative is compensation rather than rollback: pre-register a compensating action for every irreversible action (close the PR, retract the message, delete the branch), and execute them in reverse order. This rests on one brutal premise: you have to think at the very beginning about what will need compensating. If a task runs for two hours before deciding to give up, and one of the earlier actions never had a compensation registered, then you are left cleaning up by hand.

The corollary is direct: the density of irreversible actions sets the upper bound on the risk of a long-running task. A four-hour task that is read-only throughout and finally produces a document is trivial to both recover and roll back. A four-hour task that commits twenty changes along the way has, by the fourth hour, effectively already lost full rollback capability and can only move forward. This is exactly why batching actions like "open a PR" and "send a message" at the very end of the task is itself a form of risk control, and not merely a matter of process convenience.

A Pre-flight Checklist for Resumability

If a task is to support resumption, the following items have to be settled at design time, because they cannot be retrofitted afterwards:

1. Every node's key is recomputable from its inputs rather than randomly generated.
2. Every node with an external side effect has a stable, queryable idempotency identifier.
3. An LLM node's output is frozen the moment it succeeds; recovery reads it from disk rather than recomputing it.
4. The RUNNING state carries a lease and a heartbeat, and is reclaimed automatically after process death.
5. Dependency outputs carry digests, so the invalidation scope can be computed when requirements change.
6. The four pause intentions -- park, narrow, rewind, abort -- are modeled separately.
7. Budgets are enforced at the scheduling layer, with a whitelist of wrap-up nodes.
8. The order of irreversible actions is deliberately arranged at the end of the task, with compensating actions registered.

Not one of these eight items is about the model. Every single one is about state and lifecycle, and that in turn demonstrates something: the bottleneck for letting an Agent run a long task has never been how long the model can think, but whether the system is capable of storing "the act of thinking" safely outside of time.

Concurrent Branches and Rate Limits: The Scheduler Is the Real Bottleneck

As soon as the task graph supports parallelism, a frequently underestimated problem surfaces: the throughput of the whole task is often not determined by the model at all, but by the rate-limit window of the external services. Take a task that looks like only twenty-four steps: if eighteen of those steps all call the same code-search API, and that API's quota is ten calls per minute, then no matter how clever your scheduler is, the floor for this task is two minutes.

The judgment this implies is direct: you should identify shared external quotas when you design the task graph, and schedule quota as a first-class resource, rather than letting each branch slam into the wall on its own and then back off and retry individually. Independent backoff is bad because every branch burns its own waiting time in duplicate, and they are quite likely to retry in lockstep, turning smooth traffic into spikes and making the rate limiting worse.

On "when should this be parallel and when should it be serial," there is one pragmatic criterion: look for write conflicts between branches. Purely read-only branches should almost always run in parallel, because there are no dependencies between them and the only constraint is external quota. But as soon as writes are involved, you have to answer one question first: could two branches modify the same thing? If yes, they cannot run in parallel; if no, run them in parallel with confidence -- and record the conclusion of that judgment explicitly in the task graph, rather than leaving it to be detected dynamically at runtime.

Recording it explicitly matters, and it doubles as a precondition for recovery. Suppose two branches were serialized because of resource shortage; after recovery, if the concurrency conditions have changed (say, the rate-limit window has been widened), do you want to keep the original serialization decision, or re-evaluate it? Keeping the original decision is safer and more predictable; re-evaluating is more aggressive but may change the order of outputs and therefore the result. The default should be to keep the original decision, and to annotate the state with "this serialization was decided by quota policy," so that a user looking at progress can understand why it is slow instead of assuming the system has a bug.

There is another class of branch conflict that does not come from external quota but from a shared model context. A great many long-running tasks are designed as "have several expert Agents each offer a view on the same material in turn, then aggregate." Such tasks look naturally parallel, but there is a hidden cost: parallel branches cannot see each other's conclusions, so they can end up producing mutually contradictory analyses, and the final aggregation stage then has to spend a great deal of effort reconciling the conflict. If there is a convergent property between these conclusions -- that is, later analyses are supposed to correct earlier judgments -- then serial execution is actually better. Misclassifying such tasks as "no dependencies, safe to parallelize" is one of the more insidious forms of waste in long-running work.

Retry Backoff Has to Be Deterministic

One last easily overlooked detail: the backoff policy must be deterministic. A lot of implementations use a backoff with random jitter, which is correct for a single-request scenario but destroys recoverability in a long-running task, because "the moment of the retry" itself becomes part of the state, and that state is lost when the process restarts. The result is that after recovery the retry timing no longer lines up with the original rhythm, and the system may issue a request the instant a backoff expires, slamming the rate limiting you had just relieved right back into the wall. The correct approach is to persist "the next time a retry is allowed" as an explicit field and honor it after a restart.

This also explains why the "stateless retry" model does not work for long-running tasks: once you introduce waiting across processes, the intention to wait itself has to be persisted, or else the new process simply has no way of knowing that a backoff window is in effect.

The State Store Itself Becomes a Bottleneck

Once you have folded all the state into a single state-machine design, it is easy to overlook one engineering reality: the cost of storing state eventually exceeds the cost of executing state. In a task with several hundred nodes, every step writes a full snapshot, and as the state grows a single write can become substantial. Layer on top of that a very fine checkpoint granularity, and the write amplification is easily enough to make the storage layer the bottleneck.

The three common approaches each carry a trade-off. A full snapshot is the easiest to implement and needs only one read on recovery, but the write amplification gets severe as the state grows. Event sourcing records only the deltas in state, which makes writes cheap and naturally carries a complete history of operations, but recovery has to replay the entire event stream, so recovery itself gets slow when the state is large. A periodic snapshot plus an incremental log is the compromise between the two: recovery reads only the most recent snapshot plus the increments that follow it, capturing the advantages of both, at the price of higher implementation complexity and the need to think about snapshot version compatibility.

If you take only one piece of advice from this section, make it this: store "the minimum state needed for recovery" separately from "the history needed for auditing". The former should be small and fast to read as a whole, so make it a snapshot; the latter can be large and append-only, so write it as an event stream. Mixing the two is the most common design mistake, because audit information is typically one or two orders of magnitude larger than recovery information, and bundling them together keeps recovery reads slow forever.

Schema Evolution: Letting the State Machine Outlive You

Another characteristic of long-running tasks is longevity. A task suspended for three days and then resumed may find that your code has shipped several versions in the meantime. If the state structure carries no version marker, recovery will hit a combination of fields it does not recognize: in the mild case it errors out, and in the severe case it interprets the old state in the wrong way and silently produces a wrong result.

There are three minimum requirements: the state object carries a schema_version; reading goes through an explicit migration function that lifts an old version up to the current one; and when migration fails, the system refuses to resume and raises an alert instead of continuing with unknown fields. The third point is especially important -- "failing honestly" is far better than "continuing to execute under ambiguity," because problems produced by the latter are almost impossible to locate after the fact.

This also explains one design orientation: the fields in your state should be semantically stable rather than coupled to the implementation. Recording a checkpoint as "completed through node N" is dramatically more stable than recording it as "workspace snapshot #7 has been saved, containing the hashes of the following 12 files" -- the latter is invalidated in its entirety every time the compression strategy changes.

Context Compaction: In a Long Task the Most Expensive Thing Is Not Reasoning, It Is Forgetting

In a short task, context management is not a problem, because the entire session fits from beginning to end. In a long-running task it is the first-order problem. A four-hour task can produce several hundred thousand tokens of tool output, and the overwhelming majority of it is used once and never needed again: the full text of files already read, queries that were tried and failed, intermediate assumptions already overturned by later conclusions. The things that genuinely need to remain visible in the third hour may be only a few dozen lines.

Compaction is the name of the game, but a lot of implementations turn it into a purely mechanical action: once the limit is hit, truncate or summarize by some rule. That is good enough in a two-hour session, but in a task that spans days it manufactures one category of failure that is extremely hard to debug: the task never errors out; it slowly turns into a different task.

The Unit of Compaction Should Be the Work Unit, Not the Token

Compaction triggered by a token threshold has a hidden flaw: it cuts in the middle of a step. A single tool call might return thirty thousand tokens of search results, and compaction happens to land halfway through reading them, so what the model sees is an output with no conclusion in it. Worse, if that call gets compacted away before it is consumed, a replay may return different content, and the entire downstream reasoning ends up resting on evidence that has since changed.

The correct cut is to cut at semantic boundaries: a work unit (which files were read, which conclusion was reached, what was ruled out) is the minimum granularity of compaction. Only once a unit has finished completely and its conclusion has been written into state are you allowed to discard the details of how it got there. This is also why compaction should be triggered by the state of the task graph rather than by the length of the context.

There are only four categories of things that must survive compaction: the core constraints of the task, the key conclusions of the early steps, the stage the run is currently in, and the questions still left open. The intermediate process can be discarded, with one exception worth emphasizing:

• Failed attempts are the easiest thing to drop during compaction and the worst thing to drop. The information "I tried matching by filename and found nothing" is still valuable hours later, and dropping it means the Agent will simply try the same thing again.

The Silent Loss of Constraints Is the Real Cost

The most insidious harm compaction does is the loss of constraints, not the loss of facts. The user said at the outset, "only change test files, do not touch production code." That sentence may be summarized in the first compaction into "the user wants several modules modified," into "the user has confirmed the scope" after the second compaction, and then vanish entirely after the third. From that point on the Agent is doing the right thing at every step, it just happens to be doing something that no longer fully satisfies the requirement -- and nothing in the signals indicates that.

The protection is to promote constraints to persisted objects independent of the conversation: extract them into structured entries at the start of the task (scope, prohibitions, acceptance criteria, interfaces that must be preserved), store them separately, display them separately, and refill them verbatim after every compaction, rather than hoping a summary will keep them alive. This is cheap to implement, and the payoff is that requirement consistency across the whole task goes from "probably still there" to "definitely still there."

Externalized Memory: Let a Task's Memory Live on Disk Instead of Filling the Window

Context compaction solves "it does not fit." Externalized memory solves "it fits but I cannot find it." The two have completely different failure modes: the first makes the task gradually forget, while the second makes the task repeatedly fail to rediscover the clue it found two hours ago among several hundred thousand tokens, search for it again, forget it, and search again.

The memory of a long-running task should not be treated as "a piece of text that has to be remembered." It should be treated as a dataset that can be queried. The test is simple: if your memory only takes effect by virtue of the model "remembering" it, then it will not hold up in a four-hour task.

Three Memory Layers and Their Write Thresholds

In practice memory is split into three layers with different write thresholds and lifecycles, and getting this layering right is the part of the externalization design worth thinking through first:

• Working memory: the context currently being worked on, discarded on compaction. This corresponds to "I am reading this file right now." Its write threshold is zero, because it is the default path.
• Episodic memory: the actions taken and their conclusions -- what was read, what was found, what was tried, and why it was abandoned. This layer must have a write threshold: an action is only written when it produced a conclusion worth reusing, otherwise a four-hour task will write tens of thousands of lines of "I read line 40 of file_00123.cs" with no retrieval value whatsoever, drowning the conclusions that actually matter.
• Semantic memory: knowledge reused across tasks, such as "the tests in this repository require a build first." This layer needs an even higher threshold -- it has to be verified, and confirmed by a human when the task ends. Writing into semantic memory without confirmation is the shortest path to cementing an error in place.

Mixing these three layers together is a common mistake, because their consequences when they fail are entirely different: if working memory fails the task gets dumber, if episodic memory fails the task repeats work, and if semantic memory fails then the next task starts from the wrong place. Pollution of that last layer is irreversible, because the task itself has already ended and nobody is going to check it.

Write Thresholds: Not Every Piece of Information Deserves to Become Memory

When setting the threshold for episodic memory, there is one criterion you can use directly: the reusability test. Assume this action will never be performed again tomorrow; if a new task knew this piece of information at the start, would it make a different choice? If yes, write it. If not -- "read a file and found it empty," say -- do not, because records like that are almost always pure noise.

In the other direction, there is a category of information that must be written unconditionally, however trivial it looks: failed attempts and their reasons. This is the single most valuable and most frequently discarded category in externalized memory. If an Agent spent forty minutes in a directory before confirming there was nothing there, then having that piece of information on record means nobody has to spend forty minutes looking again next time.

Retrieval Is Easier to Overlook Than Writing

A lot of systems design writing carefully and then leave a hole in retrieval: stuff all the memory back into the context and hope the model notices the few relevant entries. This degrades a retrieval problem into an attention problem, and it is guaranteed to fail once the memory volume passes a few thousand entries.

The workable approach is to make retrieval driven by task state rather than initiated by the model: the scheduler knows what the next node is, so it knows roughly what category of memory that node will need, and can retrieve the relevant few entries before dispatch and attach them to the node's input. This reversal of direction matters a great deal -- it is not "the Agent remembers to look something up," it is "the system knows what it needs to know." The latter is far more reliable in a long-running task.

Memory also needs a shelf life. A value read from a configuration file an hour ago may well be stale half an hour later, especially if someone else was editing the repository while the task ran. Tag every memory with the time it was produced and its source (which file, the output of which command), and only then can retrieval decide whether it deserves to be trusted. This point looks trivial, but it is what decides whether the conclusions of a long-running task are "based on the current world" or "based on a snapshot of the world from an hour ago."

Task Decomposition and Replanning: A Plan Is an Estimate, Not a Commitment

Task decomposition is a one-shot decision in a short task. In a long-running task it is an estimate that has to be revised over and over. The difference is this: the decomposition of a short task is usually proven right or wrong within thirty minutes, whereas the decomposition of a long task gets used dozens of times and is continuously refuted by the environment while it is being used.

A system that treats the plan as a commitment will, somewhere in the first hour, run into the predicament of "the plan no longer matches reality, but the system is still straining to complete the plan." This is one of the most typical failure shapes of a long-running task: the Agent is busy completing a plan that no longer holds, and all of its effort is spent on the wrong goal.

Decomposition Granularity: Cut by Verifiability, Not by Importance

The common decomposition heuristic is to cut by importance -- do the most critical part first. That is the correct strategy for short tasks, because a short task almost always gets finished. But for a long-running task, cutting by importance commits one specific mistake: the most important work is often the most uncertain and the most likely to fail, so it gets put first, it gets stress-tested the hardest, and the deterministic work behind it never gets done at all -- leaving you with zero delivered.

The decomposition that suits long-running tasks much better is to cut by verifiability: give every node an explicit signal, obtainable by some means, that says whether it was accomplished. Reading code, running tests, calling an API and getting a structured response -- all of these have clear signals. Nodes like "analyze the architecture" or "assess whether this is reasonable," which have no acceptance criterion, should go last, or be broken down to a granularity that can be verified.

The reason is resource allocation: a long-running task will certainly be interrupted (budget exhausted, user pauses, environment error), so "what is left behind when it is interrupted" is the central design question. A decomposition cut by verifiability guarantees that every prefix is a valuable deliverable -- if 12 nodes are done, the conclusions of those 12 nodes stand on their own, without depending on the remaining 12.

Replanning: The Trigger Condition Matters More Than the Algorithm

There are two things about replanning that are easy to get backwards. The first is treating it as a high-frequency action: replan every few steps. The result is a plan that oscillates endlessly, where the conclusions of the first few steps get overturned before they are ever verified, and the task turns into thrashing in place.

The second is making it fire only after a failure. But the moments when a long-running task most needs replanning often have no failure at all -- you simply discover that some assumption in the original plan no longer holds. Take the task "fix the circular dependencies in 20 modules." When you get to number 15 and discover that the remaining 5 all stem from the same root cause, then fixing them one by one from there on is pure waste.

There are three practical trigger conditions, and they are worth defining explicitly rather than leaving them to the model's judgment in the moment:

1. Conflicting evidence: a new observation contradicts an existing conclusion. Note that this requires "existing conclusions" to be recorded explicitly in task state, otherwise the conflict cannot be detected at all.
2. Pattern recognition: several consecutive nodes produce isomorphic results (the same class of error, or different manifestations of one root cause). This usually means the decomposition is wrong, not the execution.
3. Declining marginal value: the new value added by the last N nodes falls below some threshold, which means the task has entered the region of diminishing returns and "what counts as done" has to be redefined.

• The easiest mistake to make when replanning is to treat "thinking over again what to do" as replanning. In reality replanning has to preserve at least three things: conclusions that are complete and still valid, hypotheses that have been disproven (and that must be explicitly marked in the context as rejected), and the core constraints that still hold. Drop the first two and you have started from zero, throwing away the previous several hours.

Plan Versioning: Making Replanning Itself Traceable

If replanning simply overwrites the old plan, you will not be able to answer a question that a long-running task will eventually be asked: "why was the approach in the second hour different from the approach in the fifth hour?" That question turns out to be extremely valuable during a post-mortem, because it separates "the system arbitrarily changed approach" from "the system reasonably changed approach in light of new evidence."

The implementation is light: every replan produces a new plan version that records what changed relative to the previous version (nodes added, nodes removed, ordering adjusted, goal narrowed), rather than storing only the new plan. The cost is almost zero, and the payoff turns the whole task into an auditable chain of decisions. This is the same idea as the earlier separation of the decision layer from the execution layer, applied at the planning layer.

One last easily overlooked detail: subtasks that have already been dispatched are not automatically canceled when the plan changes. So replanning has to deal explicitly with in-flight nodes -- do you wait for them to finish, cancel them, or let them finish and discard their results? Without that handling there will be a gap between the updated plan version and actual execution, and the artifacts inside that gap are usually wrong.

Two Shapes of Going Off the Rails: Goal Drift and Self-Loops

There are only a few ways an Agent goes wrong on a long task, and they boil down to two, whose detection signals and repair methods are completely different. Conflating them is a common source of misdiagnosis: a lot of teams see "the task went sideways" and attribute it to the model not being good enough, when in reality half the cases are loops and half are drift.

Goal Drift: Every Step Is Reasonable, Together They Are Not What the User Wanted

Drift is characterized by the fact that every individual step looks correct. It does not start by going wrong at some particular step; the decision path itself gradually diverges from the original goal. There are three common mechanisms:

• Frame-by-frame erosion of constraints: constraints get summarized away when the context is compacted, as discussed earlier. Every compaction loses a little precision, and after accumulating over dozens of them the task is no longer inside the original requirement -- yet not one step looks wrong.
• Proximity anchoring: the model tends to anchor new decisions on the most recently observed object rather than on the original goal. The user says "find the performance problems"; the first few steps look at one endpoint, spot an obviously slow query, and then all subsequent work revolves around that single query -- when what the user actually cares about is the overall performance profile.
• Subgoal creep: small goals produced along the way ("split up this function") gradually acquire equal standing with the main goal, and can even displace it.

The key to detecting drift is do not detect behavior, detect the goal. Behavior always looks reasonable -- it is reading, searching, editing files. The effective method is periodic re-evaluation: put the current plan, the list of completed nodes, and the original requirement side by side, and let the model (or a human) answer one question: "if we run this plan to completion, will it satisfy the original requirement?"

The interval for this check should be deliberately sparse, say once every dozen-odd completed nodes. There are two reasons. First, the check itself consumes a model call, so running it densely is not cost-effective. Second, drift is cumulative: dense checks can only detect drift that just happened, while a sparse check has a chance of catching the situation where "we have already gone a long way in the wrong direction," which is only visible by contrasting against the original requirement.

• A very practical test: if a task completed many nodes in a row and not one of them can be described as "this directly corresponds to some item in the user's requirement," you can basically confirm drift. Efficiency is not alignment.

Self-Loops: Consuming the Same Failure Over and Over

Loops are easier to spot than drift because they have obvious repetition: the same tool, the same parameters, the same error message, again and again. They usually do not happen at random; there is a definite cause.

The most deceptive variety is a loop disguised as progress through variation: a different search keyword each time, a different file path, the same question rephrased. On the surface every step looks different, but substantively nothing was learned. Detecting this by comparing call parameters alone is not enough; you have to compare semantics.

The effective criterion is the new-information volume mentioned earlier: normalize and deduplicate the tool output, then count the new content. If the new-information volume is near zero across several consecutive steps, then no matter how the parameters vary, the Agent is consuming the same failure.

The repair methods deserve emphasis, because most people's first instinct is to "raise the maximum step count," which is almost always wrong. Raising the step count only causes the same failure to be consumed more times. The genuinely effective repairs, in priority order, are:

1. Tell it what has already been tried: push the duplicated content and the current goal back into the context and state explicitly that these approaches have been attempted without progress. This resolves a substantial share of loops caused by "the model does not know what it has already tried."
2. Switch tools or switch information sources: not just a different query term, but a different category of source -- from reading code to running tests, from searching to inspecting runtime data.
3. Escalate to a human node: if the same failure has already consumed a significant budget without progress, the correct action is to stop and ask a person, rather than keep spending.
4. Abandon the current path and fork: mark this path as rejected, keep the reason for rejection in the context, and take a different route.

What these four approaches share is that they all change what the next step will be, rather than merely increasing the number of steps available. This distinction is the most valuable single judgment in handling loops.

The Shared Precondition for Both: Failed Paths Must Be Remembered

Drift and loops are especially stubborn in long-running tasks for one reason that has come up repeatedly above: rejected paths leave no trace. What gets dropped during compaction is the search process, and what is usually preserved is the successful conclusion, so two hours later the model has no idea that a road was already walked and found to be a dead end.

So no matter how much loop detection and drift checking you have built, if "the rejected paths and their reasons" are not persisted as first-class state, all of those detections will eventually fail -- they are only postponing the failure, not preventing it. This conclusion holds for any Agent that has run for several hours and requires no further argument.

The Real Semantics of Resumption: What You Think Is Recovery May Be Replay

The previous sections covered checkpoint granularity, idempotency keys, and invalidation propagation. This section deals with a more fundamental question: what does the recovery action itself actually do. A lot of systems implement "restore from a checkpoint" as "re-execute the remaining steps," and the difference between those two sentences is enormous in a long-running task.

Re-executing the remaining steps carries an implicit assumption: completed steps can be safely skipped. That assumption holds only if "the world has not changed." At the scale of hours, the world changing is the normal case.

Verify First on Recovery, Do Not Just Continue

In a correct recovery flow, the first step is not to execute any node but to verify whether the outputs of the completed nodes still hold. This verification has three layers of increasing cost, and you can decide how deep to go on demand:

• Existence check: is the output still there (the file not deleted, the remote object still present)? The cheapest, but it basically only rules out the crudest problems.
• Fingerprint check: does the content digest of the output match what was recorded, and what about the inputs it depended on? This layer catches "I drew a conclusion based on some file, and that file has changed."
• Semantic check: do the premises in the outside world still hold? The conclusion "CI is green," for example, requires querying CI again, because a green light from a few hours ago may now be red.

The result of the verification has to drive invalidation propagation rather than merely triggering one more re-run. This is the real purpose the input_digest mechanism serves in the recovery scenario: it turns "should this be redone" from a matter of judgment into a computable question.

The Three Recovery Modes and What Each Costs

Engineering-wise you have to distinguish three recovery modes, whose implementation complexity and correctness requirements differ enormously:

Recovery mode	What it does	When it applies	Primary risk
Trust recovery	Skips all completed nodes outright	The task is purely read-only and very short	Every output is stale once the world changes, and there is no trace of it
Fingerprint recovery	Verifies digests, re-runs whatever downstream has been invalidated	The default choice; suits most long-running tasks	Digests operate at the syntactic level and cannot see semantic invalidation
Semantic recovery	Re-validates external premises, re-forks where necessary	Dependencies on CI, deployment state, or a collaborator's output	Expensive, and it requires an answer to "who does the verifying"

Most teams jump straight from the first tier to the third, or go the other way and stay parked on the first tier, bolting on verification only when something goes wrong. The latter is more dangerous, because the verification logic never gets exercised on the normal path.

"Exactly-Once" Is an Illusion; At-Least-Once Plus Idempotency Is Reality

The execution semantics of a long-running task can only ever be at-least-once: a node may be executed several times, and you cannot determine how many times it actually was. The lessons of distributed systems apply here without reservation -- do not chase exactly-once execution, instead make repeated execution harmless.

Concretely, in implementation terms, this means three things, none of which can be omitted:

1. Every node has a stable, recomputable key, so a replay does not produce a new node.
2. Every node with an external side effect has an idempotency identifier, and that identifier is written down before the side effect occurs.
3. On recovery, claim the existing side-effect records first, and only then decide whether to execute.

The one most easily missed among these three is the timing in the second item. "Write the idempotency record after execution succeeds" looks like the natural ordering, but it leaves a window: the execution succeeded, the record was not written, the process died, and at recovery there is nothing to go on, so it executes again. Whenever someone asks whether that window can be closed with logging, the answer is always no, because the missing thing is precisely that log line.

Recovery Storms: One Restart Triggering Many Replays

One last easily overlooked risk: the recovery action itself can amplify. If a process, on startup, reclaims all RUNNING nodes into PENDING while the scheduler simultaneously spins up multiple executors, then those executors will claim the same batch of nodes at the same time. In a system with no lease mechanism this leads to the same node being executed many times, and each execution may produce an external side effect.

The protection is the standard lease: a node carries a lease with an expiry when it is claimed, the executor renews the lease periodically, and it checks that the lease is still valid before executing. On recovery you reclaim only the nodes whose lease has expired, not every RUNNING node.

This detail looks like infrastructure trivia, but it decides whether resumption is safe at all in a multi-executor environment. And a multi-executor environment is virtually inevitable the moment you move from a single machine to production, because the elasticity it brings, the necessity of long-running tasks, and cost control are all directly related.

Cost Control: Turning the Bill into a Scheduling Signal

The three-tier budget gates described earlier are defense. This section is about offense -- how to get more value out of every dollar. The essence of defensive budget control is "stop when you are about to exceed the limit," while what a long task actually needs is "finish the most important things within the budget."

Compute a Cost Density for Every Node

The definition of cost density is simple: the budget this node consumes, divided by the task progress it advances. Progress can be approximated by the number of completed nodes, the number of work units covered, or the estimated value of the nodes remaining in the task graph.

Once you have this metric, the scheduler can do something that would be pointless in a short task: make trade-offs proactively while budget still remains. Concretely, when you find that a branch's cost density stays persistently high, stopping it early and reallocating its budget to other branches is usually a better deal than letting it run to completion.

def should_continue(node: Node, used_token: float,
                    progress_delta: float, density_limit: float) -> str:
    """成本密度闸门：在节点执行中途中决定是否继续。

    progress_delta 是这个节点在一段时间内推进的进度估计。
    密度持续偏高意味着这个节点正在吞噬预算却没在交付。
    """
    if progress_delta <= 0:
        #零推进：不一定是失败（可能正在跑一个长耗时工具），
        #但绝不能在这种状态下继续派发新节点
        return "hold"

    density = node.cost_used / progress_delta
    if density > density_limit * 3:
        #严重偏高：立即中止，保留已获得的产出
        return "abort"
    if density > density_limit:
        #偏高：允许完成当前节点，但不再派发它的后继
        return "finish_no_successor"
    return "continue"

The key to this logic is separating "abort" from "do not dispatch successors." A lot of implementations, once the cost limit is exceeded, kill the entire branch outright, which marks even the work that already completed as failed -- and that work is actually valuable. Preserving the output and dropping what comes next is the handling that better fits the logic of cost.

Cost Is Not Only Tokens

There is a category of cost in long-running tasks that is frequently ignored entirely: waiting cost. A task that runs for four hours prevents the user from doing anything else for those four hours, even when the task itself only needs ten minutes of actual computation. This cost never appears on the bill, but it is the dominant user cost of a long-running task.

That observation carries a direct engineering implication: moving uncertainty out of the waiting time is a more effective cost optimization than compressing tokens. The concrete techniques include running in parallel everything that can be parallelized up front (read-only branches), collecting all the information that can be gathered in one pass at the start rather than asking repeatedly mid-run, and scheduling the nodes that need human confirmation as early as possible rather than late (fail fast and expose problems early instead of running for three hours before asking).

That last one is frequently done backwards. A lot of systems follow the flow of "run everything that can be run, then go to the user for confirmation at the end," on the grounds that "asking too many times will bother the user." But this is dangerous in a long-running task: if the parameters that need user confirmation are not asked about until the third hour and the user gives a different answer, most of the first two hours of work is wasted. Moving the confirmation point earlier costs a few more interruptions but avoids wasting an entire stretch of work, and for a long-running task that trade is usually worth it.

Caching and Reuse: Where the Big Money Goes in a Long Task

Long-running tasks have a cost structure that short tasks do not: large amounts of repeated context. Task constraints, the basic structure of the codebase, and intermediate conclusions that have already been verified all get resent over and over across dozens of calls. Under token-based billing that repetition is a real, measurable cost.

The mitigation options sit at clearly distinct levels. Prompt caching addresses "the same prefix is being billed over and over," and the prerequisite is that you put the stable part first, the volatile part last, and do not insert changing content in the middle. Reducing duplicated content addresses "I was sending too much to begin with," and the approach is to replace long inline payloads with a reference that gets fetched when needed. Task-level reuse addresses "the same node did the same thing across multiple runs," and the approach is to lift the deterministic part out of the model call and turn it into code.

The third one yields the most and is the most frequently overlooked. A substantial portion of the work in a long-running task is genuinely deterministic -- rule-based matching, format conversion, metric computation. Taking that portion away from the model both saves money and eliminates the failure modes of that class of step. The criterion is simple: if you can write a function that does it, then it should not be occupying a model call.

Designing Human Checkpoints: When to Stop, How Long to Stop, and What to Ask

Earlier we split pause intentions into park, narrow, rewind, and abort, and that was the semantic layer. This section addresses a more practical question: where should human checkpoints go, and what should each one ask of the human.

A common design mistake is treating human checkpoints as "the fallback for error handling" -- escalating to a human only when the task fails. That design is barely workable for short tasks, and in a long-running task it is exactly backwards: failures in a long-running task are extremely expensive, and discovering a problem only after it fails usually means several hours have already been wasted. Human intervention has to be a deliberately designed first-class mechanism, not an exception path.

Four Kinds of Checkpoint and Their Different Values

Ranked by "how much a human intervention buys you," human checkpoints come in four kinds, with increasing implementation difficulty:

• Confirmation type: ask once before executing an irreversible action. The simplest, with clear value, at the cost of an interruption. Well suited to large amounts of money and broadly impactful actions.
• Direction type: confirm the overall direction before committing a lot of cost. For example, "I plan to start from these five modules; which two do you think I should fix first?" This one has the highest cost-performance ratio, because it trades one interruption for several hours of uncertainty.
• Exception type: the system has hit a situation it has no authority to decide on its own. The difficulty lies in defining "no authority": too permissive and the human gets buried, too strict and the system guesses stubbornly on ambiguous cases.
• Acceptance type: confirm before delivery after everything is done. It looks like the most necessary one, but in practice it has the lowest value, because if at that point you discover the direction was wrong, several hours of investment can no longer be recovered.

The practical implication of this ranking is direct: move human checkpoints as early as you possibly can. For a task that runs four hours, if you only do acceptance review at the end, the amount of work this human intervention can save is zero. What actually pays is asking once at minute ten.

Ask Everything at Once Rather Than Five Times

The cost of human intervention is not only the interruption itself. There is also the cognitive reconstruction cost for the user -- they have to recall the task background, re-read the existing output, and rebuild their judgment from scratch. A user who comes back after two hours away spends the first several minutes just figuring out "what the situation actually is now," and that process alone eats ten-plus minutes.

So good checkpoint design carries one explicit requirement: ask every question that needs asking within a single interaction. That means that when the system triggers a checkpoint, it should first gather all the pending questions and present them at once, rather than asking one, waiting for an answer, and then asking the next.

What gets presented matters too. A qualifying checkpoint should contain four things, and missing any one of them leaves the user unable to decide quickly: what has been completed so far (a concise list, not a running commentary), where things are stuck right now (narrowed down to a specific decision point), what the options are and what each costs, and your recommendation with its reasoning. The fourth is the one most often left out, and it has the largest impact -- without a recommendation, the user is effectively doing the system's planning work.

Implementing Parked Waits Correctly

Earlier it was said that the parked state must disable timeout monitoring and stall detection. Here is one implementation-level point to add: a park needs an expiry policy.

A parked task that waits forever will see the environment change, the human forget, and dependencies get upgraded. The reasonable approach is to give the parked node an explicit condition for re-evaluation rather than a pure time condition -- for example, "wake up when there is a new commit on repository X," "wake up when CI finishes," "if nobody handles it within 48 hours, mark it as needing re-confirmation."

• The most easily overlooked metric of human checkpoints is "the number of interruptions." It correlates negatively with system quality: a task that requires bothering the user five times in order to finish usually means it has not narrowed uncertainty down far enough.

There is one simple test for whether a checkpoint is well designed: take the human intervention away -- does the task get clearly worse? If the answer is "no, just a bit more rework," then the checkpoint is unnecessary; it only makes you feel more in control. If the answer is "yes, and the cost is high," then it is necessary. Using this as the admission bar for every new checkpoint is more reliable than any heuristic.

Where Things Stand in 2026: What Has Converged and What Is Still a Bet

This section does not make predictions. Instead, it layers the mechanisms from the previous sections by "how reliable the current implementations are." The point of layering them is that the things on top can be used as they are, while the things at the bottom need an escape route left in the design. The concrete implementation details vary a great deal between teams, but the stability of the state can be judged from the structure of the code itself.

The Parts That Have Basically Converged

Separating task state from the execution process is almost no longer a design choice that needs arguing for. Putting state into external storage and making the execution process a replaceable, short-lived component has become the de facto standard practice for long-running task systems. The supporting pieces -- checkpoints, leases, idempotency keys -- have largely settled into place too, and the differences lie mainly in how they are wrapped rather than in their semantics.

Context compaction as a mandatory mechanism has spread just as widely. The form that has genuinely been adopted is "compact along semantic boundaries, keep failed attempts separately," rather than simple token-count truncation. The constraint externalization described earlier is not yet standard, but it is an improvement with a very low cost and a clear payoff, and typically requires nothing more than adding a standalone constraint store to an existing architecture.

Enforcing budgets at the scheduling layer is also consensus. Writing the budget into the prompt instead of enforcing it at the scheduling layer is now hard to find in any serious long-running task. What drives that shift is the maturity of cost governance: when a single task can cost tens or hundreds of dollars, "discovering the overspend after the fact" is not acceptable.

The Parts Still Converging but Still in Disagreement

The layered structure of memory has a clear direction (working, episodic, and semantic), but there is still no consensus on the precise boundaries and write thresholds of each layer. The most contested issue is admission to semantic memory: whether a given judgment should be distilled into cross-task knowledge or kept only among this task's conclusions. This bears directly on the risk of cementing errors in place, and strategies vary widely between teams.

The granularity of task decomposition and the timing of replanning likewise have no unified answer. Approaches that favor early decomposition and frequent replanning react quickly to environmental change, but pay a high price in plan oscillation; approaches that favor coarse decomposition and little replanning are stable, but lose a lot when they go off the rails. Real systems tend to choose by task type rather than merely by configuration.

The placement of human checkpoints varies far more at the product level than at the technical level. The technical implementation (pause, resume, invalidation propagation) is mature, but the product question of "how often do we interrupt the user" has no technical answer and can only rest on assumptions about the user. This is worth nailing down during technical selection, because it works in reverse to determine how flexible the architecture has to be.

The Parts Still Being Bet On

Quality evaluation for long tasks is currently the weakest link. Deciding whether a task was "done well" is a matter of just looking at the output in a short task; in a long task the output is usually dozens of scattered conclusions plus a report, and it is very hard to measure with a single metric. The result is that improvement work on these systems lacks reliable feedback and has to rely on manual post-mortems.

Credibility management for cross-task knowledge has no good answer either. Can a conclusion verified in one task be used directly in the next one? How much verification counts as enough? Earlier we said "writing to semantic memory requires human confirmation," but when the number of tasks is large it is impossible to confirm every single entry, so it turns into sampling -- and how to set the sampling criteria is currently left to each team's own judgment.

The cost-quality trade-off curve has an unclear shape for long-running tasks. On a short task it is a clean Pareto frontier: spending more budget buys higher quality, right up to saturation. On a long task the curve is more complicated -- extra budget may buy more coverage, or it may buy a great deal of duplicated labor.

How to Judge Whether an Approach Is Reliable

If you want one evaluation standard for yourself, ask these four questions. All of them can only be answered by inspecting the structure of the code, never by reading documentation:

1. When the process dies, is the task state fully readable? If recovery requires rebuilding state from logs, then it is not yet a genuinely persisted task.
2. When the same node executes twice, does one extra thing appear in the external system? If so, there is no idempotency anywhere on the side-effect path.
3. When the user changes requirements, can the system compute which work has been invalidated? If not, a requirement change can only be handled by starting over.
4. When the task goes off the rails, how long until it is noticed? If the answer is "after it finishes," then the value of everything in between is close to zero.

If any one of these four questions cannot be answered, that system is unreliable on the scale of hours, no matter how strong its model capability is or how good its demo looks. This is also the practical meaning of the sentence at the very beginning of this article: the bottleneck of a long-running task is not the model, it is state and lifecycle management.

Closing Thoughts

Back to the original question: how does an Agent that runs for hours survive a process restart. The answer condenses into a single sentence -- move everything mutable and lossy out of process memory and onto a persisted task graph, then turn the process into an executor that is free to die at any moment and can be replaced after death.

This shift sounds like an engineering detail, but it changes the nature of the problem. An in-process Agent is "an entity that is currently thinking," and all you can do is wait for it to finish or kill it. A persisted Agent is "a state machine with half of its progress stored externally," and you can pause it at any time, replace the executor, change its goal, and then let it continue from where it was interrupted. Only the latter genuinely has what it takes to handle long-running tasks -- not because it is smarter, but because it no longer has to stay alive continuously.

A last warning about a place where people easily go astray: the most important parts of this architecture -- idempotency keys, input fingerprints, invalidation propagation, budget gates -- are none of them features that make a demo look more impressive. All they do is make the system behave with more dignity when something goes wrong. But in the world of long-running tasks, dignity is precisely the whole point: a task that runs for four hours will never be praised by users for finishing; they will be abandoned entirely the moment it crashes halfway, produces a half-finished artifact, or quietly spent two hours doing the same useless work twice.