When you push a DSH plugin from "it runs locally" to "others dare to use it," the real watershed appears: the distribution method determines where the build artifacts come from, defensive programming determines whether edge cases will bring down the entire Agent, and incident postmortems determine whether the same pit will be stepped into a second time. These three things appear to belong to three separate layers—deployment, coding, and culture—but within DSH's engineering system they form a single chain: choosing the wrong release method will cause users to hit pitfalls, boundary bugs will erupt after launch, and teams that don't write postmortems will repeatedly rediscover the same defect in the same way. This article is the first half of "Publishing DSH Plugins + Defensive Programming + Incident Postmortems: Turning Plugins into Trustworthy Assets." It first thoroughly explains the build artifacts and authorization differences of the three distribution paths, then dissects field by field the build chain of the prepare script and the defensive approach of orthogonal result reporting, and finally moves into the four postmortem questions and the threshold judgment of "what kind of bug is worth writing a postmortem for." After reading this half, you should be able to answer one question: why can a plugin with 178 green unit tests and 100% line coverage still crash in the very first second after the editor connects?

npm / tarball / Git: Build Artifacts and Authorization Differences of Three Distribution Paths

Once local installation works, the next step is to distribute the plugin to others. There is an easily overlooked premise here: publishing to a public registry is not mandatory. The official DSH documentation provides three distribution routes—npm publishing, tarball delivery, and Git installation. Their user installation commands appear to differ only in parameters, but what is delivered underneath is completely different, and this difference directly determines whether what users receive is loadable code or source code that fails to load.

First, look at the three installation commands. The first is dsh plugin add your-package. When installing, the user pulls from the npm registry a package that has already been published, and the package contains a built lib/ directory, that is, prebuilt artifacts. The second is dsh plugin add ./hello-plugin-0.1.0.tgz. What the user receives is the compressed package you produced with pnpm pack, and its contents are likewise the result already built at the moment you packaged it. The third is dsh plugin --profile demo add github:you/hello-plugin. Note that the source here is a GitHub repository, and what pnpm clones down is source code, not build artifacts.

These three paths have mutually different requirements for build artifacts, and the core question for judging them is: is anyone actually running your build script on the user's side. For the npm and tarball paths, users do not need any build authorization during installation, because the artifacts were already fixed at the moment you published them; Git installation is the most flexible—users can directly point to a branch, a commit, or a fork—but the cost is that on their side the source code must actually be compiled, which runs straight into the hurdle of "build scripts."

示意图
Comparison chart of three publishing routes: npm publishing, tarball delivery, and Git installation respectively deliver prebuilt lib/, a pnpm pack package, and source code, along with their different requirements for build authorization.

Organizing the above differences into a table will make it more intuitive. Please note the last column in the table—"whether build authorization is required"—it is the key variable that distinguishes Git from the other two paths:

MethodUser install commandWhat gets installedRequires build authorization
npm publishdsh plugin add your-packagePrebuilt lib/ codeNo
tarball deliverydsh plugin add ./hello-plugin-0.1.0.tgzThe package produced by pnpm packNo
Git installdsh plugin --profile demo add github:you/hello-pluginSource code (not build output)Yes (pnpm ≥ 10)

Why does the Git path additionally require build authorization? The root cause is that pnpm tightened the execution of dependency build scripts starting with version 10. When you install a package from Git, pnpm needs to run that package's build script during installation, and such scripts are in a state that requires explicit authorization by default. In other words, npm and tarball don't "need authorization" because they are more advanced, but because they simply don't need to run a build on the user's machine—the artifacts are already ready-made. Git install shifts the act of "building" to the user's side, and the authorization issue comes along with it.

What does this mean for plugin authors? If you want zero-friction distribution, npm and tarball are the more worry-free choices: users install with a single command and can use it right away, without needing to understand the build process or encountering an authorization prompt. If your plugin targets developers who are willing to tinker with source code and need to track a particular branch, the flexibility of Git install is worth that cost, but you must write the prepare script well—because without it, a TypeScript package arrives with no lib/ output, and loading will fail outright.

Why Git install must include prepare: the build chain from source to publish entry

Git install pulls source code. The consequences of this statement are more serious than it sounds: no step in the installation process will run the build script for you. Many people think, "I clearly wrote a build command in package.json, so why didn't it take effect during installation?" That's exactly where the problem lies—build is a script you need to invoke manually, not part of the installation lifecycle.

But after a Git install, pnpm does run one specific script: prepare. This is precisely the hook that plugin authors must take advantage of. The first thing an author must do on their side is provide a prepare script so that pnpm builds the publish entry from source after the Git install completes. When a user runs dsh plugin --profile demo add github:you/hello-plugin, pnpm clones the repository, installs the dependencies, and then triggers prepare; only at that moment is your code actually compiled into a loadable form.

This leads to the most important design constraint for the prepare script: it must be self-contained. Self-contained means it cannot assume it is running in a "context that only exists in the development environment." The most typical counterexample is a monorepo checkout—in your development repository, the plugin package may sit next to shared TypeScript configs, root-level workspace dependencies, and type definitions referenced by other packages. These things run without any problem locally because the monorepo has everything laid out. But when a user installs your plugin from Git, what they pull down may be just this package itself, or a repository snapshot without a complete monorepo structure. Any assumption that "there are other packages next to it" or "there is a certain config at the root" will turn into a build failure on the user's side.

There is another layer to keep in mind with prepare: it runs at install time, not at publish time. This means the environment it faces is uncertain—the user's Node version, package manager version, and platform may all differ from your development machine. So the less prepare does, the better. It should do only one thing: "transpile src into the publish entry point." Don't stuff development-time actions like type checking, linting, testing, or documentation generation into it. Keeping prepare minimal means minimizing the pitfalls users might hit during installation.

To summarize the responsibilities of the author and the user along this Git installation chain:

  • Author side: Provide a self-contained prepare script that runs automatically after pnpm completes the Git install, building the source into the publish entry point.
  • Author side: prepare must not depend on context that exists only in the development environment, such as a monorepo checkout sitting alongside it.
  • User side: Under pnpm ≥ 10, build scripts must be authorized. This is not a step that can be bypassed.
  • Shared premise: The Git path delivers source code, not build artifacts, so "being able to clone it" does not equal "being able to load it."

prepare Script in Practice: Transpiling src/ Directly with a Dedicated tsdown Config

turtle-ui is a working example you can reference directly. Its prepare runs a dedicated tsdown config that transpiles src/ directly, without project references and without type checking. None of these three design choices is arbitrary—each one dissolves the "self-contained" risk mentioned earlier.

First, the "dedicated config." turtle-ui prepares a separate tsdown config specifically for the publish path, rather than reusing the one used during development. The development config often carries project references, incremental caches, type declaration generation, and other things that speed up local development—and these are precisely what depend most on the monorepo context. The dedicated config strips out all external dependencies, keeping only the single chain of "transpiling the TypeScript in src/ out."

Next, "transpiling src/ directly" and "not using project references." TypeScript project references require the referenced projects to actually exist on disk and to have already been built—fine in a monorepo, but not necessarily true in the user's installation directory. Transpiling src/ directly bypasses the entire reference graph, requires no pre-building of any dependency packages, and does not depend on the references field in tsconfig.

Finally, "no type checking." Type checking is a development-time quality gate; it should be done in CI and before local commits, not placed on the path where users install a plugin. Leave type checking to the development workflow, and let prepare be responsible only for producing runnable JavaScript. That way, even if the user's TypeScript version differs from the author's, a single type error won't cause the entire installation to fail.

Below is a directly pasteable package.json scripts snippet from dsh-hello-plugin, which binds prepare to a dedicated config:

{
  "name": "dsh-hello-plugin",
  "scripts": {
    "prepare": "tsdown -c tsdown.publish.ts"
  }
}

This configuration contains only one line of script, but it is packed with meaning. tsdown -c tsdown.publish.ts explicitly specifies the configuration file, preventing tsdown from automatically picking up the default development configuration; the publish in the configuration file name is also a clear signal, reminding future maintainers that this configuration serves the publishing path rather than local development. When you modify the development configuration, you will not accidentally change the build behavior when users install.

If you distill this approach into a checklist, you can self-check as follows:

  1. Does package.json contain a prepare script, and does it point to a dedicated build configuration?
  2. Does this configuration directly transpile src/, rather than relying on an already-built referenced project?
  3. Are project references disabled, avoiding the requirement that other packages exist on disk?
  4. Is type checking excluded, leaving quality gates in the development workflow?
  5. If you clone the repository alone into an empty directory and run installation once more, can prepare succeed independently?

The last self-test is the most effective: clone the plugin repository alone into an empty directory without a monorepo structure, and run a Git installation once. If prepare hides a dependency on "that package next to it," this step will immediately expose it, rather than waiting for users to discover it for you.

Report orthogonal results independently: why timedOut, signal, and exitCode cannot be nested

Now that the plugin can be distributed, the next thing to solve is that class of boundary bugs that "cannot be tested in normal times but cause trouble as soon as they go live." The official documentation summarizes them as "hard-won defect category rules"—each one comes from a real release or a defect that almost made it to release. The first and most important defensive pattern is reporting orthogonal results independently.

What does orthogonal mean? A result can simultaneously have multiple properties, and these properties do not have a subordinate relationship with one another. The documentation gives an excellent example: a process may have already timed out, yet still end with exit code 0, because it caught the termination signal. If the code treats only "non-zero exit code" as the sole marker of failure, or nests the reporting of timedOut inside the exitCode branch, this process will be misjudged as a normal success, even though it was in fact already terminated by the timeout mechanism.

示意图
Comparison chart of bad and good examples in defensive programming: the incorrect and correct ways of writing five patterns—result reporting, dispose settling, credential erasure, link deletion, and callback isolation—are shown side by side.

This is the problem that "reporting orthogonal results independently" is meant to solve. The documentation clearly states: each independent fact (timedOut, signal, exitCode) should be reported separately, and you must never nest the reporting of one flag inside the branch of another flag. Because once nested, the caller loses a complete view of the facts—it can only see what the outer branch allows it to see, and a run that terminated early will fall exactly into the gaps of the nested structure.

Why are these three fields orthogonal? Analyze them one by one:

  • timedOut: maintained by yourself in the timeout timer, indicating "you actively initiated termination"; it is unrelated to how the process ultimately ends.
  • signal: which signal terminated the process. After a timeout, you send SIGTERM; if the process does not handle this signal, it will end with this signal.
  • exitCode: the process exit code. The key point is that a process that catches SIGTERM and chooses to exit gracefully can return 0—this is completely legitimate semantically, but it absolutely does not mean this run was "successful."

The state space that these three can combine into is exactly the part that nested representations lose. Consider the following combinations:

timedOutsignalexitCodeActual semantics
falsenull0Normal success
falsenullnon-zeroNormal failure
trueSIGTERM0Timed out, but the process caught the signal and exited gracefully (most easily misjudged)
trueSIGTERMnullTimed out, and the process was killed by the signal

The third row is that classic trap. If the caller's logic is written as "treat it as success if exitCode === 0", this row will be classified as success; the correct approach is to check timedOut and signal at the same time. Only by returning the three independent facts flat does the caller have a chance to make the correct judgment; if you nest them, you have made an incorrect simplification on the caller's behalf. This is also why the documentation emphasizes that these rules should be read "before writing lifecycle, concurrency, subprocess, or cleanup code"—it is not a preference about coding style, but a way to prevent a simple edge case from taking down the entire Agent.

Defensive style in run.ts: a checklist for maintaining fields from spawn to close

To put the above principles into concrete code, refer to the implementation in packages/my-shell/src/run.ts. What it does is run a subprocess and orthogonally report three independent facts—timedOut, signal, and exitCode. The whole piece of code is not long, but the timing for maintaining each field matters, and it is worth breaking down item by item.

First look at the interface definition. In RunResult, the three facts are each declared as independent fields, and the comments also clearly mark their status as "independent fact 1/2/3":

// File path: packages/my-shell/src/run.ts
// Run a subprocess and orthogonally report three independent facts: timedOut, signal, exitCode.
import { spawn, type ChildProcess } from 'node:child_process'

export interface RunResult {
  timedOut: boolean            // Independent fact 1: whether it timed out
  signal: NodeJS.Signals | null // Independent fact 2: whether it was terminated by a signal
  exitCode: number | null       // Independent fact 3: exit code
  stdout: string
  stderr: string
}

export function run(argv: string[], timeoutMs: number): Promise<RunResult> {
  return new Promise((resolve, reject) => {
    const child: ChildProcess = spawn(argv[0], argv.slice(1), {
      stdio: ['ignore', 'pipe', 'pipe'],
    })

    let stdout = ''
    let stderr = ''
    child.stdout.on('data', (d: Buffer) => (stdout += d))
    child.stderr.on('data', (d: Buffer) => (stderr += d))

    // Independent fact 1 is maintained separately: timeout is a flag, unrelated to the exit code.
    let timedOut = false
    const timer = setTimeout(() => {
      timedOut = true
      child.kill('SIGTERM') // Timeout triggers termination
    }, timeoutMs)

    child.on('close', (code, signal) => {
      clearTimeout(timer)
      // The three fields are returned independently: the process may have timedOut=true and exitCode=0,
      // because after the timeout it caught SIGTERM and exited with 0.
      resolve({ timedOut, signal, exitCode: code, stdout, stderr })
    })

    child.on('error', reject)
  })
}

Let's go through the maintenance points in this code one by one:

  1. stdio configuration: stdio: ['ignore', 'pipe', 'pipe']. stdin is set to ignore, preventing the child process from inheriting the input stream from the parent process, which could cause unexpected interaction or hanging; stdout and stderr are both set to pipe, so that they flow back to the parent process as data events and can actually be accumulated into the result.
  2. stdout / stderr accumulation: String concatenation is used in the two data events. Note that the data callback receives a Buffer, and here the implicit conversion of the template string is relied upon to obtain text; accumulation must begin at the same moment the listener is attached, otherwise data that arrives earlier than the listener will be lost.
  3. Independent timedOut flag: let timedOut = false is declared in the Promise scope, rather than being derived from exitCode. This is the core of the entire pattern—a timeout is a fact you actively set, not a guess inferred from the process's outcome. The comment also states this clearly: this flag is unrelated to the exit code.
  4. Timer and SIGTERM: After setTimeout expires, timedOut = true is set first, then child.kill('SIGTERM') is called. Setting the flag before sending the signal ensures that no matter how the process responds to the signal, the fact of the timeout has already been recorded.
  5. Timing of clearTimeout: clearTimeout(timer) is called on the first line of the close callback. Otherwise, after the process ends normally, that timer will keep counting, eventually triggering a redundant kill after the result has already resolved, and even causing operations on resources that have already been reclaimed.
  6. Order of resolve fields in the close callback: resolve({ timedOut, signal, exitCode: code, stdout, stderr }). Three independent facts are laid out flat in the same object, with none nested under another's branch; the close callback provides both the code and signal parameters, exactly corresponding to the two outcome dimensions the user cares about.

There is one more thing that is easy to miss: child.on('error', reject). A failure of spawn itself (for example, the executable does not exist) goes through the error event, not close. If error is not listened for, this Promise will hang forever, and the caller will never receive any result—this itself is another scenario of "a simple edge case dragging down the entire Agent". run.ts rejects it out, turning the failure into an explicit error rather than an infinite wait.

dispose settling, credential wiping, link deletion, callback isolation: four easily missed cleanup actions

Result reporting solves "how to state the facts clearly"; next we need to solve "how to clean up resources thoroughly". Among the five defensive programming patterns summarized in the documentation, the four other than result reporting all revolve around lifecycle, concurrency, child processes, and cleanup code: dispose settling, credential wiping, link deletion, callback isolation. Together they serve one goal—preventing a simple edge case from bringing down the entire Agent.

First, look at dispose settling. When a plugin is unloaded or the system shuts down, dispose is not just a matter of "sending a notification" and being done; it must ensure that everything it started has truly stopped. The boundary conditions to check include: whether running child processes are terminated and awaited for reclamation, whether all timers are cleared, whether all listeners are removed, and whether asynchronous tasks are properly cancelled rather than left dangling. A common bad smell is that dispose only flips a flag without waiting for in-progress operations to finish—the phrase "settling" emphasizes "waiting for it to actually stop," not "issuing a stop command."

Second is credential wiping. During runtime, a plugin may come into contact with API keys, tokens, and temporary credentials, and these values may still linger in memory objects, log buffers, or debug output after the lifecycle ends. The boundary conditions to check include: whether dispose or the end of the lifecycle explicitly clears fields that hold credentials, whether error paths will log credentials along with everything else, and whether serialization results carry content that should not be included. The key to credential wiping is that "error paths must be wiped too," because when an exception occurs, it is easiest to casually dump the entire context.

Third is link deletion. A plugin may create temporary files, symbolic links, listening ports, or other externally visible references. These are not in-process objects, and GC will not clean them up for you. The boundary conditions to check include: whether creation and deletion come in pairs, whether there is still an opportunity for cleanup when an exception interrupts execution, and whether deletion failures are explicitly handled rather than silently swallowed. This connects to the lesson from incident review 0004—mistaking "partial success" for "complete success" is the most typical misjudgment in cleanup logic.

Fourth is callback isolation. The callbacks you register with the framework run on the framework's call stack, and any exception may affect the caller's flow. The boundary conditions to check include: whether exceptions thrown in a callback are confined to its own scope, whether the callback assumes that certain external state definitely exists, and whether concurrent triggering of the callback shares mutable state. The documentation places this item in the "concurrency" group precisely because callbacks are often invoked concurrently—shared mutable state is the most easily overlooked landmine in callback isolation.

Organize these four actions into a checklist so you can refer to them before coding:

Cleanup actionCategory to addressKey boundary conditions
dispose settlingLifecycleChild processes terminated and reclaimed, timers cleared, listeners removed, asynchronous tasks cancelled and awaited to finish
Credential wipingCleanup codeHolding fields explicitly cleared, error paths do not log, serialization does not carry sensitive values
Link deletionCleanup codeCreation and deletion paired, cleanup opportunity remains even when interrupted by exceptions, deletion failures are not silent
Callback isolationConcurrencyExceptions do not leak out, external state is not assumed, concurrent triggering does not share mutable state

These four categories of actions share one common criterion for judgment: if the way it fails is "it cannot be detected in normal testing and only blows up in production," then it belongs to this group. They are not feature implementations, but an order that functionality silently maintains even when nothing goes wrong; once some boundary is broken, what is exposed is often not a small error but the entire Agent being unable to continue running.

Four review questions: what broke, what the mechanism is, why the safety net did not catch it, and what protection was added

Defensive programming can reduce defects, but it cannot eliminate them. The approach taken by the DSH repository is to write up every bug that "should not have appeared but did" as a postmortem, and to leave behind protections at the level of tests, documentation, and rules. First, we need to clarify what a postmortem applies to: it records bugs that appear in real users, merged PRs, or released versions—that is, bugs that appear where they should not.

示意图
Postmortem process and test pyramid diagram: the path from incident discovery to the four-question review, and then to the four layers of protection—unit tests, coverage gates, real API e2e, and snapshots.

The value of a postmortem is not in that one line of fix code, but in answering four questions. The four-question structure given in the documentation is as follows, and each question has a clear audience:

  1. What broke: use a short paragraph so that a busy reader can absorb the key points within thirty seconds. This question determines the readability of the postmortem; if it is written poorly, it becomes a long article no one reads.
  2. What is the mechanism: explain the root cause in plain language, without blaming individuals. Note that this question asks for an explanation at the level of mechanism, not "who wrote it wrong."
  3. Why did every safety net fail to catch it: identify gaps in tests, tools, and conventions, rather than a one-off typo. This question elevates "a bug" to "a class of vulnerability."
  4. What protection was added: tests, AGENTS.md rules, ADRs, so that the same kind of bug clearly errors next time.

The third question is the center of gravity of the entire postmortem method. What deserves attention is why our process let it through, not merely that one line of fix. Take postmortem 0001 as an example: the plugin had an extra export default apply, the Loader's unwrapExports picked up a bare function, and the inject on the namespace was entirely lost, causing the editor (Zed) to report cannot get property "agents" without inject as soon as it connected and the first session/new ran. The answer to this question is not "the author slipped," but rather "178 green unit tests + 100% line coverage were all present, but all tests mounted via manual ctx.plugin(...), bypassing the real Loader's loading path." So the added protection was not simply deleting that default export, but deleting the default export, adding a real Loader smoke test that requires no key, and establishing the rule "test the real entry path; line coverage does not equal behavior coverage."

Postmortem 0002 is equally typical: the author used disabled: !!js ... intending to conditionally enable the filesystem plugin, but Cordis only evaluates JS expressions inside the plugin config; when reading the disabled config item directly, it sees a truthy object, so seven filesystem scenarios called tools that did not exist in the registry and returned UNKNOWN_TOOL. The answer to the third question is "snapshot refresh treated deterministic replay as behavioral correctness—it proved that the regression was stably reproduced, but it did not prove that the filesystem tools were actually registered." The added protections include switching to an explicit filesystem overlay, a static config guard that rejects expression nodes in Loader config item metadata, and the snapshot framework rejecting structured UNKNOWN_TOOL results.

Retrospectives 0003 and 0004 respectively point to gaps at the type and information level: "the Web composition does not provide the model with identity information about the current GUI, canonical URL, or run mode," and "the sandbox result type can only express a set of substrings, and cannot express that a Landlock failure must exit with code 125 plus a single line of fatal diagnostics." Putting the four cases together, a common lesson can be drawn: tests must go through the real entry path. Manually mounting, mocking everything, and treating snapshot refreshes as acceptance all make it possible for "all units green, but the product broken."

This also explains why DSH's layered testing strategy has four layers, each covering a blind spot the previous layer cannot catch:

LayerCommandWhat it catches
Unit testspnpm run testvitest runs in-package tests, prioritizing boundaries, error paths, event ordering, and concurrency races
Coverage gatepnpm run test:coverage100% coverage per file; uncovered lines are often dead code that should be deleted
Real API e2epnpm run test:e2eCalls real provider APIs with keys; automatically skipped when keys are missing, keeping keyless CI green
Snapshotspnpm run test:snapshot / test:webKeyless expected output covers external behavior; browser snapshots are compared via Chromium replay

Because the repository belongs to DeepSeek itself, there is one special principle: inference is cheap here, so do not skimp on real API tests. Keyless tests can only prove the underlying pathway; only running with keys can prove that the agent can connect to a real model and work normally. The most valuable are smoke tests—starting a real example, sending a prompt, and checking the external world—they can catch the kind of problems that mocks cannot find, where "all unit tests are green, but the product is broken." Line coverage is a necessary condition, never a sufficient one; it proves that lines were executed, not that the functionality works as delivery expects. In case 0001, 100% coverage still let two integration bugs through, which is the best footnote. A companion principle is verify the external world, not self-reports: e2e assertions should rerun commands or reread files from the outside; keyword probing of the agent's own output will let a cheating agent pass, and assertions on unmodified files should be byte-for-byte identical.

What kind of bug is worth writing a retrospective for: hidden, systemic, and costly to rediscover

Since retrospectives are so valuable, should every bug get one? The document's answer is no. Write a retrospective only when a bug satisfies all three conditions at once: hidden, systemic, and costly to rediscover. These three thresholds pull retrospectives back from "a running log" to "an asset."

The first is hidden: the mechanism is not obvious, and even a careful engineer has to work hard to rederive it. Like disabled: !!js in 0002 being read as a truthy object, or in 0004 the combination of the harmless landlock-run: partial enforcement notification with a non-zero exit code being misjudged as a sandbox failure—these root causes are not written directly in the error message, and must be traced back along the chain to be seen clearly. By contrast, a misspelled variable or a null pointer is clear at a glance from the stack, and does not need a retrospective to record the mechanism.

The second is systematicity: the escape was caused by a gap in tests, tooling, or conventions. This threshold is the easiest to overlook, but it is precisely what a postmortem should truly distill. The escape cause of 0001 was "all tests bypass the real Loader"; the escape cause of 0003 was "the Web composition never provided the model with the identity information of the current GUI"; the escape cause of 0004 was "the test matrix never constructs the combination of a notification followed by a non-zero subprocess exit"—none of these are one-off typos, but rather a blank spot in the entire quality system. A one-off typo has no systematicity, and writing it down cannot be transformed into protection, because next time the mistake will be a different typo, not the same gap.

The third is the high cost of rediscovery: it consumed real debugging time, and next time it will be the same. If a problem requires a large amount of time to relocate every time it appears, then solidifying it yields a clear return. This is also why the documentation emphasizes that "a postmortem is a retrospective failure record"—it serves future debuggers, not to convict a past mistake.

Beyond the three thresholds, it must also be made clear what a postmortem does not write: bugs that do not belong to "places where they should not appear" are not written. If a bug appears on a development branch and has not yet been merged, then it counts at most as an ordinary bug fix and does not need its own record. The essence of the threshold mechanism is to keep postmortems scarce—only when every postmortem is worth reading carefully does the postmortem culture hold up.

So where exactly do the newly added protections land? The documentation gives the landing points as tests, AGENTS.md rules, and ADRs. Tests make the same kind of bug clearly error out in CI next time; AGENTS.md rules write constraints into the collaboration conventions so that subsequent contributors see them before starting work; ADRs record architecture-level decisions and their rationale. The three correspond one-to-one with the fourth of the four questions—if a postmortem is finished but adds no new protection, then it is merely an emotional vent, not an engineering asset.

Back to the first half of this piece: we unpacked the fundamental differences among the three distribution paths—npm, tarball, and Git—in terms of build artifacts and authorization, explained why Git installation must rely on the prepare script, and why this script must be self-contained; we looked at the practical approach of turtle-ui using a dedicated tsdown configuration to directly transpile src/, and also walked field by field through the maintenance checklist in run.ts from spawn to close; finally we entered the four-question structure and three thresholds of incident postmortems. The parts to continue exploring in the second half are: how documentation discipline prevents "documentation and source code drift"—that is, how the verify-type-equiv gate and "one fact, one home" mechanisms actually operate; and how to write a truly useful thirty-second executive summary.

In the first half we advanced the plugin from running locally to a distributable asset: the three release paths—npm, tarball, and Git—each have their costs, and the prepare script must be self-contained, otherwise the source code pulled down by Git installation will never grow a lib/. But distributability is only the first layer—what truly determines whether a plugin can be trusted long-term is its behavior under boundary conditions, and whether maintainers have the ability to distill systematic protection from an incident. The four real postmortems, the four-layer testing strategy, and the two documentation disciplines in this section are the entire process by which the dsh repository turns "plugin as asset" into an engineering fact.

Postmortem 0001: export default drops inject, why 178 green tests did not stop it

This is the most stinging of the four postmortems, because it happened in a place that seemed impossible to get wrong. What broke: as soon as the editor (Zed) connected to dsh's ACP server, the first session/new request directly reported cannot get property "agents" without inject. Note the wording of this error—it does not say that some field is undefined, but that agents was accessed in the case of missing inject. In other words, the inject declaration that should have been attached to the plugin namespace disappeared entirely after loading.

The mechanism is very specific and worth reconstructing frame by frame. The plugin had an extra export default apply. In a normal module export, this looks like just one extra export, harmless; but dsh's Loader goes through a step called unwrapExports when loading a plugin module—its job is to extract the "plugin body" from the module's export object. When the module has a default export, unwrapExports prioritizes taking default, so what it gets is that bare function apply, rather than the complete export object carrying namespace metadata (including the inject declaration). The inject on the namespace is thus discarded entirely.

So the chain becomes: Zed connects → the ACP server receives session/new → it needs to resolve the agents service → the plugin's declared inject no longer exists → it reports cannot get property "agents" without inject. Nowhere in the entire process did any line of code "write the logic wrong"; the error comes entirely from the mismatch between the export shape and the loader contract.

Why didn't the safety net catch it? This is the place in the article where advanced readers most need to stop and think for thirty seconds: at the time, the repository had 178 green unit tests + 100% line coverage. The numbers are very pretty, but they all construct the context by manually mounting the plugin via ctx.plugin(...). Manual mounting means: the tests themselves hand the plugin object directly to ctx.plugin, skipping the Loader's unwrapExports step. In other words, what the tests verify is "the plugin object behaves correctly after being mounted," while what happens in production is "what the plugin object looks like after the module is loaded by the Loader." Between these two things lies a critical function, and all the tests start running after it.

This is the sharpest footnote to line coverage is not behavior coverage: those 178 tests did indeed execute every line, including every line in the plugin body, but not a single test walked the real entry path. Coverage proves code was executed; behavior coverage proves the delivery path was verified. The two cannot substitute for each other.

There are three new safeguards, tightening layer by layer:

  • Remove the default export—eliminate from the source the possibility of the Loader mistakenly taking the bare function, turning the implicit contract of "export shape" into a hard constraint.
  • Add a real Loader smoke test that requires no key—no longer manually calling ctx.plugin, but instead letting the test go through the complete Loader loading process. This test requires no key, so it can live permanently in keyless CI, with extremely low cost and extremely high interception power.
  • Establish the rule "test the real entry path; line coverage is not behavior coverage"—write it into AGENTS.md, so that when newcomers add tests they by default first ask "did I go through the real entry?"

For plugin authors, the direct conclusion of this retrospective is: do not casually add export default in your plugin module; if you really need a default export, first confirm the Loader's unwrapExports semantics. The more general lesson is: any combination of "loader + module shape" may hide this kind of pitfall, and writing a smoke test that goes through the real loading path for it is worth more than adding a hundred unit tests.

Retrospective 0002: How a single literal !!js object permanently disabled the filesystem snapshot tool

What broke: The filesystem snapshot tool failed in all seven filesystem scenarios, all of which were calling tools that did not exist in the registry at all, uniformly returning UNKNOWN_TOOL. In other words, the tools were not "failing to execute"—they were "never registered in the first place."

What the mechanism was: The author wanted to use the config item disabled: !!js ... to conditionally enable the filesystem plugin. This is a common custom-tag pattern in YAML, with the intent being "the value of disabled is determined by evaluating a JS expression." The problem lies in the fact that Cordis only evaluates JS expressions inside the plugin config—only within the parsing context of the config object are such expression nodes actually executed into boolean values. But when someone directly reads the top-level config item disabled, what they see is a truthy object (the unevaluated expression node itself), not false.

Objects are truthy, so "conditional enable" was statically judged as "disabled," and the filesystem plugin was permanently disabled. All seven scenarios therefore hit unregistered tools and returned UNKNOWN_TOOL.

There is a layered fact here that is easily overlooked and deserves its own table: the same disabled field has completely different semantics depending on where it is read.

Read locationValue seenTruthiness resultConsequence
Inside plugin config (Cordis evaluation context)Boolean value after JS expression evaluationBased on the expression's actual resultConditional enable takes effect
Directly reading the top-level disabled config itemUnevaluated literal !!js objecttruthyPlugin is statically disabled, tools are not registered

Why didn't the safety net catch it? The snapshot framework treated "deterministic replay" as "behaviorally correct." The logic of snapshot testing is: store an expected output, then compare whether it matches on every subsequent run. It did prove that "the regression is stably reproduced"—every run produces the same result, very deterministic. But it did not prove at all that "the filesystem tools were actually registered." When the bug itself is deterministic (always UNKNOWN_TOOL), the snapshot instead solidifies the error as a baseline, and consistency masks correctness.

Three new safeguards were added as well:

  • Switch to an explicit filesystem overlay—no longer rely on fragile expression conditions; express "enable/disable" as an explicit structure.
  • A static config guard rejects expression nodes in Loader config item metadata—block such dangerous evaluation nodes at the config parsing stage, making it fail at startup rather than silently degrade at runtime.
  • The snapshot framework rejects structured UNKNOWN_TOOL results—add a semantic rule to snapshots: if the output contains a structured result like UNKNOWN_TOOL indicating "the tool was not registered," judge it as a failure directly and do not allow it to be treated as a valid baseline.

Takeaways for plugin authors: the scope of config expressions must be explicit; do not treat "it will be evaluated at A" as "it will be evaluated everywhere"; for any conditional enable/disable switch, prefer an explicit overlay over embedded expressions. In addition, adding semantic assertions to snapshot tests (rejecting specific structured errors) is a low-cost way to upgrade "consistency" to "correctness."

Retrospective 0003: The Web agent validated a substitute server instead of the GUI hosting the session

What broke: The agent modified the GUI source code, but did not know which URL the current session corresponded to or which process was hosting it—so it "validated" a completely unrelated server and confidently declared it a pass.

What the mechanism was: There were several consecutive errors throughout the process. In the first step, the agent accessed the bare Vite service, received an HTTP 200, and treated it as a success signal; but that 200 was actually returning a blank page—a correct HTTP status code does not mean the page is correct. In the second step, it then went to validate a substitute dsh web server on another port, treating the behavior of that substitute instance as the behavior of the GUI it was supposed to fix. In the third step, the most fatal one, it never probed port 3081, where the real GUI hosting the current session was located.

Looking at the three actions together, the agent did not "fix the wrong thing"—it "misidentified the object." It did not know at which address its changes should be verified, so it casually grabbed something that could return 200 and started scoring it.

Why didn't the safety net catch it? The root cause lies in missing identity information: the Web composition did not provide the model with any information about the current GUI, the canonical URL, or the run mode (development/production). Lacking the premise of "who am I, and where should I be verified," the model could only rely on port-scanning-style guesswork. The second problem was that the regression test treated "process timeout" as "fast failure"—the intent was to expose failures faster, but the result was false positives that masked the real problem.

New safeguards added:

  • The launcher publishes the canonical loopback URL and the actual run mode—through environment variables + a prompt section, the "canonical URL" and "production/development mode" are explicitly communicated to the model so it does not have to guess.
  • Standalone Vite service mode is rejected at configuration time—mechanically preventing the agent from mistaking bare Vite for the validation target.
  • Layered real-path tests cover the CLI, prompts, runtime facts, and browser HMR—turning "what the agent should know" into testable assertions.

This retrospective is extremely valuable for people doing Agent engineering: tool correctness depends on identity context. When a model needs to operate a web application, the "canonical address + run mode" must be injected as runtime facts; otherwise it will make decisions based on weak signals such as status codes. At the same time, using timeouts to masquerade as failures is a common testing anti-pattern that conflates "slow" with "wrong."

Retrospective 0004: Landlock partial enforcement notification caused child process failures to be misclassified

What broke: On older Landlock ABI kernels, ripgrep exits normally with exit code 1 when there are no matches—this is ripgrep's established semantics (1 means no match, 2 is an error). But dsh presented this normal "no match" as a SANDBOX_UNAVAILABLE sandbox failure, reporting a successful search as an environment breakdown.

What the mechanism was: On older kernels, the launcher prints a harmless line of notification: landlock-run: partial enforcement (older Landlock ABI), indicating that the kernel supports partial enforcement. The harness's decision logic used a case-insensitive substring match for landlock-run:, and combined this substring with "any non-zero exit code" for its judgment—as long as this prefix appeared in the output and the process exited with a non-zero code, it determined that the runner had failed. ripgrep's no-match exit code 1 happened to satisfy this combination, so it was misclassified as sandbox unavailable.

Why didn't the safety net catch it? The sandbox result type can only express a set of substrings, and cannot express a more precise contract: "a Landlock failure must simultaneously satisfy exit code 125 + one line of fatal diagnostics." The test matrix also never constructs the combination of "a notification followed by a non-zero subprocess exit"—every test either tests the notification or tests the failure exit, and never puts the two together. The gap is that no one covers the combined scenario.

The core of the fix is RunnerFailureRule, which expresses "what kind of situation counts as a runner failure" as a set of structured fields rather than a vague substring:

FieldMeaningProblem solved
Allowed exit codesA whitelisted set of exit codes (e.g., a Landlock failure must be 125)Eliminates the loose matching of "any non-zero code counts as a failure"
Line-by-line fatal signatureFatal diagnostic text that must be matched line by lineDistinguishes harmless notification lines from genuinely fatal lines
Precisely excluded informational linesExplicitly lists lines that are informational and do not participate in the judgment (such as partial enforcement notifications)Stops "a notification followed by a non-zero exit" from being misjudged

At the same time, filesystem search switched to ctx.subprocess running the packaged ripgrep directly, no longer going through sandboxed bash—decoupling "sandbox judgment" from "search execution" and avoiding this kind of misclassification at the path level.

Putting the four retrospectives together, one common lesson can be distilled: tests must go through the real entry path. Manually mounting, mocking everything, and treating snapshot refreshes as acceptance all make it possible for "all units green, yet the product broken." 0001 was the load path being bypassed, 0002 was snapshots treating determinism as correctness, 0003 was missing identity context + timeouts masquerading as failures, 0004 was no one covering combined scenarios—four faces, one root cause.

Four layers of testing: what pnpm run test / test:coverage / test:e2e / test:snapshot each catch

Once you know the root cause, you need a layered strategy to close it. The tests in the dsh repository are layered, with each layer specifically filling in the blind spots the previous layer cannot catch—not a simple stacking, but a division of labor.

LayerCommandWhat it catchesWhat it doesn't catch
Unit testspnpm run testvitest runs in-package tests, prioritizing boundaries, error paths, event ordering, and concurrency racesReal load paths, real API behavior
Coverage gatepnpm run test:coverage100% coverage per file; uncovered lines are often dead code that should be deletedBehavioral correctness—a line being executed ≠ the feature working as expected
Real API e2epnpm run test:e2eCalls real provider APIs with keys, verifying the agent can connect to real modelsAutomatically skipped in environments without keys
Snapshotspnpm run test:snapshot / test:webKeyless expected output covers external behavior; browser snapshots are compared via Chromium replaySemantic correctness—requires additional rules to reject structured errors like UNKNOWN_TOOL

Unit tests are the first net, and also the easiest one to get wrong. Their focus is not "covering every line," but rather prioritizing boundaries, error paths, event ordering, and concurrency races—precisely the categories that never surface during normal testing but blow up the moment you ship. When writing unit tests, asking yourself "am I constructing the most devious possible input ordering" is far more valuable than asking "how many lines did I cover."

Coverage gates require 100% per file. There is an easily overlooked byproduct here: uncovered lines are often dead code that should be deleted. If your 100% gate forces you to write a test for a particular line, and you cannot imagine when that behavior would ever occur, then in all likelihood that line should not exist in the first place. Coverage is a tool for "discovering redundancy," not merely a KPI.

Real API e2e requires keys, and automatically skips when keys are absent, thereby keeping keyless CI green—this is the precondition for it to persist in open-source/multi-cloud environments. But the repository has one particularly noteworthy principle worth copying into your own project: inference is cheap here, so do not skimp on real API tests. Because keyless tests can only prove that the underlying pathway is connected; only running with keys can prove that the agent can actually work against a real model.

The snapshot layer covers external behavior with keyless expected outputs, and browser snapshots are compared via Chromium replay. Its value is catching "unexpected drift in the external contract," but as shown in retrospective 0002, it must be paired with semantic-level rejection rules, otherwise it will fossilize errors into the baseline.

Above all layers, the highest-value one is the smoke test: launch a real example, send a prompt, and check the external world. They can catch the class of problems where "all unit tests are green but the product is broken"—problems that mocks simply cannot detect. To emphasize that conclusion once more: line coverage is a necessary condition, never a sufficient one. It proves that lines were executed, not that functionality works as the delivery expects. The fact that 0001's 178 green lights + 100% coverage still let two integration bugs slip through is the best footnote to this.

Verify the external world, not self-reports: e2e assertions and byte-by-byte comparison

Having layers is not enough—how assertions are written equally determines success or failure. There is an iron rule here: verify the external world, not self-reports.

What is a self-report? It is reading the agent's own output text after e2e finishes, checking whether keywords like "success" or "completed" appear. This approach has a fatal flaw: an agent that cheats (or is merely overconfident) can pass the test simply by writing the correct keywords into its output. The test is effectively rewarding "sounding good" rather than "doing it right." More insidiously, even if the agent does not cheat, its self-description may be inconsistent with the actual state—in retrospective 0003, the agent accepted the wrong server yet declared success, a textbook case of "self-report diverging from external truth."

The correct approach is:

  • e2e assertions should re-run commands, or re-read files from the outside—do not trust what the agent said, only trust independently observable state.
  • Assert that unmodified files are byte-for-byte identical—not "roughly unchanged," not "content contains," but byte-for-byte identical. This catches those hidden side effects like "formatted it on the way" or "accidentally changed line endings."
  • Judge success or failure using structured results, not natural language. Structured signals like UNKNOWN_TOOL should be directly recognized as failures by the test framework, rather than letting the model describe them.

Connect this principle with retrospectives 0001 and 0004: the disease in 0001 was that the test went through a fake entry point (self-reported mounting), and the disease in 0004 was that the decision logic was too broad (substring matching). Their remedies point in the same direction—verify with independently checkable external facts, not with in-process self-reports or fuzzy matching. Asserting external side effects (file contents, the precise contract of a process exit code, the real response body of a URL) is harder to cheat than asserting internal state, and it is closer to what users perceive.

Documentation discipline: the verify-type-equiv gate and one fact, one home

Beyond code, documentation is the second asset that is prone to rot. The documentation in the dsh repository is not "done once written"; instead, it has mechanisms to prevent documentation from drifting away from the source code.

The first gate is called verify-type-equiv. Its way of working is quite hardcore: it uses the TypeScript parser to extract the symbols of type declarations from the source code, along with the JSDoc attached to those declarations, and then asserts that the code blocks in the documentation match both at the same time. This means that the type definitions pasted into the documentation are not hand-copied text, but mirrors verified by the gate. When you change a documented type declaration or its JSDoc, the gate fails until you synchronously update the pasted content in the documentation. This eliminates the most common kind of drift: "the documentation copied an old version."

The second discipline is called one fact, one home: each fact is maintained in only one file, and the other files reference it. Specifically for tool schemas, its "source of truth" is in adding-a-tool.md, and other pages reference it rather than copying it. The benefit of doing this is: change one place, and everything stays consistent; copy it in three places, and once you change two of them, they start lying.

Chinese and English documentation are maintained through bilingual pairing, and updates have a clear order: first run pnpm run gen-doc-graphs to update English, then update Chinese and verify the pairing. The order cannot be reversed, because graph generation is based on English.

Translate this discipline into your own project: put the single source of truth in one place, and do not let the same configuration be scattered across three documents. Once you find that a configuration item is described in multiple places, immediately designate one source of truth and change the other locations to references. Furthermore, if you are also writing type-related documentation, make "type snippets in the documentation must be verifiable by the parser as equivalent to the source code" into a gate, and the payoff will far exceed expectations.

Latest practice as of September 2026: write the four retrospective questions, orthogonal reporting, and prepare self-containment into CI gates

Retrospective culture is most easily misunderstood as "writing documentation." In reality, a retrospective is complete only when it produces a new protective mechanism. The four questions in the dsh repository are asked very clearly:

The four retrospective questionsWhat they need to answer
What brokeUse a short paragraph to let busy readers absorb the key points within thirty seconds
What the mechanism isExplain the root cause in plain words, without blaming individuals
Why every safety net failed to catch itIdentify gaps in tests, tools, and conventions, rather than a one-off typo
What protection was addedTests, AGENTS.md rules, ADRs, so that similar bugs clearly error out next time

Note the thirty seconds in the first question: this is not a figure of speech. A busy reader needs to grasp the summary within half a minute, so the formula for a summary is what broke → the root cause in plain words → why it escaped → the lesson that can be carried forward long-term. At the same time, not every bug is worth writing a postmortem about; write one only when three conditions are met simultaneously: subtle (the mechanism is not obvious, and even a careful engineer would have to work hard to re-derive it), systemic (the escape was caused by gaps in tests, tooling, or conventions), and costly to rediscover (it consumed real debugging time, and it will do so again next time). This threshold turns a postmortem from a "running log" into an "asset worth archiving."

By September 2026, the practice that truly makes postmortems compound is to solidify their conclusions into CI checks and AGENTS.md rules, rather than leaving them in documents and relying on people's self-discipline. The following AGENTS.md excerpt can be copied directly into your repository and used together with CI:

# AGENTS.md (excerpt): reusable hard rules

## Testing
- Any new plugin must come with a smoke test that **goes through the real Loader loading path**;
  it is forbidden to cover loading behavior only with a manual ctx.plugin(...).
- Unit tests should prioritize covering: boundary values, error paths, event ordering, concurrency races.
- Coverage is 100% per file; lines for which a triggering scenario cannot be written are treated as dead code to be deleted.
- e2e assertions must verify the **external world**: re-run the command or re-read the file from outside;
  it is forbidden to probe the agent's own output for keywords.
- When asserting on an unmodified file, it must be byte-for-byte identical;
  "contains" or "roughly the same" is not allowed.

## Result reporting
- Subprocess results must be **reported orthogonally and independently**: timedOut / signal / exitCode
  each get their own field; it is forbidden to nest the reporting of any one flag inside the branch of another.

## Release
- The prepare script must be **self-contained**: it must not depend on a monorepo checkout,
  must not use project references, and must not depend on context that exists only in the dev environment.
- The Git installation path must be able to build the release entry point from source in a clean environment.

## Configuration
- It is forbidden to use expression nodes (!!js and the like) in Loader configuration item metadata;
  use explicit overlays for conditional enabling instead.
- Sandbox failure determination must use a structured RunnerFailureRule:
  allowed exit codes + line-by-line fatal signatures + precisely excluded informational lines.

The accompanying CI checks (illustrative; rewrite them according to your pipeline syntax):

# Key CI pipeline steps (pseudo-YAML; rewrite for your actual platform)
steps:
  - name: unit
    run: pnpm run test

  - name: coverage-gate
    run: pnpm run test:coverage   # 100% per file; uncovered lines are treated as dead code

  - name: real-api-e2e
    run: pnpm run test:e2e        # auto-skips when the key is missing; keyless CI stays green
    env:
      DSH_API_KEY: ${{ secrets.DSH_API_KEY }}

  - name: snapshot
    run: pnpm run test:snapshot   # reject structured errors such as UNKNOWN_TOOL from entering the baseline

  - name: web-snapshot
    run: pnpm run test:web        # Chromium replay comparison

  - name: doc-graph
    run: pnpm run gen-doc-graphs  # update the English documentation diagrams first

  - name: verify-type-equiv
    run: pnpm run verify-type-equiv  # documentation type blocks must be equivalent to source symbols + JSDoc

Once these three things are written into CI, the four categories of defects shift from "something people have to remember" to "the build fails if it doesn't pass":

  1. Test the real entry path — if 0001's default export drops inject, and there had been a smoke test going through the Loader at the time, it would have gone red before Zed even connected.
  2. Report orthogonal results independently — timedOut, signal, and exitCode are each returned independently, so the caller won't mistake "timed out, then caught SIGTERM and exited with code 0" for a normal success. A child process can be timedOut=true and exitCode=0 at the same time, and both facts must be visible.
  3. prepare must be self-contained — a Git install pulls source code, not build artifacts, and no step automatically runs your build script; if prepare depends on a neighboring monorepo, what users get is a TypeScript package with no lib/, and loading fails outright.

That way, the postmortem is no longer a "memorial after the fact" but a positive loop of "turning one incident into a set of safeguards." The question that truly matters is always this: the value of this bug isn't in that one-line fix, but in why the process let it through, and what safeguard was added so that the same kind of problem fails loudly next time.

As for the illustration, this section happens to center on the incident postmortem flow and testing pyramid diagram provided by the repository: it places "discover incident → four-question attribution → add safeguard" together with the layered pyramid of "unit / coverage / e2e / snapshot" in the same diagram, making it clear at which layer a safeguard should be added.

Summary and Best Practices

Condense the entire article (publishing + defensive programming + incident postmortem) into an actionable checklist. It serves both as an acceptance sheet for plugin authors and as daily discipline for maintainers.

Publishing and delivery:

  • Choose among the three distribution paths as needed: npm and tarball deliver prebuilt artifacts, requiring no build authorization on the user side; a Git install pulls source code and requires build authorization (pnpm ≥ 10).
  • When installing via Git, the plugin must provide a prepare script so that pnpm builds the publish entry from source after installation.
  • prepare must be self-contained: it cannot assume a surrounding monorepo checkout, must not use project references, must not do type checking, and should transpile src/ directly. Reference form: "prepare": "tsdown -c tsdown.publish.ts".
  • Choosing the wrong distribution method will definitely cause trouble on the user side: making users perform extra authorization, or using source code as an artifact, are both predictable failures.

Five patterns of defensive programming:

  • Report results orthogonally: timedOut, signal, and exitCode each become independent fields — one fact, one field, never nested reporting.
  • dispose stops cleanly: cleanup must wait until things have settled; don't let half the resources be released first.
  • Credential wiping: wipe as soon as you're done, leaving no residue.
  • Link deletion: handle link semantics when deleting, to avoid accidental deletion or dangling links.
  • Callback isolation: an exception in one callback must not drag down the entire Agent. The overall goal is just one sentence: don't let a simple edge case take down the entire Agent.

Postmortem culture:

  • Four questions set the frame: what broke / what is the mechanism / why did every safety net fail to catch it / what protection was added.
  • A postmortem is worth writing only when all three conditions are met: subtle, systemic, and costly to rediscover.
  • A postmortem must produce verifiable protection: a test, an AGENTS.md rule, or an ADR.
  • Four verified lessons: don't add a redundant export default (0001); don't rely on the scope of a config expression (0002); inject the spec URL and run mode into the Agent (0003); use structured rules rather than substring matching for sandbox failure determination (0004).

Testing and verification:

  • Four layers of division of labor: unit tests catch boundaries/errors/ordering/races; coverage is 100% per file and incidentally surfaces dead code; e2e with keys proves real model integration; keyless snapshots and Chromium replay catch external drift.
  • The premise for keeping keyless CI green is that e2e automatically skips when the key is missing.
  • Verify the external world, not self-reports: e2e re-runs commands or re-reads files from the outside; do not do keyword probing on the agent's own output; unmodified files are byte-for-byte identical.
  • Key takeaway: line coverage is a necessary condition, never a sufficient one.

Documentation and long-term maintenance:

  • Use the verify-type-equiv gate to ensure that type blocks in the documentation are equivalent to source symbols + JSDoc, and any change triggers an error.
  • One fact, one home: the source of truth for the tool schema is in adding-a-tool.md, and other pages reference rather than copy it.
  • Bilingual paired maintenance order: first run pnpm run gen-doc-graphs to update the English, then update the Chinese and verify the pairing.

Finally, back to the line that runs through the entire series: the model is responsible for being smart, the Harness is responsible for being reliable. Constraints are not restrictions, but the cornerstone that makes the Agent predictable, auditable, and replayable. Everything is a plugin—put policy at extension points rather than writing it into the loop, so the system can evolve without going out of control. The real entry path beats all mocks—coverage, snapshots, and smoke tests each play their part, together preventing "all green but broken." And failure is not scary; what is scary is not knowing why: postmortem culture turns one incident into a set of protections, making today's plugin an asset that tomorrow can be relied on with confidence.