evals

name evals description Inspect, validate, or run isolation-gated st2 agent-team eval cells. Use for corpus inventory, free preflight, an explicitly authorized cell run, or adding a held-out-graded cell. evals Use the repository's canonical surfaces: AGENT-SPEC.md for current hand-authored st2 agent declarations; CATALOG.md for the generated inventory, cost bands, exclusions, and evidence; cells/<cell>/<cell>.kdl for executable cell behavior. Default workflow Run the free gate first: bin/check-corpus.sh Inspect the no-execution overnight plan with: bin/overnight.sh --dry-run bin/overnight.sh --dry-run --cell <cell> --cell <cell> Never infer permission for a paid/model run. When one cell is explicitly authorized: st2 eval ./cells/<cell>/ --keep When an exact conservative paid queue is explicitly authorized, repeat --cell in the approved order: bin/overnight.sh --run --cell <cell> --cell <cell> --state-dir .eval-runs/overnight The common Codex reset-available banner stops the default run before the next cell. A future full unattended run therefore requires separate human review of both --all and the higher-spend informational-banner opt-in: bin/overnight.sh --run --all --allow-informational-reset-banner --state-dir .eval-runs/overnight Paid --run rejects an omitted selector, duplicate, unknown, or retired cell. Explicitly selected maintained model-free cells execute through the same st2 lifecycle and receipts without provider checks; Claude/Codex binary and authentication checks apply only when the selected subset needs that provider. Hard quota/rate-limit errors stop unconditionally in both modes. Never infer approval for either paid command. Cell rules One canonical root KDL per cell and one declared max-timeout . The eval runner owns the native flat bus and pty roots. Every model subject and model judge selects its model and medium effort explicitly. Every bus-connected model agent uses native bare ding , one cold-start drain, and event-first DING wakeups. Every Claude workspace materializes the canonical Claude loader/persona/hooks; every Codex workspace materializes the canonical AGENTS/hooks and trusts them at launch. Fixtures are synthetic and resettable; checked-in Git metadata is frozen as _git . Graders are held out, ownership-aware, and mutation-valid where practical. No paid rerun is a substitute for shell, parse, reset, model-policy, and PII gates. After changing inventory or structured evidence, run: bin/generate-catalog.sh --write bin/check-corpus.sh