n8n-create-instance-ai-eval

name n8n:create-instance-ai-eval description Authors a new Instance AI workflow eval case — written locally as JSON, calibrated against a real build, then pushed to the LangTracer suite CI runs — build cases, behaviour/process cases, credential cases, and seeded (mid-conversation) cases — with intent-driven expectations. Use when adding or changing an Instance AI workflow eval, or debugging why one is flaky. Create an Instance AI workflow eval Each eval is one JSON case — authored locally as a file in packages/@n8n/instance-ai/evaluations/data/workflows/ (the disk loader auto-discovers *.json , no registration step), with a LangTracer suite as its durable home. Cases validate against harness/schema.ts ( .strict() — unknown keys fail at load). The eval README is the exhaustive field reference; this skill is the opinionated how . Committing new case JSONs into the repo is no longer the recommended approach. Author the file locally (uncommitted), calibrate it against a real build, then push it to a lang-tracer suite with eval:langtracer-push (see Push to a lang-tracer suite ) — --suite baseline for the consolidated corpus n8n CI runs, or a dedicated capability suite like agents . The suite is the home for the case; the eval CLI reads it back via --source langtracer . You still write the JSON file — it's just the input to the push, not a committed artifact. Seeded cases. An inline seed pushes with the case — the case-write API stores it verbatim, so the suite is its home like any other case. Only a seed.mode: "replay" case is refused (listed under skipped: ): it's reconstructed from a LangSmith trace at run time, so it dies when that trace is pruned and has no durable home. Don't commit a replay case either — derive a synthetic case from it. See case-shapes.md . Set the autonomy level first Before you source, draft, or run anything, decide how hands-on the driver wants to be — and say it back. This skill runs at one of two autonomy levels. If the request makes the level clear ("just author and calibrate it yourself" vs. "stop me at each step", or an explicit mode), adopt it, state it in one line, and note how to override (e.g. "say 'stop me at calibration' to add a checkpoint"). If it's not clear, ask the driver one question offering the two levels before doing any work. The skill has four natural decision gates — selection (which real failure to encode), shape + expectations (archetype, must-haves, scope trim), calibration (classify each red and resolve keep/loosen/drop), and push (kind + tier). The level decides what happens at each gate: Level Who decides when to stop Behaviour autonomous agent Runs all four gates start-to-finish; reports a decision log at the end for the driver to review — with the pushed case, its suite, and the source thread as links ( Share links, never bare ids ); includes the observation id when the driver opted to record one; and includes a Linear ticket proposal for any kept capability-gap red ( Capability gap → propose a Linear ticket ). checkpoint driver, per gate Stops at each gate with a compact proposal + recommendation ; driver says "go" or redirects. At the calibration gate, hands the driver a link to the just-built thread on the live instance plus login credentials so they can review the real conversation and workflow themselves before confirming (below). Calibration is special-cased at both levels. A calibration verdict that flips a case's meaning — a real capability-gap red vs. a harness-caused red, or any loosening that would let a known-bad build pass — is surfaced explicitly (interactively in checkpoint; in the decision log in autonomous), never silently committed. It's the one call where a quiet mistake corrupts the suite, so it never fully auto-commits. Recording a source observation is optional, not a fifth gate. After a real thread passes selection, offer to save why it was selected and what the developer observed with LangTracer's create_observation tool (see sourcing-cases.md ). If the driver declines, has not stated a preference in autonomous mode, or the write fails, continue with the eval. Never block drafting, calibration, or push on an observation. create_observation and update_observation write only to LangTracer; do not add a LangSmith feedback or sync step. Checkpoint calibration — review the real thread on the instance. Because the calibration verdict is trust-critical, in checkpoint mode you don't ask the driver to trust your reading of the run. You built the case against a live instance with --keep-workflows (step 4), so the thread and the workflow are still there — hand the driver a direct link and let them look: Thread: <base-url>/assistant/<threadId> — the exact conversation the case ran (the run prints the threadId ; the built workflow prints as BUILT (<id>) and opens at <base-url>/workflow/<id> ). Login: the email + password the eval signs in with (the owner you seeded on the instance — see running-evals.md ; the default local seed is nathan@n8n.io / PlaywrightTest123 ). Present, per red: the assertion, whether it went green/red, your proposed classification (real capability gap / harness limitation / noise) and keep/loosen/drop, and the review link. The driver logs in, reads the thread and the workflow, and confirms or redirects before you write the verdict back into the case description . Share links, never bare ids Every lang-tracer entity has a shareable web page, but the CLI and the MCP hand you numeric ids — eval:langtracer-push prints + created <slug> (#621) , get_eval_run returns a run number, list_conversations returns thread ids. An id is unclickable: the driver has to go find it. Whenever you name a case, suite, thread, cluster, or run in anything a human reads — a checkpoint proposal, the end-of-run decision log, a PR description, a Linear ticket, a Slack message — render it as a link, keeping the id in the label: pushed as [#621](https://lang-tracer.n8n-maintenance.workers.dev/test-cases/621) Build links off the web base ( LANGTRACER_URL , in production https://lang-tracer.n8n-maintenance.workers.dev ). Never off the API bases — ${LANGTRACER_URL}/api/v1 and /api/mcp are machine endpoints, and a link into either 404s for the driver or dumps JSON. Entity URL Where the id comes from Test case <base>/test-cases/<id> push output (#<id>) ; create_test_case / search_test_cases Suite <base>/suites/<suiteId> push header Suite "<slug>" (#<id>) ; list_suites Source conversation <base>/conversations/<threadId> list_conversations / get_conversation Cluster report <base>/clusters/<id> list_cluster_runs / get_latest_cluster_run Eval run (sweep) <base>/results?sweep=<runId> list_eval_runs / get_eval_run ( runId is sweeps.id , the "run #N") Two links that are not lang-tracer and don't take this base: the built thread ( <base-url>/assistant/<threadId> ) and workflow ( <base-url>/workflow/<id> ) live on the n8n instance the eval ran against. When both are relevant — reviewing a calibration red, writing a capability-gap ticket — give both, labelled, so nobody has to guess which host a link points at. Where the best cases come from The strongest cases encode a real failure, not an invented premise. Two connections help you find and verify one: LangTracer clusters real conversations into capability-gap themes (discover what actually fails, at scale), and LangSmith holds the raw traces (verify exactly what happened in a run). LangTracer is the discovery layer; the durable artifact is almost always a synthetic case you author from what you learn (use seed.mode: "replay" only per case-shapes.md ). See sourcing-cases.md for connecting the MCPs and the discover → verify → encode workflow. Pick the case shape first The corpus is four archetypes. Decide which you're writing before you draft — it determines the fields, the grading, and how you validate. They compose (a seeded case can still assert outcome), but the primary shape drives the work. Archetype Question it answers Primary fields Build (default) Does the workflow the agent builds actually work ? outcomeExpectations + executionScenarios Behaviour / process Does the agent converse correctly (ask the right clarifying question, not re-ask, honour a correction, respect plan approval)? processExpectations + multi-turn director script; often build-only Credential Does the build behave correctly given a specific credential view? credentials[] Seeded Start mid-thread, with prior work already in place, and drive the turn under test seed (authored mode: "inline" ; "replay" for a local check) Build is documented in full below. The other three, the director-script vocabulary, and the seeding modes are in case-shapes.md . Core principle (all shapes) Write expectations from intent, then calibrate against a real build. Decide up front what makes any correct solution correct — the must-haves implied by what the user actually said — then build the workflow once for real to calibrate granularity: loosen what's over-specified, confirm the must-haves are achievable, and catch requirements the agent legitimately satisfies a different way. Don't transcribe one observed build into assertions — that overfits the eval into "did the agent reproduce that run" instead of "did it solve the problem." Keep the conversation in the user's voice. State the goal and real constraints the way a user would — don't name node types, wire up the structure, or restate your outcomeExpectations in the prompt. If the conversation spells out the build, the case only tests whether the agent can follow instructions and the expectations become tautological; the gap between what the user asks for and how a correct workflow realizes it is the capability under test. Even when the anchor is honoring a user's stated technical preference, phrase it as their need + constraint ("I need field X and the built-in node doesn't expose it, so pull it straight from the API") — not as an implementation spec ("use an HTTP Request node"). Write the conversation in English unless the user asked otherwise (or the case exists specifically to test non-English handling). Sourced real threads are frequently non-English — translate the intent into English when you rewrite the prompt in the user's voice; the failure mode is the anchor, not the original language. Trim to the smallest multi-turn conversation that reproduces the issue. Real sourced threads are long (dozens of turns of setup, debugging, and tangents) — do not transcribe them. Distill to the fewest turns that still drive the build or behaviour under test. Every retained turn must earn its place: a turn stays only if it is load-bearing — a value the agent must ask for (withheld until asked, via a director note), a correction/push-back the case exists to test, or a plan approval that gates the build. If removing a turn doesn't change what's tested, remove it. Collapse to a single turn whenever the whole request can be stated at once without a load-bearing exchange; keep it multi-turn only for those exchanges, and keep each director script in one turn (don't fabricate assistant "done" turns to sequence steps — see case-shapes.md ). A minimal conversation isolates the capability; a transcribed one buries it in noise and tests instruction-following. Size the build, not just the assertions. Real sourced prompts are often kitchen-sink ("production-ready, runs forever, 3 feed posts and 8 stories a day", "generate 50 articles daily") and reliably blow the ~900s build budget (see "Known harness limitations"). A faithful trim is a legitimate authoring move : reduce batch sizes, drop one of several parallel pipelines, or merge adjacent AI steps so the case builds within budget — then note the reduction in the case description ("the original request also asked for an 8-stories/day pipeline; scoped to feed posts so it builds in budget"). Keep the capability under test; cut the combinatorial bulk. A case that never builds tests nothing. Workflow These steps map to the four gates from Set the autonomy level first : sourcing (before step 1) is the selection gate, including the optional offer to record a source observation; steps 1–2 are the shape + expectations gate; steps 5–6 are the calibration gate; steps 7–8 are the push gate. In autonomous mode you flow through all of them and summarize in a decision log; in checkpoint mode you pause at each with a proposal, and at calibration you hand the driver the thread link + login to review the real build (see Set the autonomy level first ). Calibration (step 6) always surfaces meaning-flipping verdicts explicitly regardless of level. State the must-haves first. From the conversation alone, list what every correct workflow must do (trigger type, essential operations, gating condition) — those become draft outcomeExpectations . Required fields: conversation (≥1 turn, first user ), complexity , tags , and at least one of executionScenarios / processExpectations / outcomeExpectations . Draft the case from the template below; validate it loads (see "Validate"). Smoke-test the environment with one case before any batch. Run a single case end-to-end first. This validates auth / model / --base-url / the built dist for ~1/Nth the cost — distinct from validating a case . If that one case crashes at execution (especially with an identical error you'd expect to hit every case), fix the environment before running the batch (see "A red is signal" → environment check). Running 15 cases only to discover a stale-dist crash on all of them wastes a full run. Build it once against a running instance (see running-evals.md ) with --keep-workflows so the built workflow stays for inspection. Inspect — read the built workflow (the run prints BUILT (<id>) ; fetch via GET /rest/workflows/<id> ) and the HTML report's transcript to see what the agent actually did. Calibrate — sharpen assertions; never dull them to force a green. Fix assertions that are genuinely mis-sized: relax one that pins a choice the conversation left open (so a valid alternative build wrongly fails), tighten one a wrong build would slip past, and phrase executionScenarios to match how the workflow runs on mocked data. But when a scenario goes red because the build has a real gap, or because the harness can't exercise it, that red is the result — keep it and surface why (see "A red is signal", below). Never delete a scenario, weaken an assertion, or drop to build-only just to make the run green. Push to the suite — do NOT commit the JSON. Once calibrated, push the case into its curated lang-tracer suite with eval:langtracer-push (see Push to a lang-tracer suite ); the suite is the case's home, not the repo. Leave the data/workflows/*.json file uncommitted (or delete it once it's in the suite). Committing new case JSONs into the repo is no longer the approach. (An inline seed pushes with the case; only a replay case is refused — it's a local throwaway; see case-shapes.md .) For a sourced case, finish by linking it to its source thread/finding over the MCP — see Link the pushed case to its source . Hand back links, and a ticket proposal if the case found a gap. Report the pushed case as <base>/test-cases/<id> , not #<id> ( Share links, never bare ids ), and if calibration kept a real capability-gap red, propose a Linear ticket for it ( Capability gap → propose a Linear ticket ) rather than leaving the gap as a red case nobody owns. --iterations N is available to measure flakiness (pass@k / pass^k) — reach for it when you suspect a case is non-deterministic or before promoting it to a gated tier, not as a routine step (each iteration is a full build + execution). Gut-check: if you can't picture a plausible wrong build that this case reliably turns red , the assertions are too loose to guard anything. Confirm the precondition fired, not just the green. For any conditional assertion — "when X happened, the agent did Y" (most processExpectations , and any behaviour case) — a pass has two readings: the agent did Y, or X never happened and the assertion passed vacuously. A behaviour case that hinges on the mock producing a specific failure (e.g. an AI node simulated to empty so a downstream parse node fails) is the classic trap: if the mock instead returns parseable data, the failure never occurs and the case guards nothing while showing green. Calibration must read the execution trace and the agent's finalText ( buildTrace.finalText in the verifier snapshot, or the HTML report) and verify X actually materialised — the direct-loop eval-results.json does not persist per-expectation judge reasoning, so pass/fail alone can't tell you which reading you got. A sourced failure that no longer reproduces is still worth keeping — it's now a regression guard. When you encode a real failure and calibration shows the current build handling it correctly (behaviour drifts across versions), the case doesn't lose value: it flips from capability-gap (currently red) to regression guard (currently green, catches a re-introduction). Keep it — but only after the non-vacuous check above proves it would turn red on the bad behaviour, else the "guard" guards nothing. A red is signal — surface it, don't work around it Calibration exists to right-size assertions, not to make a case pass. When a run turns a scenario or expectation red, classify the red first — then keep it. First rule out the environment. Before reading any red as a signal about a case, check the shape of the failures across the run. If every case fails the same way — every scenario with the same execution error while builds succeed, or every build erroring identically before it starts (an `Agent error: