recursive-improve

name recursive-improve description End-to-end agent improvement pipeline. Analyzes raw execution traces, extracts insights, manages a skillbook, gathers domain context, defines metrics, builds a rubric, creates a prioritized action plan, presents it for review, and implements approved fixes. Trigger when the user says "improve my agent", "run the improvement pipeline", "apply insights", "/recursive-improve", or when eval/traces/ contains trace files. recursive-improve: Agent Improvement Pipeline End-to-end pipeline: trace analysis → skill extraction → domain context → metrics → rubric → action plan → review → fixes. Prerequisites Traces must exist in eval/traces/ . If they don't: Ask the user for their traces directory Copy .json , .md , and .toon files into eval/traces/ Skip condition: If eval/stage1_insights_summary.md already exists (from a prior run or from recursive-improve analyze ), skip Stages 0 and 1 — go directly to Stage 2. Stage 0: Trace Analysis Analyze raw execution traces to extract learnings. This stage adapts ACE's recursive reflector methodology — a structured 6-phase strategy that moves from data discovery through verified deep-dives to synthesized, evidence-backed insights. Inputs eval/traces/ — raw trace files ( .json , .md , .toon ) Phase 1: Discover Map the data shape and inventory. Do NOT judge outcomes yet — just catalog what you have. Read 2-3 trace files. Identify: Top-level keys and message schema (3 levels deep) Message format: role , content , tool_calls , turn_idx , etc. Total trace count and per-trace message counts Search for agent operating rules, policy, or instructions embedded in the traces — these are often in large strings (>500 chars). Check: role: "system" messages info.environment_info.policy or similar fields Large embedded strings in any field Build an inventory table: File Messages Has system prompt? Has tool calls? trace_001.json 42 yes yes trace_002.json 18 yes no ... Record discovered rules/policy verbatim — understanding what the agent was supposed to do is essential for evaluating what it actually did. Phase 2: Derive Evaluation Criteria Based on your discovery (schema, rules, patterns), define specific evaluation criteria to apply to every trace during the survey phase. For each criterion, state: What to look for What a violation looks like Example criteria (adapt to what you discovered): "Agent must verify customer identity before account changes" → violation: account change without prior verification tool call "Agent must not hallucinate policy details" → violation: agent states a policy that contradicts the embedded rules Phase 3: Survey Read ALL traces (if ≤ 20) or a stratified sample (if > 20, target ~15 or 30%, whichever is larger — sample by outcome, length, and complexity). For each trace, record: What was requested What the agent did (key decisions, tool calls, reasoning) How it ended (success / failure / partial) Evaluation criteria results (pass / fail / not applicable per criterion) Process in batches of ~3 traces at a time to manage context. Phase 4: Categorize Review all survey summaries. Group by task type and outcome. Select 2-3 deep-dive targets , prioritizing: Divergent outcomes — same task type, one succeeded, one failed. What made the difference? Longest/most complex traces with mistakes — most decision points, most learning potential Most common failure pattern — highest impact to fix Confident-but-wrong — traces where the agent's stated reasoning seems worth cross-checking against the data it received Rule/criteria violations that appeared across traces (even successful ones) Group targets by root cause — max 2 deep-dives per root cause. Prioritize breadth over depth . Skip short, simple, routine traces — they rarely yield learnings. Phase 5: Deep-dive For each deep-dive target, re-read the FULL raw trace — not your survey summary. Deep-dives that analyze summaries of summaries produce shallow, unverified conclusions. Two passes per target: Pass 1 — Verification: Separate what the agent claimed from what data it received . For each key claim or conclusion: What did the agent claim? What does the actual data/tool response show? Does it comply with the discovered rules? List any incorrect claims : what was claimed, what data shows, impact This catches "confident but wrong" errors — where the agent proceeds without hesitation based on incorrect reasoning — that behavioral analysis alone misses. Pass 2 — Root cause analysis: Given the verification findings and the full trace: What should the agent do differently? What is the root cause (not just the symptom)? Is this a missing instruction, a wrong instruction, a code limitation, or a reasoning failure? For divergent outcomes : compare success and failure traces side by side. What specifically made the difference? Phase 6: Synthesize Combine ALL survey summaries with ALL deep-dive results. Do not omit deep-dive findings — they contain your best evidence. Produce a list of atomic learnings . For each: Learning : one specific, actionable insight (one concept only) Atomicity score (0.0–1.0): base 1.0, deduct 0.15 per "and/also/plus", 0.20 per vague term, 0.05 per word over 15 Evidence : cite specific trace details (file name, message index, exact data) Severity : high (directly causes wrong outcomes), medium (degrades quality), low (minor inefficiency) Category : code_fix | prompt_fix | process_fix Verification findings are high-severity — when the agent's reasoning contradicted the data it received, this directly causes wrong outcomes regardless of correct procedure. Output Write to eval/stage0_trace_analysis.md : # Trace Analysis ## Discovery ### Trace Format ### Schema ### Agent Rules ### Inventory ## Evaluation Criteria 1. [criterion]: [violation description] ... ## Survey ### [trace _file.json] - Requested: ... - Agent did: ... - Outcome: success/failure/partial - Criteria: ... ## Categories ### Success patterns ### Failure patterns ### Partial completions ## Deep-dive Targets ### [Target 1: description] #### Verification findings #### Root cause analysis ### [Target 2: description] ... ## Extracted Learnings | # | Learning | Atomicity | Evidence | Severity | Category | |---|----------|-----------|----------|----------|----------| | 1 | ... | 0.92 | ... | high | prompt_ fix | | 2 | ... | 0.87 | ... | medium | code _fix | Stage 1: Skill Management Transform raw learnings from Stage 0 into a structured skillbook with quality gates. Inputs eval/stage0_trace_analysis.md — extracted learnings from Stage 0 eval/skillbook.json — existing skillbook from a prior improvement cycle (if it exists, load and update it; if not, start fresh) Step 1: Quality gate — Atomicity For each learning from Stage 0, verify the atomicity score: Score Level Action 0.95–1.00 Excellent Accept as-is 0.85–0.94 Good Accept, minor tightening optional 0.70–0.84 Fair Split into multiple atomic learnings 0.40–0.69 Poor Must split before proceeding < 0.40 Rejected Discard — too vague or compound Splitting example: Compound: "Tool X worked in 4 steps with 95% accuracy" (0.55) Split into: "Use Tool X for task type Y" (0.95) + "Tool X completes in ~4 steps" (0.92) + "Expect 95% accuracy from Tool X" (0.90) Step 2: Format as imperative commands Every skill must be an imperative command , not an observation. BAD: "The agent accurately answers factual questions" (observation) GOOD: "Answer factual questions directly and concisely" (imperative) BAD: "Missing verification step caused errors" (observation) GOOD: "Verify customer identity before making account changes" (imperative) Step 3: Deduplication If eval/skillbook.json exists, load it. For each new learning, check whether any existing skill has >70% semantic overlap. Semantic duplicates (use UPDATE, not ADD): Existing skill Duplicate (don't add) "Answer directly" "Use direct answers" "Break into steps" "Decompose into parts" "Verify calculations" "Double-check results" Step 4: Determine operations For each learning, select the operation: Situation Operation New error pattern or missing capability ADD new skill Existing skill needs refinement UPDATE with improved content Existing skill contributed to success in traces TAG as helpful Existing skill caused or contributed to error TAG as harmful Strategies contradict each other REMOVE one or UPDATE to resolve Skill tagged harmful 3+ times REMOVE No actionable insight SKIP Default to UPDATE over ADD when a similar skill exists. Step 5: Rejection filter Reject any skill that contains: Meta-commentary (not actionable): "be careful", "consider", "think about", "remember", "make sure" Observations (not commands): "the agent", "the model" — write commands to follow, not descriptions of behavior Vague terms : "appropriate", "proper", "various" — too vague to act on Overgeneralizations : "always", "never" without specific context Step 6: Skillbook size management If the skillbook exceeds 50 skills: Prioritize UPDATE over ADD Merge skills with >70% overlap Remove lowest-performing skills (most harmful tags, least helpful tags) Outputs eval/skillbook.json : { "skills" : { "section-00001" : { "id" : "section-00001" , "section" : "error_handling" , "content" : "Verify customer identity before making account changes" , "evidence" : "In trace_003.json, agent changed account without verification (msg 12)" , "justification" : "Prevents unauthorized account modifications" , "helpful" : 0 , "harmful" : 0 , "status" : "active" } } , "sections" : { "error_handling" : [ "section-00001" ] } , "next_id" : 2 } eval/stage1_insights_summary.md : # Insights Summary Generated by: recursive-improve (Stage 1) Total insights: N --- ## Insight: {skill _id} — {section} **Status:** active **Helpful/Harmful:** 0/0 **Content:** {imperative skill text} **Evidence:** {specific trace evidence} **Justification:** {why this improves the agent} --- Write both files, then proceed to Stage 2. Stage 2: Domain Context Gathering Understand the agent's world — what it does, what tools it has, and what "success" looks like. 0. Detect trace format Read 1 trace file from eval/traces/ and identify the framework: Signal Framework info.agent_info.implementation , simulation.messages[] with role / tool_calls / turn_idx tau2-bench runs[].steps[] with type: "tool" , lc_kwargs LangChain / LangSmith events[] with event_type , span_id , parent_id LlamaIndex choices[].message.tool_calls[] at top level Raw OpenAI API logs trace.spans[] with attributes , trace_id OpenTelemetry / Arize / Langfuse Record the detected format. If unrecognized, note top-level keys and proceed best-effort. 1. Detect architecture Read 2-3 traces. Determine single-agent vs multi-agent: Single agent : one conversation thread, tool calls from one identity Multi-agent : multiple agent_info entries, routing tool calls ( transfer_to_* , delegate_to_* ), distinct system prompts per agent If multi-agent: document each agent separately and note routing logic. 2. Find the system prompt Fallback chain — stop at first hit: Config files — grep for: system_prompt , system_message , instructions , AGENT_INSTRUCTION , SYSTEM_PROMPT Source code — search for prompt template strings, f-strings building system messages Trace extraction — check info.environment_info.policy , first role: "system" message, raw_data fields Not found — record SYSTEM_PROMPT_STATUS: NOT_FOUND Record both content and source location. 3. Extract tool definitions Pass 1 — Source code: Search for @tool , @is_tool , function schema arrays, tools=[] . For each: name, params, return type, side effects (READ/WRITE/GENERIC), unvalidated rules. Pass 2 — Traces: Read ALL traces (if ≤ 20) or stratified sample. Extract every unique tool_calls[].name and role: "tool" response. Record one example input/output per tool. Reconcile: Tools in source but not traces = "available but unused". Tools in traces but not source = investigate. 4. Find domain documentation READMEs, policy files, inline comments, test files describing expected behavior. 5. Catalogue behavior patterns Trace selection — stratified sampling (if > 20 traces): 2+ per unique termination_reason Shortest, longest, 2 median by message count Lowest and highest by tool call count 3+ of each pass/fail outcome Target: ~15 traces or 30%, whichever is larger For each trace, document: function call frequency, tool call sequences, success patterns, failure patterns, error patterns, policy violations, user feedback signals.