debug

name debug description Investigate stuck runs and execution failures by tracing Symphony and Codex logs with issue/session identifiers; use when runs stall, retry repeatedly, or fail unexpectedly. Debug Goals Find why a run is stuck, retrying, or failing. Correlate Linear issue identity to a Codex session quickly. Read the right logs in the right order to isolate root cause. Log Sources Primary runtime log: log/symphony.log Default comes from SymphonyElixir.LogFile ( log/symphony.log ). Includes orchestrator, agent runner, and Codex app-server lifecycle logs. Rotated runtime logs: log/symphony.log* Check these when the relevant run is older. Correlation Keys issue_identifier : human ticket key (example: MT-625 ) issue_id : Linear UUID (stable internal ID) session_id : Codex thread-turn pair ( <thread_id>-<turn_id> ) elixir/docs/logging.md requires these fields for issue/session lifecycle logs. Use them as your join keys during debugging. Quick Triage (Stuck Run) Confirm scheduler/worker symptoms for the ticket. Find recent lines for the ticket ( issue_identifier first). Extract session_id from matching lines. Trace that session_id across start, stream, completion/failure, and stall handling logs. Decide class of failure: timeout/stall, app-server startup failure, turn failure, or orchestrator retry loop. Commands # 1) Narrow by ticket key (fastest entry point) rg -n "issue_identifier=MT-625" log /symphony.log* # 2) If needed, narrow by Linear UUID rg -n "issue_id=<linear-uuid>" log /symphony.log* # 3) Pull session IDs seen for that ticket rg -o "session_id=[^ ;]+" log /symphony.log* | sort -u # 4) Trace one session end-to-end rg -n "session_id=<thread>-<turn>" log /symphony.log* # 5) Focus on stuck/retry signals rg -n "Issue stalled|scheduling retry|turn_timeout|turn_failed|Codex session failed|Codex session ended with error" log /symphony.log* Investigation Flow Locate the ticket slice: Search by issue_identifier=<KEY> . If noise is high, add issue_id=<UUID> . Establish timeline: Identify first Codex session started ... session_id=... . Follow with Codex session completed , ended with error , or worker exit lines. Classify the problem: Stall loop: Issue stalled ... restarting with backoff . App-server startup: Codex session failed ... . Turn execution failure: turn_failed , turn_cancelled , turn_timeout , or ended with error . Worker crash: Agent task exited ... reason=... . Validate scope: Check whether failures are isolated to one issue/session or repeating across multiple tickets. Capture evidence: Save key log lines with timestamps, issue_identifier , issue_id , and session_id . Record probable root cause and the exact failing stage. Reading Codex Session Logs In Symphony, Codex session diagnostics are emitted into log/symphony.log and keyed by session_id . Read them as a lifecycle: Codex session started ... session_id=... Session stream/lifecycle events for the same session_id Terminal event: Codex session completed ... , or Codex session ended with error ... , or Issue stalled ... restarting with backoff For one specific session investigation, keep the trace narrow: Capture one session_id for the ticket. Build a timestamped slice for only that session: rg -n "session_id=<thread>-<turn>" log/symphony.log* Mark the exact failing stage: Startup failure before stream events ( Codex session failed ... ). Turn/runtime failure after stream events ( turn_* / ended with error ). Stall recovery ( Issue stalled ... restarting with backoff ). Pair findings with issue_identifier and issue_id from nearby lines to confirm you are not mixing concurrent retries. Always pair session findings with issue_identifier / issue_id to avoid mixing concurrent runs. Notes Prefer rg over grep for speed on large logs. Check rotated logs ( log/symphony.log* ) before concluding data is missing. If required context fields are missing in new log statements, align with elixir/docs/logging.md conventions.