openclaw-debugging

name openclaw-debugging description Debug OpenClaw model, provider, tool-surface, code-mode, streaming, and live/Crabbox behavior by choosing the right logs, probes, and proof path before changing code, including fetching stored sessions, transcripts, and attachments as evidence. OpenClaw Debugging Use this skill when OpenClaw behavior differs between local tests, live models, providers, code mode, Tool Search, Crabbox, or CI, and the next move should be a debug signal rather than a guess. Read First docs/logging.md for log files, openclaw logs , and targeted debug flags. docs/reference/test.md for local test commands. docs/tools/code-mode.md for code-mode exec/wait and tool catalog rules. Use $openclaw-testing for choosing test lanes. Use $crabbox for broad, Docker, package, Linux, live-key, or CI-parity proof. Default Loop State the suspected boundary: config, tool construction, provider payload, fetch, stream/SSE, transcript replay, worker/runtime, package/dist, or CI. Add or enable the narrowest signal that proves that boundary. Reproduce with the same provider/model/config. Do not randomly switch models unless the model itself is the variable being tested. Compare configured state with actual run activation. Patch the root cause. Rerun the exact failing probe, then broaden only if the contract requires it. Model Transport Logs Use targeted env flags instead of global debug when the model request shape or stream timing matters: OPENCLAW_DEBUG_MODEL_TRANSPORT=1 openclaw gateway OPENCLAW_DEBUG_MODEL_PAYLOAD=tools OPENCLAW_DEBUG_SSE=events openclaw gateway OPENCLAW_DEBUG_MODEL_PAYLOAD=full-redacted OPENCLAW_DEBUG_SSE=peek openclaw gateway Useful flags: OPENCLAW_DEBUG_MODEL_TRANSPORT=1 : request start, fetch response, SDK headers, first SSE event, stream done, and transport errors at info . OPENCLAW_DEBUG_MODEL_PAYLOAD=summary : bounded payload summary. OPENCLAW_DEBUG_MODEL_PAYLOAD=tools : all model-facing tool names. OPENCLAW_DEBUG_MODEL_PAYLOAD=full-redacted : capped, redacted JSON payload. Use only while debugging; prompts/message text may still appear. OPENCLAW_DEBUG_SSE=events : first-event and stream-completion timing. OPENCLAW_DEBUG_SSE=peek : first five redacted SSE events. OPENCLAW_DEBUG_CODE_MODE=1 : code-mode tool-surface diagnostics. Watch logs with: openclaw logs --follow Common Boundaries Config vs activation: config can be enabled while the run disables tools, is raw, has an empty allowlist, or lacks model tool support. Check the actual visible tools before enforcing provider payload invariants. Tool surface: inspect final model-visible tool names, not only the tool registry or config. Code mode means exactly exec and wait only after it actually activates. Provider payload: log fields, model id, service tier, reasoning, input size, metadata keys, prompt-cache key presence, and tool names before SDK call. Fetch vs SSE: fetch response proves HTTP headers arrived; first SSE event proves provider body progress. A gap here is a stream/body/provider issue, not tool execution. Worker/dist: run pnpm build when touching workers, dynamic imports, package exports, lazy runtime boundaries, or published paths. Live keys: use the configured secret workflow for missing provider keys before saying live proof is blocked. Env checks are presence-only; never print secrets. Fetching Sessions and Transcripts Use these paths when a bug report references a chat session and you need the actual transcript, sender attribution, or attachments as evidence. CLI first (needs a configured install; safe against a live gateway): openclaw sessions list --agent <agentId> --json openclaw sessions tail openclaw sessions export-trajectory Docs: docs/reference/database-schemas.md for the store layout, https://docs.openclaw.ai/cli/sessions for the CLI. Raw store (when the CLI is unavailable, e.g. inspecting a remote host over SSH, or you need event-level detail): Per-agent data plane: ~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite . Canonical schema: src/state/openclaw-agent-schema.sql . Hosted/systemd installs keep state under the service user's home (e.g. /home/openclaw/.openclaw/... ), not root's — root may carry a separate stray install with different agents. If the expected agent dir is missing, locate the real DB: find / -maxdepth 6 -name openclaw-agent.sqlite . Web chat URLs end in a session-id fragment: /chat/<agentId>/<slug>-<hex> . Resolve it in session_nodes : session_key LIKE '%<hex>%' → current_session_id , display_name . Key shape is agent:<agentId>:<surface>:<uuid> ; subagent sessions use surface subagent . Transcript: transcript_events ( session_id , seq , event_json ). event_json.message has role ( user / assistant / toolResult ) and content (string, or parts of type text / toolCall / image ). Sender provenance: real user messages carry message.__openclaw ( senderId , senderName , senderIsOwner ); runtime-synthesized inputs do not. Use this to separate operator-authored text from injected prompts. Full-text search across transcripts: session_transcript_fts . Attachments: media://inbound/<file> URLs map to <state-dir>/media/inbound/<file> . User-attached images are not image content parts. They ride the message envelope: message.__openclaw.media[] entries with url ( media://inbound/<file> ), contentType , kind , fileName . A parts-only extractor misses every image — grep raw event_json for media:// . On a remote host, scp the files locally (one remote path per scp argument) and read them there. Hosts without a sqlite3 binary still have Node: node:sqlite needs no dependencies. node -e 'const {DatabaseSync}=require("node:sqlite"); const db=new DatabaseSync(process.argv[1],{readOnly:true}); console.log(JSON.stringify(db.prepare( "SELECT seq,event_json FROM transcript_events WHERE session_id=? ORDER BY seq" ).all(process.argv[2])))' <db-path> <session-id> Always open live stores readOnly: true ; never write a running gateway's state (see Validation rules in the root AGENTS.md ). For realistic-data work, copy the DB into a dev state dir first. Code Pointers Model payload + Responses stream: src/agents/openai-transport-stream.ts Guarded fetch/timing: src/agents/provider-transport-fetch.ts OpenAI/Codex provider wrappers: src/llm/providers/stream-wrappers/openai.ts Tool construction, Tool Search, code-mode activation: src/agents/embedded-agent-runner/run/attempt.ts Code-mode runtime and worker: src/agents/code-mode.ts src/agents/code-mode.worker.ts Tool Search catalog: src/agents/tool-search.ts Proof Choice Single helper/payload bug: local targeted Vitest. Docs/logging-only: pnpm check:docs and git diff --check . Worker/dist/lazy import/package surface: targeted tests plus pnpm build . Live provider/model behavior: same provider/model with debug flags and a real key if available. Docker/package/Linux/CI-parity: current dedicated Linux worker when capable; otherwise $crabbox . CI failure: exact SHA, relevant job only, logs only after failure/completion. Output Habit Report: boundary tested exact command/env shape, redacted observed signal, such as tool names or first SSE event timing fix location narrow proof and any remaining risk