undermind-search

name undermind-search description Stage 1 of the lit review pipeline: run an Undermind deep search from the natural-language brief produced by Stage 0 through the Undermind MCP server (agent-driven; no browser, no stored credentials), then parse + enrich the returned papers into the pipeline schema. Only use this skill when explicitly requested. Do NOT auto-trigger on general literature review or paper search requests. Undermind Search (Stage 1) Takes the Undermind brief that Stage 0 extracts and returns the papers Undermind's deep search finds, enriched and saved as <stem>.json + <stem>.bib for dedup and screening. The stage is agent-driven end to end. There is no subprocess driver: the host agent calls the Undermind MCP tools, saves what they return, and one script normalizes it. The agent — launch_deep_search → poll inspect_deep_searches → page the results. scripts/undermind_harvest.py — copies the paged tool results out of the session transcript byte-for-byte into OUT/undermind_page_<k>.txt (Claude Code ~/.claude/projects/<slug>/<session>.jsonl , following its <persisted-output> pointers; Codex ~/.codex/sessions/**/rollout-*.jsonl ). scripts/undermind_ingest.py — parses those files (or any RIS/BibTeX export), fills missing DOIs (Crossref) and abstracts/journals (OpenAlex), and writes the pipeline JSON. Importable, and runnable standalone. Prerequisite: the Undermind MCP server Undermind's official MCP endpoint is https://mcp.undermind.ai/mcp (OAuth, your own Undermind account). Claude Code: claude mcp add --transport http undermind https://mcp.undermind.ai/mcp then claude mcp login undermind in an interactive terminal. A session that was already open when you logged in only exposes mcp__undermind__authenticate ; call it once, approve in the browser, and the full tool set swaps in. Codex: in ~/.codex/config.toml : [mcp_servers.undermind] url = "https://mcp.undermind.ai/mcp" [mcp_servers.undermind.oauth] client_id = "codex" Codex completes the OAuth flow on first use. Call get_orientation once per conversation before the other tools. The flow brief ─► create_workspace (or reuse) ─► launch_deep_search(goal=brief) ─► poll inspect_deep_searches(status_only) every ~60 s (2–5 min) ─► inspect_deep_searches(papers_only, detail_level="full", limit=50, offset=0/50/…) ─► undermind_harvest.py --latest --match <name> --out-dir OUT (pages → OUT/undermind_page_<k>.txt) ─► undermind_ingest.py --input OUT/undermind_page_*.txt -o OUT/stage1_undermind.json Step by step, with OUT the run's output directory: list_workspaces → reuse a workspace for this manuscript, or create_workspace(title=<document stem>) . Keep the workspace_id . launch_deep_search(workspace_id, goal=<contents of OUT/undermind_brief.txt>, name=<short label>) . The tool returns immediately. Launch it first , then start the other channels while it runs. The brief goes in verbatim: the goal is meant to be a self-contained paragraph, which is exactly what Stage 0 writes (no keyword lists, no journal names). The MCP search asks no clarifying questions; the plan's undermind_clarifications field is informational only (a holdover from the browser flow). Poll inspect_deep_searches(names=[<name>], status_only=true) from a background Monitor, not a foreground wait. A rate_limited result means wait and retry; Undermind runs only a few deep searches concurrently. On completion, page the papers: inspect_deep_searches(names=[<name>], papers_only=true, detail_level="full", limit=50, offset=0) , then offset=50 , 100 , … until a page comes back short. Do not retype or summarize anything. Harvest the pages from the transcript, then ingest: python undermind-search/scripts/undermind_harvest.py --latest --match "<name>" --out-dir OUT python undermind-search/scripts/undermind_ingest.py --input OUT/undermind_page_*.txt \ -o OUT/stage1_undermind.json The harvester scans the newest transcript under the Claude Code projects dir ( CLAUDE_CONFIG_DIR or ~/.claude ) and ~/.codex/sessions , keeps every distinct tool result that contains [Key] Title (Year) blocks and whose header (the search-name line above the paper list) contains the --match text, and writes them as undermind_page_01.txt , _02 , …; without --match , when the transcript holds several searches, only the latest search's pages are kept and the others are named. Pages whose showing a-b of n ranges leave a gap raise a WARNING naming the missing offsets. ( --transcript PATH to name the file; --min-records N to drop short fragments). Falling back to saving a page by hand as .txt or as the {"result": "..."} JSON also works: the ingest reads both. Optional add-ons before ingest, for the same workspace: search_papers with search_type="citations" / "references" seeded on the top cite keys widens coverage; save its output as another page file and ingest it together. Input formats the ingest accepts Auto-detected from content: Input Notes Raw MCP tool text [Key] Title (Year) blocks from inspect_deep_searches , get_paper_info , or search_papers at any detail level. Journal, Date: , By … , DOI: , Link: , PDF , and the abstract paragraph are read; trailer legends are ignored. {"result": "…"} JSON The tool result saved as JSON. JSON array [{"title", "authors" (string or list), "year", "journal", "doi", "url", "abstract"}] ; only title is required. .ris / .bib The web app's Export, or any reference manager. Several --input files are concatenated and deduplicated by cite key, DOI, or normalized title, so paginated dumps just go in together. CLI undermind_harvest.py : Flag Default Description --latest (default) Scan the most recently modified transcript under the Claude Code / Codex session dirs --transcript PATH — Scan this transcript instead --match TEXT — Keep only pages whose header (search-name line) contains this text; omitted → the latest search in the transcript --min-records N 1 Skip pages with fewer paper blocks --out-dir DIR . Where undermind_page_<k>.txt is written --selftest off Run the harvester self-check undermind_ingest.py : Flag Default Description --input, -i PATH… — One or more input files (globs expanded internally; PowerShell-safe) -o, --output PATH stage1_undermind.json Output JSON (a .bib sibling is written) --sibling {bibtex,ris,none} bibtex Reference-manager copy beside the JSON --source TAG undermind source value stamped on each record --no-enrich off Skip Crossref/OpenAlex enrichment --selftest off Run the parser self-check Graceful degradation If the MCP tools are unavailable (server not added, not authenticated, or a Codex/Claude session without MCP), the agent prints UNDERMIND_DEFERRED , writes [] to stage1_undermind.json , and continues; the ingest does the same when no input file exists or nothing parses. The brief is always available in undermind_brief.txt for a manual run in the web app (Export → RIS, then ingest the file). The autonomous orchestrator.py never runs this stage: it has no MCP client. Output schema Same as the other stages, with source: "undermind" : { "title" : "..." , "authors" : "A, B" , "year" : "2024" , "doi" : "https://doi.org/10.x/y" , "abstract" : "..." , "journal" : "..." , "url" : "..." , "source" : "undermind" , "verified" : true , "citations" : 0 , "open_access" : false } Troubleshooting Only mcp__undermind__authenticate is available — the session predates the login. Call it, approve in the browser, and the tools appear. claude mcp list says "Needs authentication" although login worked — a stale mcp-needs-auth-cache.json in the Claude config dir; harmless to delete. rate_limited — Undermind caps concurrent deep searches; wait and retry. The harvester finds no pages — the tool results have not been flushed to the transcript yet, or the newest transcript belongs to another session; wait a moment and retry, or pass --transcript with the session file. --match is case-insensitive and is tested against the page header (the search-name line), never against paper text. A page parses to zero records — the tool output format drifted. Run --selftest , compare the saved page with the header pattern in parse_mcp_text , and adjust; the JSON-array input is the format-independent fallback.