peekaboo

name peekaboo description Use Peekaboo for macOS desktop automation, screenshots, visual UI maps, native accessibility inspection, app/window/menu/dialog control, native app and browser chrome control, browser-page MCP tooling, MCP diagnostics, and Peekaboo repo validation. Use when Codex needs current macOS UI state, direct desktop control, privacy-aware live smoke tests, or changes to the Peekaboo repository. Peekaboo CLI timing flags accept bare milliseconds or ms / s suffixes, for example 500 , 500ms , 2s , or 1.5s . Coordinate input uses --at x,y ; add --global when a targeted coordinate should remain screen-global. Peekaboo is a macOS automation CLI and agent runtime. Prefer the freshly built repo binary, live help, and canonical docs over copied command references because command surfaces move quickly. Peekaboo 4 uses press for standalone keys and chords, drag --from/--to for drags, see --tree for CLI AX inspection, action for named accessibility actions, and real noun/verb subcommand trees. Use verify for stable predicates instead of fixed sleeps. JSON action results include an effect value; read-only results omit it. See docs/v4-migration.md when converting older invocations. Start Here In repo work, build and use the current-source binary: pnpm run build:cli BIN= " $(swift build --package-path Apps/CLI --show-bin-path) /peekaboo" " $BIN " --version Record the installed PATH binary separately when diagnosing environment drift: command -v peekaboo && peekaboo --version Confirm permissions and current tool surfaces before automation: " $BIN " permissions status --json " $BIN " tools --json " $BIN " learn Find command docs: node scripts/docs-list.mjs Canonical References Live CLI help: peekaboo <command> --help Full agent guide: peekaboo learn Tool catalog: peekaboo tools Command docs in this repo: docs/commands/README.md and docs/commands/*.md Permissions and bridge behavior: docs/permissions.md , docs/bridge-host.md , docs/integrations/subprocess.md Repo rules: AGENTS.md Observation Strategy Use peekaboo see --tree --no-screenshot (CLI) or inspect_ui (MCP) for native macOS AX text, labels, buttons, text fields, control state, and element IDs when a screenshot would add noise. Use peekaboo see for screenshots, visual layout, annotated maps, pixels, colors, screen/menu-bar targets, or cases where AX text is missing or incomplete. Use browser for browser page content, forms, DOM/a11y snapshots, console, network, page screenshots, and performance traces when browser tooling is available. Use native Peekaboo tools for app chrome, browser toolbars, menus, dialogs, permissions, windows, and non-browser apps. Treat element IDs from see or inspect_ui as valid only for the current visible state. After a mutating action, verify from the action result or fetch fresh state. Operating Rules Use peekaboo see --json --path /tmp/<name>.png or peekaboo see --tree --no-screenshot --json before element interactions so you have fresh element IDs and snapshot IDs. Prefer the exact element ID string from the current snapshot for clicks and typing; treat ID shapes as opaque. Use labels when IDs are unavailable and coordinates only as a last resort. Check peekaboo permissions status --json before assuming a capture or control failure is a CLI bug. Use --json when another tool or agent needs to parse results. Respect the user's desktop: avoid destructive app/window actions unless requested. If a command fails because the target UI changed, recapture with see before retrying. see --json element bounds are screen coordinates. Snapshot IDs keep element actions tied to the observed UI. When using see in agent smoke tests, pass an explicit /tmp --path so capture artifacts do not land in a user-visible default location. Prefer --foreground only when an app requires a key window, Space switch, or foreground mouse event. Background delivery is the default when Peekaboo can resolve a target process. Common Workflows # AX-only state when a screenshot would add noise. peekaboo see --tree --no-screenshot --app Calculator --json > /tmp/peekaboo-calc-ax.json # Visual layout plus element IDs and snapshot ID. peekaboo see --app Calculator --path /tmp/calc.png --json > /tmp/calc.json ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts j.dig("data","snapshot_id"); puts JSON.pretty_generate((j.dig("data","ui_elements")||[]).map{|e| e.slice("id","label","identifier","bounds")})' # Click an element discovered in the current snapshot. SNAP=$(ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts j.dig("data","snapshot_id")' ) ELEMENT_ID= "<element-id-from-current-snapshot>" peekaboo click --on " $ELEMENT_ID " --snapshot " $SNAP " --json # Browser page content and DOM-oriented actions belong to browser tooling. peekaboo browser status --json # Browser toolbar, menus, permission prompts, and native app chrome still belong to Peekaboo. peekaboo menu list --app Safari --json # Routine management examples stay read-only by default. peekaboo clipboard get --json peekaboo permissions status --all-sources --json peekaboo app list --include-hidden --include-background --json peekaboo config provider list --json peekaboo agent sessions --json peekaboo press cmd+ shift +t --app Safari --window-id 12345 peekaboo verify --app Safari --window-exists -- timeout 2s --json Input Path Testing Peekaboo has two broad input paths: UIAX/action path: accessibility actions such as AXPress , AXSetValue . Synthetic path: pointer/keyboard events, commonly the CAEvent/CGEvent-style path. Useful overrides: # Confirm command exposes the override. peekaboo click -- help | rg 'input-strategy|actionOnly|synthOnly' # UIAX/action click path from a saved snapshot. peekaboo see --app Calculator --path /tmp/calc.png --json > /tmp/calc.json SNAP=$(ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts j.dig("data","snapshot_id")' ) ELEMENT_ID= "<element-id-from-current-snapshot>" peekaboo click --on " $ELEMENT_ID " --snapshot " $SNAP " --input-strategy actionOnly --json --focus-background # Direct accessibility action; good for proving UIAX independent of pointer events. peekaboo action AXPress --on " $ELEMENT_ID " --snapshot " $SNAP " --json # Synthetic click path; allow focus if you need visible app state to mutate. peekaboo click --on " $ELEMENT_ID " --snapshot " $SNAP " --input-strategy synthOnly --json --foreground # Negative control: coordinates cannot use actionOnly. peekaboo click --at 10,10 --input-strategy actionOnly --json Interpretation: actionOnly success proves live AX re-resolution and action invocation. synthOnly success proves coordinate resolution and event delivery, but verify app state independently. action AXPress is the cleanest UIAX smoke test. Compare with Computer Use or another AX inspector when labels/descriptions differ. Calculator Smoke Test Calculator is a handy fixture because it exposes descriptions and identifiers. BIN= " $(swift build --package-path Apps/CLI --show-bin-path) /peekaboo" " $BIN " permissions status --json > /tmp/peekaboo-skill-refresh-permissions.json " $BIN " tools --json > /tmp/peekaboo-skill-refresh-tools.json " $BIN " see --tree --no-screenshot --app Calculator --json > /tmp/peekaboo-skill-refresh-calc-ax.json " $BIN " see --app Calculator --path /tmp/peekaboo-skill-refresh-calc.png --json -- timeout 10s > /tmp/calc.json ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts JSON.pretty_generate((j.dig("data","ui_elements")||[]).select{|e| ["Clear","AllClear","One","Two","Add","Equals","StandardInputView"].include?(e["identifier"].to_s)}.map{|e| e.slice("id","label","identifier","description","help","bounds")})' SNAP=$(ruby -rjson -e 'j=JSON.parse(File.read("/tmp/calc.json")); puts j.dig("data","snapshot_id")' ) BUTTON_ID= "<button-id-from-current-snapshot>" " $BIN " action AXPress --on " $BUTTON_ID " --snapshot " $SNAP " --json " $BIN " click --on " $BUTTON_ID " --snapshot " $SNAP " --input-strategy actionOnly --json --focus-background Expected current behavior: see --json includes bounds for each ui_elements entry. see --tree --no-screenshot --json and see --json should expose element IDs plus Calculator identifiers such as One , Two , and StandardInputView . Copy the returned ID exactly instead of assuming a prefix or shape. tools --json should include browser , click , inspect_ui , and see . Snapshot-backed UIAX must use the captured app/window, not the frontmost app. Repo Validation node scripts/docs-lint.mjs ruby -e 'h=File.read("skills/peekaboo/SKILL.md").split(/^---\s*$/,3)[1]; keys=h.lines.grep(/^[A-Za-z0-9_-]+:/).map { |line| line.split(":",2).first }; abort("frontmatter keys: #{keys.inspect}") unless keys.sort == ["description","name"]' ! rg -n 'elem_[0-9]+' skills/peekaboo/SKILL.md ! rg -n '^allowed-tools:' skills/peekaboo/SKILL.md pnpm run build:cli BIN= " $(swift build --package-path Apps/CLI --show-bin-path) /peekaboo" ; " $BIN " --version " $BIN " click -- help | rg -- '--foreground|--focus-background|--input-strategy|Opaque element ID' " $BIN " see -- help | rg -- '--json|--annotate|--app|--no-web-focus' " $BIN " see -- help | rg '\-\-tree|\-\-no-screenshot|\-\-app|\-\-json' git diff --check -- skills/peekaboo/SKILL.md docs/agent-skill.md docs/commands/see.md docs/automation.md scripts/docs-lint.mjs Notes: If tests fail with no such module 'Testing' , record it as local toolchain fallout; still run builds/lint/live smoke tests. SwiftPM may warn about Commander identity conflicts; do not chase unless the task is dependency hygiene. Keep live validation artifacts under /tmp/peekaboo-skill-refresh-* and do not commit or publish screenshots or UI dumps. Keep this skill compact. Do not vendor generated command references here; update canonical CLI docs or Commander metadata instead.