Overview
Pilot, a rewrite of my original voice-driven computer-use agent, operates a Mac the way a careful person would: it reads the screen through the Accessibility API and screenshots, acts through real UI elements instead of guessing pixels, and checks its work against ground truth before calling a task done. It runs on Claude through the official Claude Code CLI, or on any Anthropic or OpenAI-compatible endpoint (Gemini, OpenRouter, Ollama, LM Studio).
Key Features
- Planner and executor: Opus 5.5 plans only when needed while Sonnet 5.5 executes, batching actions and seeing a fresh screen each turn
- No pixel guessing: every observation lists real UI elements, pressed or filled by id through AXPress and AXValue
- Reflex layer: screen-settle detection, waiting for editable fields, loop notes and a disk check before "done", all without a model call
- Self-tuning:
pilot tune calibratemeasures a model's grounding, image budget and effort, then writes a profile citing each measurement;ablateandauditprune scaffolding a model no longer needs - Safety as an invariant: permission only from your words, approval before risky actions, never answers macOS security dialogs, and a corner-of-the-screen kill switch
Results
On PilotBench (15 real macOS tasks), the default configuration passes 15/15 with a 12.9 s median task and no false "done" or safety failures: the same success as an Opus executor at 28% less time and 36% less cost. Over six rounds of harness work, total time on the suite fell from 702 s to 229 s and model calls from 141 to 68.