Computer-use

A Python agent that listens to speech, looks at the screen, and drives the mouse and keyboard to finish multi-step desktop tasks without a click from me. It runs on a local loop: capture, decide, act, verify, repeat - and the interesting engineering is entirely in the verify step.

Brief

The problem

Voice control for a desktop is an old idea with a bad reputation, because most implementations are command recognisers: a fixed grammar, a list of verbs, and a wall the moment you say something the author did not anticipate. What I wanted was not recognition but competence - say "clean up my downloads folder and open the invoice", and have something work out the steps.

Multimodal models made that plausible, but they made a specific new problem unavoidable. A model that is handed a screenshot and asked "what should I click?" will happily answer, and be wrong, and then be confidently wrong about having been wrong. The screen is a lossy, low-resolution, constantly-changing representation of a UI whose real state lives in an application's memory. Every design decision in this project is an attempt to stop that gap from compounding.

The agent is not limited by what it can do. It is limited by how well it can tell whether it already did it.

How I got there

Attempt one: pure vision. Send the screenshot, let the model return a click target, click it. This worked in exactly the setting it was demoed in and failed everywhere else. Two reasons. First, the model was reading a downscaled image and returning pixel coordinates in whatever space it had been thinking in, so anything smaller than a large button was a coin flip. Second, and worse, there was no notion of state: after a click, the next screenshot looked slightly different, and the model had no way to know whether it was looking at the result of its own action or a fresh problem. It would re-issue the same click, on a menu that was now open, and take down the whole session.

Attempt two: an accessibility-tree agent. I wired up the OS accessibility APIs and gave the model a structured list of elements instead of pixels. Much more reliable targeting - and then it fell apart on the half of the desktop I actually use. Terminals, browsers with canvas content, Electron apps, and anything drawing its own widgets expose either nothing or a tree so deep it cost more tokens than the screenshot did.

What shipped is a hybrid: screenshot plus an OCR layer, with the model choosing between a small set of parameterised primitives rather than free-form coordinates, and an explicit post-action verification step in every cycle.

The loop

One cycle, start to finish: capture the display; run OCR over it; assemble a perception packet (the frame, the element list, the last few actions, and the task as restated); send it to the model; receive a structured intent; sanitise it; execute it; capture again; verify. Then either continue, retry with a different strategy, or stop.

The packet is the part that decides quality. Sending a bare screenshot with "here is what happened, what next" produces an agent with no memory of its own intentions, so the packet carries the previous three actions with their verification results, plus the current task decomposition. That is a small amount of context and it removes a disproportionate number of mistakes, because most of the mistakes were the agent re-doing something it had just done.

Two counters bound the loop. A step counter caps total actions per task. A no-progress counter increments when a verification fails or when the post-action screen is unchanged from the pre-action screen, and aborts at three. The distinction matters: an action that fails loudly is fine, an action that appears to succeed while nothing changes is the one that loops forever.

Perception

Each cycle grabs the display and runs Tesseract over it in parallel with the frame going to the model. The OCR pass exists because language models are unreliable at reading small UI text under time pressure and very reliable at reading it when the text has already been handed to them as a string with a bounding box. So the OCR output comes back as a menu of labelled targets - string, position, confidence - and the model picks from the list instead of inventing coordinates. When OCR finds nothing useful, the model can still fall back to raw pixel targeting, which is the accessibility-tree lesson kept as an escape hatch rather than a primary path.

Speech is handled by the Live API rather than a separate transcription stage, which removes one round trip and one source of error. Barge-in matters more than it sounds like it should: an agent that cannot be interrupted while it is talking is an agent you do not trust with your cursor.

Action

The action space is deliberately narrow: click, double-click, type, key combination, scroll, drag, wait, and done. Not because the model cannot imagine more, but because a narrow parameterised set is auditable and reversible. Every action is logged as a structured record with the perception state that justified it, which is the only reason debugging this project was ever possible.

Type is the dangerous one. Free-text injection into a focused field can produce anything, so typing goes through a sanitiser that refuses to emit shell metacharacters into a terminal, and destructive confirmations are never auto-accepted. There is a hard cap on actions per task; when it is hit, the agent stops and reports rather than continuing to guess.

Verification

This is the part I would defend in an interview. After each action the agent captures again and asks a narrow question - not "did that work?" but "does the element I expected to appear exist now, and is it in the position I expected?" That is answerable from the OCR layer with reasonable precision, and it converts a vibes judgement into a boolean.

Failures are handled by class rather than by retry count. A click that produced no change usually means a stale frame, so: wait and re-capture. A click that opened the wrong thing means a misread target, so: escape and re-perceive. Three consecutive no-progress cycles abort the task with the transcript of what it tried. The alternative - letting the loop keep going - is how you end up with an agent that has opened forty-seven copies of the same window.

Latency

The budget per cycle is roughly: capture 20 - 40 ms, OCR 150 - 400 ms depending on screen density, model round trip the rest. On a full-resolution display, OCR was the second-largest cost after the model, and it is the one that could be cut, because a fast model call that has to guess at small text costs more in retried cycles than a slow OCR that hands it the truth.

The practical consequence is that a three-step task takes ten to fifteen seconds and a ten-step task takes the better part of a minute, which changes what the interface has to be. Silence is indistinguishable from failure, so the agent narrates: what it can see, what it is about to do, and whether the last thing worked. Narration is not a flourish; it is the feedback channel that lets me interrupt a wrong plan at step two instead of discovering it at step nine.

Safety

Anything that can move your cursor can delete your files. Three limits are non-negotiable. A kill switch: a held key combination that aborts immediately and releases the input devices, checked in the loop rather than in an event handler, because an event handler does not run while the loop is blocked. An allowlist of applications the agent may act inside, defaulting to deny. And a hard refusal set - it will not type into a password field, will not confirm a purchase, will not touch anything the OCR layer classifies as a destructive dialog.

None of this is sophisticated, and all of it is load-bearing. The interesting safety property of a computer-use agent is not alignment, it is blast radius.

Results

The honest statement about results is that this project taught me more about agent loops than about voice interfaces. It is a research tool that I use daily, not a product, and the failure mode I could never design away is the one where the model faithfully completes a task I described badly. That is not a bug you can fix in code; it is the price of giving a system authority over your screen.

What I'd change

Ground the target selection in a proper UI-element model rather than OCR plus heuristics; that is where the field has gone since, and the accessibility-tree dead end in attempt two was directionally right - it was the coverage that was wrong, not the idea.

Add a dry-run mode that prints the planned action sequence and waits for confirmation, so trust can be granted incrementally rather than all at once. Make the loop's state explicit: the cycle counter and the abort logic are doing work that a real state machine would do far more legibly, and half the debugging time went on the implicit version. And version the prompt packets, because changing one line of the perception template silently changes the success rate of every task in the library, and I learned that the hard way.

Credits

• Google DeepMind - the Gemini Multimodal Live API
• The Tesseract OCR project, and its page-segmentation modes
• The Open Accessibility specification, which made attempt two possible and informative
• Anthropic's computer-use tool documentation, for the shape of a sane action space
• PyAutoGUI, which is the least glamorous and most essential dependency here