Screen Angel

Screen Angel

Part of MOOLLM · This skill's directory

Read and act on a live desktop through the platform accessibility APIs. Ask what is on screen, match widgets with CSS-like selectors, outline them where the user can see, grab images of particular windows and controls, and watch focus change.

The 1990s desktop assistant had every part of this except the one that mattered. It could enumerate the widgets and could not tell what they were for. That part is you. This skill is the socket where a recognizer plugs into a desktop.

The resolution pyramid

Four files, same subject, increasing detail. Read down only as far as you need.

Beyond the skill, in the app itself: PROTOCOL.yml is the wire format both implementations must agree on, and SCREEN-ANGEL.yml is the layer's own design document.

Nothing below requires MOOLLM. This file carries the whole protocol in its body. The CLI is standard-library Python. The only real dependency is a running Screen Angel.

Quick start

cd apps/screen-angel

./cli/screen-angel info                 # can I see anything at all?
./cli/screen-angel windows              # what is open?
./cli/screen-angel query 'button'       # what buttons are in the focused app?
./cli/screen-angel show 'button'        # ...and outline them on the real screen
./cli/screen-angel say "I see 7 buttons"

No install, no virtualenv, no pip. If info reports a missing permission, stop and fix that first: every query will otherwise return empty rather than failing, which looks exactly like a broken selector.

The model

One Electron app binds the platform accessibility APIs — AXUIElement on macOS, UI Automation on Windows — and listens on a local socket. Everything else is a client.

   your agent  ──┐
   CLI         ──┼─→  local socket  ─→  Electron main  ─→  native addon  ─→  the OS
   MCP server  ──┘                            │
                                              └─→  transparent overlay window

Three properties of that shape matter to you:

Queries see the focused application only. There is no way to walk a background app's tree. That is a property of the platform APIs, not of this interface. Use windows to see what else exists, and ask the user to switch.

The overlay is ours, not the application's. Anything you highlight or say is drawn on a transparent window floating above everything. It is honest about being an addition. It never modifies the application underneath.

Nothing types or clicks. See What this cannot do. Do not tell a user you pressed something.

Selectors

A deliberate subset of CSS. If you know CSS you already know this, with two omissions that are the interesting part.

Combinators

Union is worth reaching for more than it sounds. The question you usually have is not "where are the buttons" but "what can I interact with here", and that is several roles at once: button, checkbox, combobox, textfield, radiobutton. Each branch is matched independently, and the results come back deduplicated and in walk order, so a union reads exactly like a single-role query — an element two branches both match appears once.

Predicates

Attributes: name, value, role. Values may be bare, 'single', or "double" quoted. Quote anything containing a space.

Pseudo-classes

:focused, :enabled, :disabled

What is deliberately missing

No sibling combinators (+, ~) and no :nth-child. Sibling order is not stable between releases of the application you are pointing at. A selector that depends on position is a selector that breaks on the day the app updates, silently, by matching the wrong thing rather than nothing. Match on what a widget is, not where it sits.

If you find yourself wanting :nth-child, the answer is usually a name or value predicate on the row itself, or query the container and read its children from the returned paths.

Examples

button                              every button
button[name=Save]                   exact name
button[name*=Sav]                   Save, Save As…, Saved
window > toolbar button             buttons directly inside a toolbar inside a window
textfield:focused                   the field with the caret
menuitem[name*=Export]              anything exporty in a menu
row text[value]                     non-empty text cells inside rows
checkbox:disabled                   greyed-out checkboxes

button, checkbox, combobox          anything clickable, in walk order
textfield, checkbox:focused         whichever of the two has the caret
dialog button, dialog checkbox      the controls of a dialog and nothing else

Roles

Platform roles are normalized into one lowercase vocabulary, so a selector written on a Mac has a chance on Windows. AXButton and UIA control type 50000 both arrive as button.

The full vocabulary is 41 names, taken from UI Automation's control types because that set already exists and is documented:

appbar     button      calendar   checkbox  combobox   custom     datagrid  dataitem
document   group       header     headeritem hyperlink image      list      listitem
menu       menubar     menuitem   pane      progressbar radiobutton scrollbar semanticzoom
separator  slider      spinner    splitbutton statusbar tab       tabitem   table
text       textfield   thumb      titlebar  toolbar    tooltip    tree      treeitem
window

Four of these are worth knowing before you guess:

macOS additions that have no UIA equivalent keep their own names, e.g. colorwell, ruler, rulermarker, radiogroup. When in doubt, run tree and read the roles that come back rather than guessing — nativeRole sits beside role on every element for exactly that reason.

UIA's localized control-type names are deliberately unused: they change with the display language, and a selector that only works in English is a selector that fails in Amsterdam.

Commands

Global flags work on either side of the subcommand: --json, --socket PATH, --timeout SEC.

Looking

Showing

Watching and serving

Output is a human table on a terminal and JSON when piped, so screen-angel windows reads well and screen-angel windows | jq parses. --json forces it either way.

Exit codes: 0 success, 1 request failed, 3 no app listening. The third is separate because the fix is to launch the app, not to change the command.

Reading a query result

PATH      ROLE        NAME          VALUE        BOUNDS          
0.3.12    button      Save                       1204,88 64×24   f
0.3.14    button      Cancel                     1276,88 64×24   

2 matched · 524 visited · 250ms

• path — walk-order handle, e.g. [0, 3, 12]. This is what other calls take.
• visited / durationMs — what the walk cost. Not decoration: see budgets.
• f flag — focused. d means disabled.

path is a handle, not an identity

path is valid until the tree changes. It is not an id. Do not store it and expect it to mean the same widget after the user scrolls, resizes, or opens a menu. Re-query instead. Containment is computed from path prefixes, which is why a > b needs no parent pointers and no tree reconstruction.

Budgets

Every accessibility attribute read is a synchronous round trip to another process. An unbounded walk of a browser window takes seconds and can make the target application appear to hang.

So walks are budgeted: maxDepth 12 and maxNodes 4000 by default.

2 matched · 3998 visited · 4120ms · TRUNCATED — budget ran out, raise --nodes or narrow the selector

truncated: true does not mean the application has no more widgets. It means the walk stopped. Treating a truncated empty result as "the button isn't there" is the single most likely wrong conclusion available here. Narrow the selector — a specific role prefix prunes enormous amounts of tree — or raise --nodes deliberately.

Images

A capture never returns pixels. It returns a descriptor: size, format, byte length, a sha256, and which transports this connection may use. You fetch the bytes in a second request, in the encoding you can actually use.

That indirection exists because the common question — "did a dialog appear", "is Save enabled" — is answered by the tree in a few hundred bytes. An interface that answered it with a 4K screenshot would charge every caller for pixels most of them did not want.

Grabbing part of the screen

One method, and every argument has a default that means "the whole thing, at a size a model can read".

{"method": "capture.grab", "params": {"target": {"element": [0, 3, 12]}}}

Targets are handles you already hold from a JSON reply, never rectangles you had to compute: {element: path}, {selector: "..."}, {window: pid | "focused"}, {screen: n | "focused"}, or {rect: {...}} as an escape hatch.

region names a part of the target instead of measuring it: all (default), top, bottom, left, right, thirds, ninths (top-left … bottom-right), title-bar, status-bar. Resolved against current bounds at capture time, so it cannot go stale between the query and the grab the way a computed rectangle can.

size is fit (1024px longest edge, the default), full, or a number. 1024 is not a compromise: a 4K screenshot is more expensive and worse for recognition than a 1024px one. If you need detail, ask for a smaller region, not a bigger image.

pad is tight, snug (default, 8px), loose (48px), or a number. A control captured at exactly its bounds is often unrecognizable — a checkbox becomes a grey square and its label, the only thing saying what it does, is outside the crop.

marks draws numbered markers on matching elements and returns a legend:

{"method": "capture.grab",
 "params": {"target": {"screen": "focused"}, "marks": "button"}}

You get an image with 1, 2, 3 drawn on the real widgets plus marks: [{n, path, role, name, bounds}]. Now "the third one" is unambiguous without a single coordinate crossing the wire. The markers are ours and vanish when cleared.

capture.batch takes several grabs and answers in one round trip. Six separate grabs cost six times the latency for no reason when you knew all six targets up front. A failed grab returns an error in its own slot rather than failing the batch.