macOS Desktop Control

What it does

When your agent uses it

• Opening a native Mac app and clicking through its interface
• Reading what is currently on the screen
• Filling in a form inside a native window
• Automating a repeatable local GUI flow

Example prompts

• “In Finder, open my Downloads folder and tell me what is in it.”
• “Click Export in the Keynote window and tell me what the dialog shows.”
• “What's on my screen right now?”

Requirements

• OpenLoaf Desktop running on macOS

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

1. 1Look at Survey's Menu bar section — almost every app's menu bar is menu_click-able
2. 2If a menu item has (⌘⇧X), MacosAct type="key" is more reliable than click
3. 3Last resort: observe the screenshot, then coord click
4. 4If none of these work, stop and tell the user "I don't have a safe path for this". Don't guess.

What it can do on your machine

Read from SKILL.md and the folder at commit f7eccf6. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md.

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Hosts in commands or code, which the agent is likely to contact:

  • github.com

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

macOS Desktop Control loads about 2.4k tokens when it runs. Until then it costs about 144 tokens; SKILL.md has 868 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~144

When it runs · the whole SKILL.md, loaded when a task matches

~2.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from OpenLoaf/OpenLoaf at commit f7eccf6, republished under its AGPL-3.0 licence (© OpenLoaf). 868 words, ~2,430 tokens.

name

macos-control-skill

description

Triggered when the user asks the AI to directly operate their macOS desktop: open an app, click a button, fill a form inside a native window, read what's currently on screen, automate a local GUI flow. Typical phrasing: "in Finder, please…", "click X in the Y app", "what's on my screen right now". **Available only in OpenLoaf Desktop (macOS)**; other platforms return a desktop-only error — fall back to `BrowserAct` or hand the task back to the user. **Not for**: in-page web interaction (→ `browser-ops-skill`), local file I/O (→ `Read`/`Edit`/`Write`).

tools

MacosSurvey, MacosObserve, MacosAct, MacosListWindows, MacosCaptureWindow

1. Survey    → MacosSurvey(app)  — get a mental model of the app
2. Plan      → pick a path from the "Recommended path order" the survey gives you
3. Act+Verify → MacosAct executes; MacosObserve confirms the result

1. Known intents       MacosAct type="intent"         100% confidence
2. Menu navigation     MacosAct type="menu_click"     works on self-drawn apps too
3. Keyboard shortcut   MacosAct type="key"            read the shortcut from the menu tree
4. AX path click       MacosAct type="click" ref=...  only when Survey verdict is ax-rich
5. Coord click         MacosAct type="click" point=... last resort, read pixels off a screenshot

App: 微信 (WeChat.app) (com.tencent.xinWeChat, pid=86795)

AX profile: self-drawn — AXWindow children are only chrome buttons
  richness: 0.995 (203 nodes)
  windowChildRoles: [AXCloseButton, AXFullScreenButton, AXMinimizeButton]

Known intents (from registry):
  - id: open_moments    — Open Moments feed
  - id: open_chats      — Switch to chats tab
  - id: open_discover   — Open Discover tab
  ...

Menu bar (34 entries with shortcuts, sample):
  - File > Lock WeChat  (⌘L)
  - View > Chats  (⌘1)
  - View > Moments  (⌘⇧4)
  ...

Windows (2):
  - windowID=227474 visible 1060×859 "Weixin"  (main)
  - windowID=260481 hidden 710×831

Recommended path order:
  1. Known intents (highest confidence): MacosAct type="intent" — 5 registered
  2. Menu navigation: MacosAct type="menu_click" — 34 reachable entries
  3. Keyboard shortcut: MacosAct type="key"
  4. AX path click: **DO NOT USE** on this app.
  5. Coordinate click: last resort.

1. MacosAct { type: "launch_app", app: "WeChat" }
2. MacosSurvey { app: "WeChat" }
   → sees Known intents contains open_moments
3. MacosAct { type: "intent", app: "WeChat", intent: "open_moments" }
4. MacosObserve { appFilter: "WeChat" } — verify Moments is showing

1. MacosSurvey { app: "Safari" }
   → Known intents has open_url (requires url arg)
2. MacosAct { type: "intent", app: "Safari", intent: "open_url", args: { url: "https://github.com" } }
3. MacosObserve { appFilter: "Safari" } — verify the page loads

1. MacosSurvey { app: "Finder" } → ax-rich; recommends menu_click / AX path click
2. Want a new Finder window → menu bar shows "File > New Finder Window (⌘N)"
3. MacosAct { type: "key", keys: ["cmd", "n"] } — shortcut beats click
4. MacosObserve to verify

1. MacosSurvey { app: "WeChat" } → Windows: 2 entries, windowIDs 227474/260481
2. MacosCaptureWindow { windowID: 227474 } — capture the main window directly (even if preview covers it)
3. Subsequent MacosAct click coords are now based on that window's image

1. MacosObserve { appFilter: "..." } to get the text field ref
2. MacosAct { type: "click", ref: { app, path: [...] } } to focus
3. MacosAct { type: "type", text: "..." }
4. MacosAct { type: "ax_action", ref: { app, path: [submit button] }, action: "AXPress" }
5. MacosObserve to verify

© OpenLoaf, AGPL-3.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in apps/server/src/ai/builtin-skills/macos-control/en of OpenLoaf/OpenLoaf.

Open the folder on GitHubat commit f7eccf6