Agent skill

macOS Control Skill

by OpenLoaf in OpenLoaf/OpenLoaf

当用户要求 AI 直接操作他的 macOS 桌面时触发:打开某个 App、点击某个按钮、填写窗口里的表单、读取当前屏幕内容、自动化本地 GUI 流程。典型说法"帮我在访达里…"、"帮我在某 App 里点一下…"、"看看我屏幕上显示的是什么"。仅在 OpenLoaf Desktop(macOS)中可用;非桌面端会返回 desktop-only 错误,应改用 BrowserAct…

AGPL-3.0Auto-check passed

Install macOS Control Skill

skills CLI
$ npx skills add OpenLoaf/OpenLoaf --skill macos-control-skill -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install OpenLoaf/OpenLoaf macos-control-skill --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/OpenLoaf/OpenLoaf.git skills-src && mkdir -p .claude/skills && cp -r skills-src/apps/server/src/ai/builtin-skills/macos-control .claude/skills/macos-control-skill && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
macos-control-skill
GitHub stars
108
Token cost
~1.6k tokens
SKILL.md length
362 words
Files
1
Skills in repo
33
Repo updated
First seen
Licence
AGPL-3.0

At a glance

当用户要求 AI 直接操作他的 macOS 桌面时触发:打开某个 App、点击某个按钮、填写窗口里的表单、读取当前屏幕内容、自动化本地 GUI 流程。典型说法"帮我在访达里…"、"帮我在某 App 里点一下…"、"看看我屏幕上显示的是什么"。仅在 OpenLoaf Desktop(macOS)中可用;非桌面端会返回 desktop-only 错误,应改用 BrowserAct…

  • Works in 4 steps: 看 Survey 的 Menu bar 段 —— 几乎所有 app… → 菜单项如果带 (⌘⇧X) 这种 shortcut,用 MacosAct… → 走不通再 observe 截图,按像素 coord click → …
  • SKILL.md covers 核心思维:先理解,再动手, 工具全景, 执行路径的优先级(由 Survey 告诉你) and Survey 返回长什么样, plus 7 more sections
  • Reaches github.com

What it does

macOS Control Skill is an agent skill from OpenLoaf/OpenLoaf. 当用户要求 AI 直接操作他的 macOS 桌面时触发:打开某个 App、点击某个按钮、填写窗口里的表单、读取当前屏幕内容、自动化本地 GUI 流程。典型说法"帮我在访达里…"、"帮我在某 App 里点一下…"、"看看我屏幕上显示的是什么"。仅在 OpenLoaf Desktop(macOS)中可用;非桌面端会返回 desktop-only 错误,应改用 BrowserAct 或让用户自己操作。不用于:网页内交互(→ browser-ops-skill)、本地文件读写(→ Read/Edit/Write)。

Its SKILL.md is about 1.6k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It works with macOS. The repository describes itself as: 🍞Open-source, local-first AI workspace with Agents, multi-model chat (GPT/Claude/Gemini/DeepSeek), Notion-like docs, AI image & video generation, email, calendar & terminal…. The licence is AGPL-3.0.

Example prompts

  • “帮我在访达里…”
  • “帮我在某 App 里点一下…”
  • “看看我屏幕上显示的是什么”
  • “/macos-control-skill”

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

  1. 看 Survey 的 Menu bar 段 —— 几乎所有 app 的菜单栏都可以 menu_click
  2. 菜单项如果带 (⌘⇧X) 这种 shortcut,用 MacosAct type="key" 更稳
  3. 走不通再 observe 截图,按像素 coord click
  4. 完全走不通时停下来告诉用户"这条路径我没找到安全做法",别猜

What it can do on your machine

Read from SKILL.md and the folder at commit f7eccf6. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Hosts in commands or code, which the agent is likely to contact:

    • github.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

macOS Control Skill loads about 1.6k tokens when it runs. Until then it costs about 74 tokens; SKILL.md has 362 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~74
When it runs · the whole SKILL.md, loaded when a task matches
~1.6k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from OpenLoaf/OpenLoaf at commit f7eccf6, republished under its AGPL-3.0 licence (© OpenLoaf). 362 words, ~1,623 tokens.

Download SKILL.mdSave it as .claude/skills/macos-control-skill/SKILL.md (or your agent's skills folder).
name
macos-control-skill
description
当用户要求 AI 直接操作他的 macOS 桌面时触发:打开某个 App、点击某个按钮、填写窗口里的表单、读取当前屏幕内容、自动化本地 GUI 流程。典型说法"帮我在访达里…"、"帮我在某 App 里点一下…"、"看看我屏幕上显示的是什么"。**仅在 OpenLoaf Desktop(macOS)中可用**;非桌面端会返回 desktop-only 错误,应改用 `BrowserAct` 或让用户自己操作。**不用于**:网页内交互(→ `browser-ops-skill`)、本地文件读写(→ `Read`/`Edit`/`Write`)。
tools
MacosSurvey, MacosObserve, MacosAct, MacosListWindows, MacosCaptureWindow

macOS 桌面控制指南

核心思维:先理解,再动手

不要看到 app 就直接 click/type。人类操作一个陌生 app,会先"逛一圈"——看它长什么样、有哪些入口、哪里是关闭按钮、哪里是业务区。你也必须这样。

三阶段框架:

1. Survey(理解)  → MacosSurvey(app) 拿 app 心智模型
2. Plan(选路径)  → 按 Survey 给的"Recommended path order"挑一条路
3. Act + Verify   → MacosAct 执行;MacosObserve 确认结果

跳过 Survey 直接 Act 是最常见的失败。微信 / QQ / 飞书 / 钉钉这类自绘 UI 的 AX tree 只有 3 个窗口按钮(close / min / zoom),模型凭猜点 path=["0","0"] 就会命中红色关闭按钮,整个 app 被关掉。Survey 会把这类陷阱提前告诉你。

工具全景

工具什么时候用
MacosSurvey(app)每次接触某个 app 的第一步。返回 app 心智模型:已知 intent、AX 丰富度、菜单栏地图、窗口清单、推荐执行顺序。会话内缓存 5 分钟。
MacosObserve(appFilter?)看当前具体 UI 状态——截图 + AX 树 + 窗口清单。Survey 之后用,以及每次 MacosAct 后用来验证结果。
MacosAct执行动作。支持 intent / launch_app / menu_click / applescript / key / click / type / scroll / drag / wait / ax_action。
MacosListWindows(appFilter?)仅枚举窗口(比 Observe 轻量)。多窗口场景里挑特定 window 用。
MacosCaptureWindow(windowID)按 windowID 截特定窗口,可以截到被遮挡或最小化的窗口(Window Server 保留合成 buffer)。

全部仅在 OpenLoaf Desktop(macOS) 可用。非桌面端工具不注册;会话里不要尝试调它们。

执行路径的优先级(由 Survey 告诉你)

MacosSurvey 返回里有一段 Recommended path order,顺序从高确定性到低:

1. Known intents       MacosAct type="intent"          确定性 100%
2. Menu navigation     MacosAct type="menu_click"      自绘 UI 也能用
3. Keyboard shortcut   MacosAct type="key"             读菜单栏的 shortcut 字段
4. AX path click       MacosAct type="click" ref=...   仅当 Survey 判定 ax-rich 时安全
5. Coord click         MacosAct type="click" point=... 最后手段,从截图读像素

严格按顺序下探。能用前面的就不走后面。Survey 判 self-drawn 的 app 上,第 4 步直接跳过——对窗口 chrome 按钮的 AX path click 会被拦下(返回 WINDOW_CHROME_BLOCKED 错误)。

Survey 返回长什么样

App: 微信 (WeChat.app) (com.tencent.xinWeChat, pid=86795)

AX profile: self-drawn — AXWindow children are only chrome buttons
  richness: 0.995 (203 nodes)
  windowChildRoles: [AXCloseButton, AXFullScreenButton, AXMinimizeButton]

Known intents (from registry):
  - id: open_moments    — 打开朋友圈 / Open Moments feed
  - id: open_chats      — 切换到聊天列表
  - id: open_discover   — 打开"发现"页
  ...

Menu bar (34 entries with shortcuts, sample):
  - 文件 > 锁定微信  (⌘L)
  - 视图 > 聊天  (⌘1)
  - 视图 > 朋友圈  (⌘⇧4)
  ...

Windows (2):
  - windowID=227474 visible 1060×859 "Weixin"  (main)
  - windowID=260481 hidden 710×831

Recommended path order:
  1. Known intents (highest confidence): MacosAct type="intent" — 5 registered
  2. Menu navigation: MacosAct type="menu_click" — 34 reachable entries
  3. Keyboard shortcut: MacosAct type="key"
  4. AX path click: **DO NOT USE** on this app.
  5. Coordinate click: last resort.

"Known intents" 就是别人已经替你踩过坑的路径。优先用。

典型场景

场景 A:打开微信朋友圈
1. MacosAct { type: "launch_app", app: "WeChat" }
2. MacosSurvey { app: "WeChat" }
   → 看到 Known intents 里有 open_moments
3. MacosAct { type: "intent", app: "WeChat", intent: "open_moments" }
4. MacosObserve { appFilter: "WeChat" } — 验证朋友圈已打开

不要调完 launch_app 就直接 observe + click AX tree。那是老路径,会命中关闭按钮。

场景 B:在 Safari 打开 github.com
1. MacosSurvey { app: "Safari" }
   → Known intents 里有 open_url(args: url required)
2. MacosAct { type: "intent", app: "Safari", intent: "open_url", args: { url: "https://github.com" } }
3. MacosObserve { appFilter: "Safari" } — 验证页面加载
场景 C:读屏幕

用户问「我屏幕上显示什么」、「当前 App 里 xx 是什么」:直接一次 MacosObserve,从截图 + AX 树提炼答案。不需要 Survey,不需要 Act。

场景 D:未注册的 intent / ax-rich app

Survey 返回 ax-rich 且没有对应 intent 时,优先 menu_click / key,其次 AX path click:

1. MacosSurvey { app: "Finder" } → ax-rich, 推荐 menu_click / AX path click
2. 要新建窗口 → 看菜单栏 "文件 > 新建 Finder 窗口 (⌘N)"
3. MacosAct { type: "key", keys: ["cmd", "n"] } — 用快捷键比 click 稳
4. MacosObserve 验证
场景 E:多窗口(微信主窗口 + 图片预览)
1. MacosSurvey { app: "WeChat" } → Windows: 2 个,windowID 分别 227474/260481
2. MacosCaptureWindow { windowID: 227474 } — 专截主窗口(即便被预览挡住)
3. 后续 MacosAct click 的坐标基准已更新为这个窗口的图
场景 F:填原生窗口表单(AppKit app)

Survey 看到 ax-rich,目标元素有 AX role=AXTextField:

1. MacosObserve { appFilter: "..." } 拿输入框 ref
2. MacosAct { type: "click", ref: { app, path: [...] } } 聚焦
3. MacosAct { type: "type", text: "..." }
4. MacosAct { type: "ax_action", ref: { app, path: [submit button] }, action: "AXPress" }
5. MacosObserve 验证

当 Survey 没有你要的 intent 怎么办

  1. 看 Survey 的 Menu bar 段 —— 几乎所有 app 的菜单栏都可以 menu_click
  2. 菜单项如果带 (⌘⇧X) 这种 shortcut,用 MacosAct type="key" 更稳
  3. 走不通再 observe 截图,按像素 coord click
  4. 完全走不通时停下来告诉用户"这条路径我没找到安全做法",别猜
Show full SKILL.md (163 more words)Show less

铁律

  1. 第一次接触某 app,先 Survey。不 Survey 就 Act 是最常见的失败,特别是对自绘 UI。
  2. 优先走 Survey 推荐的第一条路(Known intents)。自己另辟蹊径多半翻车。
  3. 每次 MacosAct 后 MacosObserve。UI 是有状态的,你无法预测结果。例外:intent / menu_click 成功后如果不需要二次动作,可以跳过 verify。
  4. AX path click 只在 Survey 判 ax-rich 的 app 上用。mixed 谨慎,self-drawn 禁止——chrome 按钮会被硬拦(WINDOW_CHROME_BLOCKED)。
  5. 坐标点击是最后手段。必须先 observe 看到截图像素位置;连续 2 次 coord click 后 UI 没变化,立即停止并告知用户定位失败。
  6. 敏感 app 前台(密码管理器 / 银行 / Keychain)立即停止,让用户自己操作。
  7. 缺权限时:系统设置面板已自动弹出,用自然语言告诉用户去启用 OpenLoaf,等用户确认再继续。不要重试。
  8. 非 macOS 桌面端:立即返回 desktop-only,改用 BrowserAct 或交回用户。

权限

首次调用会提示缺权限。server 已自动打开对应系统设置面板。需要:

  • screen:屏幕录制(Survey / Observe / CaptureWindow 都要)
  • accessibility:辅助功能(AX 树读取 + 合成输入)

某些系统级 app 授权后仍需重启 OpenLoaf Desktop。

坐标空间(退化到 point 时)

point.{x,y} 就是最近一次 MacosObserve 或 MacosCaptureWindow 截图上的像素坐标——图上目标在哪个像素,就把那个像素塞给 click/scroll/drag。工具内部自动换算到屏幕坐标,你不用关心 retina 缩放、窗口偏移或多显示器。

  • observe/capture_window 返回里 Screenshot: window 2120×1718 px 的那个数字就是坐标范围
  • 必须先调用过一次 observe 或 capture_window 才能用坐标动作;否则工具拒绝

安全与隐私

  • 密码、银行、密钥链类 App 不要 Survey / Observe / Act。遇到 1Password / Keychain / 银行 app 前台,立即停止并让用户自己操作。
  • 截图存会话附件目录,随会话清理;不上传云端、不训练。
  • 不确定屏幕状态时先 observe 一次再判断;不要臆测。

错误诊断

  • permissionsMissing → 已自动开设置面板,提示用户授权 → 等确认
  • desktop-only → 当前非桌面端;告诉用户切到 OpenLoaf Desktop
  • WINDOW_CHROME_BLOCKED → 你点了窗口 chrome 按钮。按返回里的 4 个候选路径重新规划(menu_click / key / coord / URL)。不要傻乎乎加 confirm_window_chrome:true 强推,除非真的要关窗
  • intent 未找到 → 用 MacosSurvey 看一下可用 intent 列表;或降级到 menu_click
  • 点击无反应 → 重新 observe,看目标节点是否真可点、是否被遮挡
  • 中途前台切换 → observe 的 app 和 act 的 app 不一致,先 key cmd+tab 切回再 act

© OpenLoaf, AGPL-3.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in apps/server/src/ai/builtin-skills/macos-control of OpenLoaf/OpenLoaf.

Open the folder on GitHubat commit f7eccf6

Compare with similar skills

macOS Control Skill next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

macOS Control Skill compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
macOS Control Skill this skillOpenLoaf/OpenLoaf108—~1.6kAutomated safety check: PassAGPL-3.0
Site ArchitectureAvdLee/RocketSimApp80311 repos~3.3kAutomated safety check: PassCustom licence
Engine Whats Newflutter/flutter179k—~978Automated safety check: PassBSD-3-Clause
macOS Spm App PackagingDimillian/Skills4k5 repos~1.2kAutomated safety check: PassMIT
Openclaw Live Updateropenclaw/openclaw392k—~3.7kAutomated safety check: PassMIT
Orca iOS Simulator Controlstablyai/orca87k1 repos~584Automated safety check: PassApache-2.0

Similar skills

  • Site Architecture

    AvdLee/RocketSimApp

    When the user wants to plan, map, or restructure their website's page hierarchy, navigation, URL structure, or internal linking.

    803 GitHub starsUsed in 11 repos~3.3k tokens
    Marketing & SEOAuto-check passed
  • Engine Whats New

    flutter/flutter

    Generates the "what's new" release summary and diff file for changes in the Flutter engine (//engine/src/flutter) between two releases (e.g., 3.47 vs 3.44).

    179k GitHub stars~978 tokensUpdated today
    MobileAuto-check passed
  • macOS Spm App Packaging

    Dimillian/Skills

    Scaffold, build, and package SwiftPM-based macOS apps without an Xcode project.

    4k GitHub starsUsed in 5 repos~1.2k tokens
    MobileAuto-check passed
  • Openclaw Live Updater

    openclaw/openclaw

    Maintain the canonical live OpenClaw main checkout, macOS LaunchAgent-managed Gateway, local macOS app, exact-head main CI, and recurring full release validation.

    392k GitHub stars~3.7k tokensUpdated today
    DevOps & CloudAuto-check passed
  • iOS Simulator control from inside Orca, with the live device view in Orca's emulator pane. Use when driving a booted Apple Simulator on macOS: taps, gestures…

    87k GitHub starsUsed in 1 repo~584 tokens
    MobileAuto-check passed
  • A catalog of recurring bug shapes in the Mole Mac cleaner, used to review safety-sensitive diffs for deletion safety, unbounded commands, shell traps and weak tests.

    70k GitHub stars~2k tokensUpdated today
    DevelopmentAuto-check passed

More from OpenLoaf/OpenLoaf

All 33 skills in this repo
  • Agent Orchestration Skill

    OpenLoaf/OpenLoaf

    Triggers when the master Agent faces a multi-step complex task and is deciding whether / how to outsource sub-tasks to built-in subagents (browser / doc-editor / data-analyst / extractor /…

    108 GitHub stars~1.9k tokensUpdated 4 mo ago
    Auto-check passed
  • Browser Ops Skill

    OpenLoaf/OpenLoaf

    Triggered when the user asks for page-level interaction with a specific webpage: login, form filling, button clicks, pagination scraping, screenshots, downloading page images, handling CAPTCHAs or…

    108 GitHub stars~1.5k tokensUpdated 4 mo ago
    Auto-check passed
  • Canvas Ops Skill

    OpenLoaf/OpenLoaf

    Triggered when the user wants lifecycle management of OpenLoaf canvases / whiteboards: create, open, filter, duplicate, delete, rename, or change ownership.

    108 GitHub stars~1.4k tokensUpdated 4 mo ago
    Auto-check passed
  • Reads, edits, converts and reviews Word documents through three dedicated tools, covering tracked changes, comments, tables, images and format conversion.

    108 GitHub stars~1.9k tokensUpdated 4 mo ago
    Auto-check passed
  • Email Operations

    OpenLoaf/OpenLoaf

    Handles a real email account through query and mutate tools: check the inbox, read, search, reply, forward, compose and organize, with sending always confirmed first.

    108 GitHub stars~1.8k tokensUpdated 4 mo ago
    Auto-check passed
  • macOS Desktop Control

    OpenLoaf/OpenLoaf

    Guides an agent to operate native macOS apps by surveying an app first, acting through intents, menus or keystrokes, and verifying each step, in OpenLoaf Desktop only.

    108 GitHub stars~2.4k tokensUpdated 4 mo ago
    Auto-check passed

Works with

Questions about macOS Control Skill

What does macOS Control Skill do?

当用户要求 AI 直接操作他的 macOS 桌面时触发:打开某个 App、点击某个按钮、填写窗口里的表单、读取当前屏幕内容、自动化本地 GUI 流程。典型说法"帮我在访达里…"、"帮我在某 App 里点一下…"、"看看我屏幕上显示的是什么"。仅在 OpenLoaf Desktop(macOS)中可用;非桌面端会返回 desktop-only 错误,应改用 BrowserAct…. macOS Control Skill is an agent skill from OpenLoaf/OpenLoaf.

How do I install macOS Control Skill in Claude Code?

Run `npx skills add OpenLoaf/OpenLoaf --skill macos-control-skill -a claude-code`. Or copy the skill folder (apps/server/src/ai/builtin-skills/macos-control in OpenLoaf/OpenLoaf) into .claude/skills/macos-control-skill in your project. Claude Code loads it when a task matches its description.

How do I install macOS Control Skill in Codex?

Run `npx skills add OpenLoaf/OpenLoaf --skill macos-control-skill -a codex`. Or copy the skill folder (apps/server/src/ai/builtin-skills/macos-control in OpenLoaf/OpenLoaf) into .agents/skills/macos-control-skill in your project. Codex loads it when a task matches its description.

Can I use macOS Control Skill in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add OpenLoaf/OpenLoaf --skill macos-control-skill -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/macos-control-skill, .gemini/skills/macos-control-skill, .github/skills/macos-control-skill and .opencode/skills/macos-control-skill in your project.

What does macOS Control Skill need to run?

SKILL.md names no scripts, command-line tools or credentials: macOS Control Skill is instructions for the agent only.

Does macOS Control Skill access the network?

SKILL.md names 1 domain. In commands or code: github.com; the agent is likely to contact it when it follows the instructions. This is read from the text; nothing was executed.

Is macOS Control Skill safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does macOS Control Skill use?

macOS Control Skill is published under the AGPL-3.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does macOS Control Skill use?

About 1.6k tokens (SKILL.md is roughly 6.5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to macOS Control Skill?

Skills that share tags, products or a category with macOS Control Skill: Site Architecture (AvdLee/RocketSimApp, 803 stars), Engine Whats New (flutter/flutter, 179k stars), macOS Spm App Packaging (Dimillian/Skills, 4k stars) and Openclaw Live Updater (openclaw/openclaw, 392k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains macOS Control Skill?

OpenLoaf (a GitHub organization) maintains it in OpenLoaf/OpenLoaf, which has 108 GitHub stars. The repository holds 33 skills in this directory. The repository was last updated on May 14, 2026.

Source: OpenLoaf/OpenLoaf on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.