M365 Agent Evaluator

What it does

When your agent uses it

• A user wants to create
• Analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI
• Intents such as evaluate my agent
• Create eval prompts

Example prompts

• “evaluate my agent”
• “test my agent”
• “run my evals”
• “/m365-agent-evaluator”

Requirements

• Node.js

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Identify the user goal: setup, dataset authoring, running evals, analyzing results, or updating an existing eval suite.
2. 2Load only the reference needed for the current goal
3. 3Detect project shape
4. 4Verify prerequisites without exposing values
5. 5Choose the workflow

What it can do on your machine

Read from SKILL.md and the folder at commit 1060873. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • npx
  • npm

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md. Its commands use npx and npm, which can reach the network depending on how they are called.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

M365 Agent Evaluator loads about 2k tokens when it runs, and up to ~16k if it reads all its reference files. Until then it costs about 104 tokens; SKILL.md has 708 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~104

When it runs · the whole SKILL.md, loaded when a task matches

~2k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~16k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NoteMentions a .env fileSKILL.md:32

  - ATK: `.env.local`, `.env.local.user`, `env\.env.local.user`, `m365agents.yml`, or `appPackage\declarativeAgent.json

• NoteMentions a .env fileSKILL.md:134

  | `.env.local` | Non-secret ATK config such as `M365_TITLE_ID`. |

• NoteMentions a .env fileSKILL.md:135

  | `.env.local.user` or `env\.env.local.user` | Local secrets such as tenant ID and Azure OpenAI key. |

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

Its licence (Custom licence) doesn't allow us to republish the file, so here is its outline and opening line. It has 708 words (~2,031 tokens).

name

m365-agent-evaluator

Read the full SKILL.md on GitHub

Files

SKILL.md and 16 other files (references) in plugins/microsoft-365-agents-toolkit/skills/m365-agent-evaluator of microsoft/work-iq.

• SKILL.md
• examples/analyze-failures.md
• examples/basic-generation.md
• examples/iterate-on-changes.md
• examples/missing-instructions.md
• examples/not-atk-project.md
• examples/run-and-analyze.md
• references/azure-setup.md
• references/eval-templates.md
• references/guardrails.md
• references/output-schema.json
• references/output-schema.md
• references/pra-framework.md
• references/prompts-schema.json
• references/remediation-patterns.md
• references/result-analysis.md
• references/workflow.md

Open the folder on GitHubat commit 1060873

Used in 1 other repository

We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders. This page covers the copy in microsoft/work-iq, which our catalogue first saw on October 7, 2026.

• microsoft/skills.github/plugins/microsoft-365-agents-toolkit/skills/m365-agent-evaluator