Agent skill

Run Evals

Prepare the environment and run the LLM-driven agent evals (e2e/agent-evals/) against a chosen sim-use binary.

Apache-2.0Auto-check passedAI & LLM Engineering

Install Run Evals

skills CLI

$ npx skills add lycorp-jp/sim-use --skill run-evals -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install lycorp-jp/sim-use run-evals --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/lycorp-jp/sim-use.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/run-evals .claude/skills/run-evals && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "run-evals" agent skill from https://github.com/lycorp-jp/sim-use/tree/main/.agents/skills/run-evals into .claude/skills/run-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-evals", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/lycorp-jp/sim-use/tree/main/.agents/skills/run-evals

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add lycorp-jp/sim-use --skill run-evals -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install lycorp-jp/sim-use run-evals --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/lycorp-jp/sim-use.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.agents/skills/run-evals .agents/skills/run-evals && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "run-evals" agent skill from https://github.com/lycorp-jp/sim-use/tree/main/.agents/skills/run-evals into .agents/skills/run-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-evals", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add lycorp-jp/sim-use --skill run-evals -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install lycorp-jp/sim-use run-evals --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/lycorp-jp/sim-use.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.agents/skills/run-evals .cursor/skills/run-evals && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "run-evals" agent skill from https://github.com/lycorp-jp/sim-use/tree/main/.agents/skills/run-evals into .cursor/skills/run-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-evals", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/lycorp-jp/sim-use.git --path .agents/skills/run-evals

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add lycorp-jp/sim-use --skill run-evals -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install lycorp-jp/sim-use run-evals --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/lycorp-jp/sim-use.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.agents/skills/run-evals .gemini/skills/run-evals && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "run-evals" agent skill from https://github.com/lycorp-jp/sim-use/tree/main/.agents/skills/run-evals into .gemini/skills/run-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-evals", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install lycorp-jp/sim-use run-evals

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add lycorp-jp/sim-use --skill run-evals -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/lycorp-jp/sim-use.git skills-src && mkdir -p .github/skills && cp -r skills-src/.agents/skills/run-evals .github/skills/run-evals && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "run-evals" agent skill from https://github.com/lycorp-jp/sim-use/tree/main/.agents/skills/run-evals into .github/skills/run-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-evals", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add lycorp-jp/sim-use --skill run-evals -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install lycorp-jp/sim-use run-evals --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/lycorp-jp/sim-use.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.agents/skills/run-evals .opencode/skills/run-evals && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "run-evals" agent skill from https://github.com/lycorp-jp/sim-use/tree/main/.agents/skills/run-evals into .opencode/skills/run-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-evals", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: run-evals
GitHub stars: 1.4k
Token cost: ~1.4k tokens
SKILL.md length: 632 words
Files: 1
Skills in repo: 4
Repo updated: Oct 9, 2026
First seen: Oct 7, 2026
Licence: Apache-2.0

At a glance

Prepare the environment and run the LLM-driven agent evals (e2e/agent-evals/) against a chosen sim-use binary.

Works in 4 steps: Decide WHICH sim-use is under test → Prepare devices and fixtures → Run → …
The user runs /run-evals
SKILL.md covers Step 1: Decide WHICH sim-use…, Step 2: Prepare devices and…, Step 3: Run and Step 4: Interpret and report, plus 1 more section
Calls make, xcrun and claude

What it does

Run Evals is an agent skill from lycorp-jp/sim-use. Prepare the environment and run the LLM-driven agent evals (e2e/agent-evals/) against a chosen sim-use binary. Use when the user runs /run-evals or asks to "run the agent evals", "run the LLM-driven tests", "eval the skill", or wants pre-release confidence that an agent reading the bundled skill still picks the right verbs. Costs real claude -p API calls — always confirm before spending.

Its SKILL.md is about 1.4k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in AI & LLM Engineering, covering LLM evaluation and End-to-end testing. It works with iOS and Android. The repository describes itself as: Give your AI agent eyes and hands on iOS Simulator and Android emulator/devices. The licence is Apache-2.0.

When your agent uses it

The user runs /run-evals
Asks to run the agent evals
Run the LLM-driven tests
Wants pre-release confidence that an agent reading the bundled skill still picks the right verbs

Example prompts

“run the agent evals”
“run the LLM-driven tests”
“eval the skill”
“/run-evals”

Requirements

Python 3

Workflow steps

4 steps, taken from the step headings in SKILL.md.

Decide WHICH sim-use is under test
Prepare devices and fixtures
Run
Interpret and report

What it can do on your machine

Read from SKILL.md and the folder at commit 30274ba. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
- make
- xcrun
- claude
- adb
- python3
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Run Evals loads about 1.4k tokens when it runs. Until then it costs about 101 tokens; SKILL.md has 632 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~101
When it runs · the whole SKILL.md, loaded when a task matches: ~1.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from lycorp-jp/sim-use at commit 30274ba, republished under its Apache-2.0 licence (© lycorp-jp). 632 words, ~1,356 tokens.

Download SKILL.mdSave it as .claude/skills/run-evals/SKILL.md (or your agent's skills folder).

name: run-evals
description: Prepare the environment and run the LLM-driven agent evals (e2e/agent-evals/) against a chosen sim-use binary. Use when the user runs `/run-evals` or asks to "run the agent evals", "run the LLM-driven tests", "eval the skill", or wants pre-release confidence that an agent reading the bundled skill still picks the right verbs. Costs real `claude -p` API calls — always confirm before spending.

This skill orchestrates the agent-eval suite: natural-language cases executed by a headless claude -p agent using the bundled skill (skills/sim-use/) against the Playground fixture apps, judged by deterministic post-condition checks. It verifies the layer the scripted E2E suites cannot: that an agent reading SKILL.md reaches for the right verbs and survives the documented pitfalls. A failure here with a green scripted layer usually means skill-prose drift, not a CLI bug.

Execution is delegated to scripts/eval.sh / e2e/agent-evals/run.py — do not reimplement their logic. Case anatomy, tags, and authoring rules live in e2e/agent-evals/README.md. Run from the repo root.

Step 1: Decide WHICH sim-use is under test

The whole run — device probing, the agent's commands, the verification layer — resolves sim-use from PATH unless overridden. Never let this be implicit:

Ask (or infer from the user's request) which binary to evaluate:
- Installed release (default): whatever sim-use resolves to on PATH.
- A development build: pass -b <path>, e.g. -b .build/out/Products/Debug/sim-use (SwiftBuild layout) or -b .build/debug/sim-use (classic). Build it first with make build.
Confirm the resolution and report it to the user before running:
bash
```
python3 -c 'import pathlib,shutil; print(pathlib.Path(shutil.which("sim-use")).resolve())'
sim-use --version
```
The wrapper prints sim-use under test: <real path> (<version>) and the run report records it under sim-use under test: — quote that line back in your summary so the human knows exactly what was evaluated.

Step 2: Prepare devices and fixtures

For each platform you intend to cover (the wrapper auto-detects reachable ones; use -p ios|android to restrict):

iOS

Device Hub (Xcode 27) must be CLOSED — pgrep dtuhidd must be empty. A simulator booted while Device Hub is open has legacy HID disconnected; sim-use's guard will (correctly) fail every case on it. If dtuhidd is running: quit Device Hub, then shutdown && boot the simulator.
Boot a simulator and wait: xcrun simctl boot <UDID> && xcrun simctl bootstatus <UDID>.
The Playground fixture must be installed. Check: xcrun simctl listapps <UDID> | grep -c com.cameroncooke.SimUsePlayground — if missing, install with scripts/test-runner.sh -b (builds sim-use + Playground, ~2-3 min).

Android

Start an emulator (not on PATH by default: ~/Library/Android/sdk/emulator/emulator -avd <AVD> &), wait for adb shell getprop sys.boot_completed → 1.
Both fixture packages must be present: adb shell pm list packages | grep -c com.linecorp.simuse should be 2 (playground + device bridge). If missing, make e2e-android installs them.
A stale bridge from an older CLI version is fine — the version parity check fires and the agent is expected to recover via sim-use android init (that recovery is itself part of what the evals exercise).

Show full SKILL.md (225 more words)Show less

Step 3: Run

bash

make eval                                  # quick tag, every reachable platform, asks before spending
make eval ARGS="-y -t quick"               # skip the cost prompt (release-gate style)
make eval ARGS="-p ios -b .build/out/Products/Debug/sim-use"   # dev build, one platform
scripts/eval.sh -- --cases <id>            # a single case (raw run.py args)

Cost: each case is a real claude -p agent (~1-3 min, real API charge; the wrapper prints an estimate and asks unless -y). Never pass -y without the user having approved the spend in this conversation.

Step 4: Interpret and report

Reports land in e2e/agent-evals/reports/<timestamp>/ (gitignored): report.md (verdict table + env header), verdicts.jsonl, and one stream-json transcript per case.

All PASS → report the verdict table, the sim-use under test line, and the report path.
FAIL → read the case's transcript before concluding anything. Classify:
1. Skill-prose drift — the agent picked a wrong verb or missed a documented pitfall the skill should have steered around → fix skills/sim-use/SKILL.md, not the case.
2. CLI regression — the right verb failed → treat as a product bug; reproduce it directly with sim-use before filing.
3. Environment/fixture noise — reboot-settling, Playground missing, Device-Hub-poisoned boot → fix the environment and re-run; if the coupling is inherent, tag the case fragile (fragile-tagged cases never gate a run).
ERROR → the harness itself broke (reset failed, claude missing); fix the environment, don't touch cases.

Things to NOT do

Don't run evals without stating which binary is under test.
Don't pass -y unless the user already approved the cost.
Don't edit or delete eval cases to make a run green — a red case is signal; classify it first (Step 4).
Don't commit anything under e2e/agent-evals/reports/ (gitignored on purpose).

Files

Just SKILL.md in .agents/skills/run-evals of lycorp-jp/sim-use.

Open the folder on GitHubat commit 30274ba

Compare with similar skills

Run Evals next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Run Evals compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Run Evals this skilllycorp-jp/sim-use	1.4k	—	~1.4k	Automated safety check: Pass	Apache-2.0	today
E2Ecallstack/react-native-pager-view	3.4k	1 repos	~2.1k	Automated safety check: Pass	MIT	today
E2Egronxb/hot-updater	1.8k	—	~1.6k	Automated safety check: Pass	Custom licence	today
E2E Current PRgronxb/hot-updater	1.8k	—	~1k	Automated safety check: Pass	Custom licence	today
Kane CLI Browser TestingLambdaTest/kane-cli	248	—	~8.4k	Automated safety check: Pass	Apache-2.0	today
E2E Defaultgronxb/hot-updater	1.8k	—	~3.1k	Automated safety check: Pass	Custom licence	today

Similar skills

E2E
callstack/react-native-pager-view
Agentic end-to-end tests with e2e, the e2e runner. An agent skill from callstack/react-native-pager-view.
3.4k GitHub starsUsed in 1 repo~2.1k tokens
Testing & QAAuto-check passed
E2E
gronxb/hot-updater
Run end-to-end OTA verification for examples/v0.85.0 with agent-device.
1.8k GitHub stars~1.6k tokensUpdated today
Testing & QAAuto-check passed
E2E Current PR
gronxb/hot-updater
Generate and execute PR-aware OTA E2E scenarios for examples/v0.85.0 by diffing the checked-out branch against its PR base branch or default branch, inferring the affected runtime, rollout, and…
1.8k GitHub stars~1k tokensUpdated today
Testing & QAAuto-check passed
Kane CLI Browser Testing
LambdaTest/kane-cli
Drives a real browser through the kane-cli tool and designs requirement-linked test suites from a PRD or a plain description, with mobile and cloud-grid runs.
248 GitHub stars~8.4k tokensUpdated today
Testing & QAAuto-check passed
E2E Default
gronxb/hot-updater
Run a fixed OTA regression flow for examples/v0.85.0 with agent-device.
1.8k GitHub stars~3.1k tokensUpdated today
Testing & QAAuto-check passed
Argent QA Flows
bbplayer-app/BBPlayer
Create repeatable QA regression E2E tests as Argent flows from test cases, tickets, or acceptance criteria.
1.2k GitHub stars~3.2k tokensUpdated yesterday
Testing & QAAuto-check passed

Questions about Run Evals

What does Run Evals do?

Prepare the environment and run the LLM-driven agent evals (e2e/agent-evals/) against a chosen sim-use binary. Run Evals is an agent skill from lycorp-jp/sim-use. Prepare the environment and run the LLM-driven agent evals (e2e/agent-evals/) against a chosen sim-use binary.

When should I use Run Evals?

Run Evals fits situations like: the user runs /run-evals; asks to run the agent evals; run the LLM-driven tests; wants pre-release confidence that an agent reading the bundled skill still picks the right verbs.

How do I install Run Evals in Claude Code?

Run `npx skills add lycorp-jp/sim-use --skill run-evals -a claude-code`. Or copy the skill folder (.agents/skills/run-evals in lycorp-jp/sim-use) into .claude/skills/run-evals in your project. Claude Code loads it when a task matches its description.

How do I install Run Evals in Codex?

Run `npx skills add lycorp-jp/sim-use --skill run-evals -a codex`. Or copy the skill folder (.agents/skills/run-evals in lycorp-jp/sim-use) into .agents/skills/run-evals in your project. Codex loads it when a task matches its description.

Can I use Run Evals in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add lycorp-jp/sim-use --skill run-evals -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/run-evals, .gemini/skills/run-evals, .github/skills/run-evals and .opencode/skills/run-evals in your project.

What does Run Evals need to run?

Going by SKILL.md and its folder, Run Evals needs the command-line tools its instructions call (make, xcrun, claude, adb and python3). Our summary lists: Python 3.

Does Run Evals access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Run Evals safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Run Evals use?

Run Evals is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Run Evals use?

About 1.4k tokens (SKILL.md is roughly 5.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Run Evals?

Skills that share tags, products or a category with Run Evals: E2E (callstack/react-native-pager-view, 3.4k stars), E2E (gronxb/hot-updater, 1.8k stars), E2E Current PR (gronxb/hot-updater, 1.8k stars) and Kane CLI Browser Testing (LambdaTest/kane-cli, 248 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Run Evals?

lycorp-jp (a GitHub organization) maintains it in lycorp-jp/sim-use, which has 1,397 GitHub stars. The repository holds 4 skills in this directory. The repository was last updated on October 9, 2026.

Source: lycorp-jp/sim-use on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Run Evals

Install Run Evals

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Step 1: Decide WHICH sim-use is under test

Step 2: Prepare devices and fixtures

Step 3: Run

Step 4: Interpret and report

Things to NOT do

Files

Compare with similar skills

Similar skills

E2E

E2E

E2E Current PR

Kane CLI Browser Testing

E2E Default

Argent QA Flows

More from lycorp-jp/sim-use

Sim Use

Bump Version Dev

Release

Questions about Run Evals

Install Run Evals

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Step 1: Decide WHICH sim-use is under test

Step 2: Prepare devices and fixtures

Step 3: Run

Step 4: Interpret and report

Things to NOT do

Files

Compare with similar skills

Similar skills

More from lycorp-jp/sim-use

Related

Questions about Run Evals