OpenHarness End-to-End Evals
HKUDS/OpenHarness
Validates OpenHarness features by running real multi-turn agent loops with live LLM calls against an unfamiliar codebase, checking actual tool execution.
Guide for adding new end-to-end tests for the Packmind CLI. An agent skill from PackmindHub/packmind.
$ npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install PackmindHub/packmind cli-e2e-test-authoring --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/PackmindHub/packmind.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/cli-e2e-test-authoring .claude/skills/cli-e2e-test-authoring && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "cli-e2e-test-authoring" agent skill from https://github.com/PackmindHub/packmind/tree/main/.agents/skills/cli-e2e-test-authoring into .claude/skills/cli-e2e-test-authoring/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "cli-e2e-test-authoring", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/PackmindHub/packmind/tree/main/.agents/skills/cli-e2e-test-authoringType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install PackmindHub/packmind cli-e2e-test-authoring --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/PackmindHub/packmind.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.agents/skills/cli-e2e-test-authoring .agents/skills/cli-e2e-test-authoring && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "cli-e2e-test-authoring" agent skill from https://github.com/PackmindHub/packmind/tree/main/.agents/skills/cli-e2e-test-authoring into .agents/skills/cli-e2e-test-authoring/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "cli-e2e-test-authoring", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install PackmindHub/packmind cli-e2e-test-authoring --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/PackmindHub/packmind.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.agents/skills/cli-e2e-test-authoring .cursor/skills/cli-e2e-test-authoring && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "cli-e2e-test-authoring" agent skill from https://github.com/PackmindHub/packmind/tree/main/.agents/skills/cli-e2e-test-authoring into .cursor/skills/cli-e2e-test-authoring/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "cli-e2e-test-authoring", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/PackmindHub/packmind.git --path .agents/skills/cli-e2e-test-authoring--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install PackmindHub/packmind cli-e2e-test-authoring --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/PackmindHub/packmind.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.agents/skills/cli-e2e-test-authoring .gemini/skills/cli-e2e-test-authoring && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "cli-e2e-test-authoring" agent skill from https://github.com/PackmindHub/packmind/tree/main/.agents/skills/cli-e2e-test-authoring into .gemini/skills/cli-e2e-test-authoring/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "cli-e2e-test-authoring", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install PackmindHub/packmind cli-e2e-test-authoringInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/PackmindHub/packmind.git skills-src && mkdir -p .github/skills && cp -r skills-src/.agents/skills/cli-e2e-test-authoring .github/skills/cli-e2e-test-authoring && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "cli-e2e-test-authoring" agent skill from https://github.com/PackmindHub/packmind/tree/main/.agents/skills/cli-e2e-test-authoring into .github/skills/cli-e2e-test-authoring/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "cli-e2e-test-authoring", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install PackmindHub/packmind cli-e2e-test-authoring --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/PackmindHub/packmind.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.agents/skills/cli-e2e-test-authoring .opencode/skills/cli-e2e-test-authoring && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "cli-e2e-test-authoring" agent skill from https://github.com/PackmindHub/packmind/tree/main/.agents/skills/cli-e2e-test-authoring into .opencode/skills/cli-e2e-test-authoring/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "cli-e2e-test-authoring", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
cli-e2e-test-authoringGuide for adding new end-to-end tests for the Packmind CLI. An agent skill from PackmindHub/packmind.
CLI E2E Test Authoring is an agent skill from PackmindHub/packmind. Guide for adding new end-to-end tests for the Packmind CLI. This skill should be used when creating new test specs in the apps/cli-e2e-tests/ directory that exercise CLI commands against a real binary and API.
Its SKILL.md is about 1.6k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Testing & QA, covering End-to-end testing. It works with Git. The repository describes itself as: Packmind seamlessly captures your engineering playbook and turns it into AI context, guardrails, and governance. The licence is Apache-2.0.
4 steps, taken from the first numbered list in SKILL.md.
Read from SKILL.md and the folder at commit 858ed50. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
No scripts in the folder and no shell commands in SKILL.md (its code samples are typescript).
From the folder's file list and the shell code blocks in SKILL.md.
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
CLI E2E Test Authoring loads about 1.6k tokens when it runs. Until then it costs about 59 tokens; SKILL.md has 399 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from PackmindHub/packmind at commit 858ed50, republished under its Apache-2.0 licence (© PackmindHub). 399 words, ~1,575 tokens.
.claude/skills/cli-e2e-test-authoring/SKILL.md (or your agent's skills folder).Add end-to-end tests that exercise the Packmind CLI binary (dist/apps/cli/main.cjs) in realistic conditions. Tests run the actual CLI as a child process and, when authentication is needed, interact with a running API via HTTP gateways.
All spec files live in apps/cli-e2e-tests/src/ and follow the naming convention <command>.spec.ts.
Two wrappers are available depending on whether the command requires authentication.
describeWithTempSpaceProvides an isolated temporary directory. No API required.
import { describeWithTempSpace, runCli } from './helpers';
describeWithTempSpace('my-command without auth', (getContext) => {
let result: Awaited<ReturnType<typeof runCli>>;
beforeEach(async () => {
const { testDir } = await getContext();
result = await runCli('my-command', { cwd: testDir });
});
it('exits with code 1', () => {
expect(result.returnCode).toBe(1);
});
it('shows an error message', () => {
expect(result.stdout).toContain('No credentials found');
});
});describeWithUserSignedUpExtends describeWithTempSpace by creating a real user account, signing in, and generating an API key. Requires a running API at http://localhost:4200.
import { describeWithUserSignedUp, runCli } from './helpers';
describeWithUserSignedUp('my-command with auth', (getContext) => {
let result: Awaited<ReturnType<typeof runCli>>;
beforeEach(async () => {
const { apiKey, testDir } = await getContext();
result = await runCli('my-command', { apiKey, cwd: testDir });
});
it('succeeds', () => {
expect(result.returnCode).toBe(0);
});
it('displays expected output', () => {
expect(result.stdout).toContain('Expected text');
});
});The context object from describeWithUserSignedUp provides:
| Field | Description |
|---|---|
testDir | Isolated temporary directory |
apiKey | Valid API key for the created user |
user | User object (email, etc.) |
organization | Organization the user belongs to |
spaceId | Global space ID for the organization |
gateway | Authenticated gateway to call the API directly |
Commands that interact with git (e.g. diff, install) require a git repo. Call setupGitRepo in beforeEach:
import { setupGitRepo } from './helpers';
beforeEach(async () => {
const { testDir } = await getContext();
await setupGitRepo(testDir);
});Use gateway from the context to seed data before running the CLI:
beforeEach(async () => {
const { gateway, spaceId, testDir } = await getContext();
await setupGitRepo(testDir);
await gateway.commands.create({ spaceId, /* ... */ });
await gateway.packages.create({ spaceId, /* ... */ });
});Use file helpers to create or modify files in the test directory:
import { readFile, updateFile, fileExists } from './helpers';
const content = readFile('path/to/file.md', testDir);
updateFile('path/to/file.md', 'new content', testDir);
const exists = fileExists('path/to/file.md', testDir);Split assertions into individual it blocks. Store the CLI result in a block-scoped let variable populated by beforeEach:
let result: Awaited<ReturnType<typeof runCli>>;
beforeEach(async () => {
result = await runCli('some-command', { apiKey, cwd: testDir });
});
it('succeeds', () => {
expect(result.returnCode).toBe(0);
});
it('displays the summary', () => {
expect(result.stdout).toContain('1 change submitted');
});For commands that require chained operations (e.g. install then modify then diff), nest describe blocks. Each level adds its own beforeEach that builds on the parent context:
describeWithUserSignedUp('diff command', (getContext) => {
beforeEach(async () => {
// setup: git repo + seed data + install
});
describe('when a change is submitted', () => {
beforeEach(async () => {
// modify file + run diff --submit
});
it('succeeds', () => { /* ... */ });
describe('when running diff again', () => {
beforeEach(async () => {
// run diff without --submit
});
it('excludes the already-submitted change', () => { /* ... */ });
});
});
});Before writing the test, ask the user: "Is the feature under test gated by a feature flag?"
If yes, the test must create a user whose email is at @packmind.com so the flag is enabled. This is controlled by the underFeatureFlag fixture option. Add .use({ underFeatureFlag: true }) immediately after the test wrapper import, before any describe or it blocks:
// When using testWithApi:
testWithApi.use({ underFeatureFlag: true });
// When using testWithUserSignedUp:
testWithUserSignedUp.use({ underFeatureFlag: true });Without this option, the fixture creates a user at @example.com, which does not have feature flags enabled, and the feature under test would not be accessible.
When the CLI command under test requires seeding a new type of API resource:
helpers/IPackmindGateway.ts. All exposed methods must be typed with Gateway<IXxxUseCase> or PublicGateway<IXxxUseCase> (imported from @packmind/types)helpers/gateways/NewResourceGateway.ts implementing the interfacehelpers/gateways/PackmindGateway.ts (reset on initializeWithApiKey)helpers/index.ts© PackmindHub, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in .agents/skills/cli-e2e-test-authoring of PackmindHub/packmind.
Open the folder on GitHubat commit 858ed50
CLI E2E Test Authoring next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| CLI E2E Test Authoring this skillPackmindHub/packmind | 317 | — | ~1.6k | Automated safety check: Pass | Apache-2.0 | |
| OpenHarness End-to-End EvalsHKUDS/OpenHarness | 16k | 1 repos | ~2.1k | Automated safety check: Notes | MIT | |
| Nemoclaw Maintainer Fix E2E FailuresNVIDIA/NemoClaw | 23k | — | ~2.6k | Automated safety check: Pass | Apache-2.0 | |
| Releaselycorp-jp/sim-use | 1.4k | — | ~1.7k | Automated safety check: Pass | Apache-2.0 | |
| Diff-Driven Smoke TestsSkyvern-AI/skyvern | 23k | — | ~5.2k | Automated safety check: Pass | AGPL-3.0 | |
| Test Pyramidkubernetes-sigs/agent-sandbox | 4.2k | — | ~1.6k | Automated safety check: Pass | Apache-2.0 |
HKUDS/OpenHarness
Validates OpenHarness features by running real multi-turn agent loops with live LLM calls against an unfamiliar codebase, checking actual tool execution.
NVIDIA/NemoClaw
Continuously maintain automatic NemoClaw main E2E results through coordinated repairs.
lycorp-jp/sim-use
Cut a sim-use release end-to-end. An agent skill from lycorp-jp/sim-use.
Skyvern-AI/skyvern
Reads your git diff, writes a handful of happy-path browser smoke tests, runs them with Skyvern or Chrome DevTools MCP and posts screenshot evidence to the PR.
kubernetes-sigs/agent-sandbox
Analyze the repo's unit and E2E tests and propose rebalancing toward a test pyramid — which E2E tests (or assertions inside them) can be covered by unit tests, which unit-level gaps genuinely need…
crowdin/crowdin-api-client-js
Run end-to-end tests against the real Crowdin or Crowdin Enterprise API using credentials from .env.
PackmindHub/packmind
Produce proof-of-execution demos of the Packmind CLI (packmind-cli) as terminal-styled images (colors and formatting preserved exactly), for embedding in a GitHub PR.
PackmindHub/packmind
Record polished UI demo videos and screenshots of a running web app using Playwright MCP — for client deliverables, release notes, feature walkthroughs, or bug repros.
PackmindHub/packmind
Guide for creating effective skills. An agent skill from PackmindHub/packmind.
PackmindHub/packmind
Audit Packmind end-user documentation (apps/doc/) for broken links, outdated CLI references, non-existent concepts, misleading information, and missing coverage.
PackmindHub/packmind
Execute the implementation plan produced by /feature-spec. An agent skill from PackmindHub/packmind.
PackmindHub/packmind
Review an implemented GitHub issue the way a senior Packmind engineer would — the human-judgment checks that ESLint, the TypeScript compiler, and e2e tests cannot catch (authorization scoping…
Works with
Categories
Guide for adding new end-to-end tests for the Packmind CLI. An agent skill from PackmindHub/packmind. CLI E2E Test Authoring is an agent skill from PackmindHub/packmind. Guide for adding new end-to-end tests for the Packmind CLI.
CLI E2E Test Authoring fits situations like: tasks that involve End-to-end testing.
Run `npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a claude-code`. Or copy the skill folder (.agents/skills/cli-e2e-test-authoring in PackmindHub/packmind) into .claude/skills/cli-e2e-test-authoring in your project. Claude Code loads it when a task matches its description.
Run `npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a codex`. Or copy the skill folder (.agents/skills/cli-e2e-test-authoring in PackmindHub/packmind) into .agents/skills/cli-e2e-test-authoring in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add PackmindHub/packmind --skill cli-e2e-test-authoring -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/cli-e2e-test-authoring, .gemini/skills/cli-e2e-test-authoring, .github/skills/cli-e2e-test-authoring and .opencode/skills/cli-e2e-test-authoring in your project.
SKILL.md names no scripts, command-line tools or credentials: CLI E2E Test Authoring is instructions for the agent only.
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
CLI E2E Test Authoring is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
About 1.6k tokens (SKILL.md is roughly 6.3k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with CLI E2E Test Authoring: OpenHarness End-to-End Evals (HKUDS/OpenHarness, 16k stars), Nemoclaw Maintainer Fix E2E Failures (NVIDIA/NemoClaw, 23k stars), Release (lycorp-jp/sim-use, 1.4k stars) and Diff-Driven Smoke Tests (Skyvern-AI/skyvern, 23k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
PackmindHub (a GitHub organization) maintains it in PackmindHub/packmind, which has 317 GitHub stars. The repository holds 35 skills in this directory. The repository was last updated on October 6, 2026.
Source: PackmindHub/packmind on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.