OpenHarness End-to-End Evals
HKUDS/OpenHarness
Validates OpenHarness features by running real multi-turn agent loops with live LLM calls against an unfamiliar codebase, checking actual tool execution.
Run integration tests to verify that extension components work together correctly.
$ npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install microsoft/vscode-python-environments run-integration-tests --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/microsoft/vscode-python-environments.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.github/skills/run-integration-tests .claude/skills/run-integration-tests && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "run-integration-tests" agent skill from https://github.com/microsoft/vscode-python-environments/tree/main/.github/skills/run-integration-tests into .claude/skills/run-integration-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-integration-tests", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/microsoft/vscode-python-environments/tree/main/.github/skills/run-integration-testsType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install microsoft/vscode-python-environments run-integration-tests --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/microsoft/vscode-python-environments.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.github/skills/run-integration-tests .agents/skills/run-integration-tests && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "run-integration-tests" agent skill from https://github.com/microsoft/vscode-python-environments/tree/main/.github/skills/run-integration-tests into .agents/skills/run-integration-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-integration-tests", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install microsoft/vscode-python-environments run-integration-tests --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/microsoft/vscode-python-environments.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.github/skills/run-integration-tests .cursor/skills/run-integration-tests && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "run-integration-tests" agent skill from https://github.com/microsoft/vscode-python-environments/tree/main/.github/skills/run-integration-tests into .cursor/skills/run-integration-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-integration-tests", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/microsoft/vscode-python-environments.git --path .github/skills/run-integration-tests--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install microsoft/vscode-python-environments run-integration-tests --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/microsoft/vscode-python-environments.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.github/skills/run-integration-tests .gemini/skills/run-integration-tests && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "run-integration-tests" agent skill from https://github.com/microsoft/vscode-python-environments/tree/main/.github/skills/run-integration-tests into .gemini/skills/run-integration-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-integration-tests", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install microsoft/vscode-python-environments run-integration-testsInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/microsoft/vscode-python-environments.git skills-src && mkdir -p .github/skills && cp -r skills-src/.github/skills/run-integration-tests .github/skills/run-integration-tests && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "run-integration-tests" agent skill from https://github.com/microsoft/vscode-python-environments/tree/main/.github/skills/run-integration-tests into .github/skills/run-integration-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-integration-tests", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install microsoft/vscode-python-environments run-integration-tests --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/microsoft/vscode-python-environments.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.github/skills/run-integration-tests .opencode/skills/run-integration-tests && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "run-integration-tests" agent skill from https://github.com/microsoft/vscode-python-environments/tree/main/.github/skills/run-integration-tests into .opencode/skills/run-integration-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "run-integration-tests", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
run-integration-testsRun integration tests to verify that extension components work together correctly.
Run Integration Tests is an agent skill from microsoft/vscode-python-environments, published by the product's own GitHub organization. Run integration tests to verify that extension components work together correctly. Use this after modifying component interactions or event handling.
Its SKILL.md is about 1.1k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Testing & QA, covering Integration testing. It works with Visual Studio Code and npm. The repository describes itself as: VS Code extension for Python environment and package management. The licence is MIT.
2 steps, taken from the step headings in SKILL.md.
Read from SKILL.md and the folder at commit 4faeedf. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
npmFrom the folder's file list and the shell code blocks in SKILL.md.
No URLs in SKILL.md. Its commands use npm, which can reach the network depending on how they are called.
From URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Run Integration Tests loads about 1.1k tokens when it runs. Until then it costs about 43 tokens; SKILL.md has 285 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from microsoft/vscode-python-environments at commit 4faeedf, republished under its MIT licence (© microsoft). 285 words, ~1,072 tokens.
.claude/skills/run-integration-tests/SKILL.md (or your agent's skills folder).Run integration tests to verify that multiple components (managers, API, settings) work together correctly.
| Action | Command |
|---|---|
| Run all integration tests | npm run compile && npm run compile-tests && npm run integration-test |
| Run specific test | npm run integration-test -- --grep "manager" |
| Debug in VS Code | Debug panel → "Integration Tests" → F5 |
Integration tests run in a real VS Code instance but focus on component interactions:
They're faster than E2E (which test full workflows) but more thorough than smoke tests.
npm run compile && npm run compile-tests && npm run integration-testPass:
Integration: Environment Manager + API
✓ API reflects manager state after refresh
✓ Different scopes return appropriate environments
✓ Environment objects have consistent structure
3 passing (25s)Fail: Check error message and see Debugging section.
| Error | Cause | Fix |
|---|---|---|
API not available | Extension activation failed | Check Debug Console |
Event not fired | Event wiring issue | Check event registration |
State mismatch | Components out of sync | Add logging, check update paths |
Timeout | Async operation stuck | Check for deadlocks |
For detailed debugging: Debug panel → "Integration Tests" → F5
Create files in src/test/integration/ with pattern *.integration.test.ts:
import * as assert from 'assert';
import * as vscode from 'vscode';
import { waitForCondition, TestEventHandler } from '../testUtils';
import { ENVS_EXTENSION_ID } from '../constants';
suite('Integration: [Component A] + [Component B]', function () {
this.timeout(120_000);
let api: ExtensionApi;
suiteSetup(async function () {
const extension = vscode.extensions.getExtension(ENVS_EXTENSION_ID);
assert.ok(extension, 'Extension not found');
if (!extension.isActive) await extension.activate();
api = extension.exports;
});
test('[Interaction test]', async function () {
// Test component interaction
});
});| File | Purpose |
|---|---|
src/test/integration/envManagerApi.integration.test.ts | Manager + API tests |
src/test/integration/index.ts | Test runner entry point |
src/test/testUtils.ts | Utilities (waitForCondition, TestEventHandler) |
npm run compile (webpack) before tests, not just npm run compile-tests (tsc)npm run compile before tests.vscode-test/)© microsoft, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in .github/skills/run-integration-tests of microsoft/vscode-python-environments.
Open the folder on GitHubat commit 4faeedf
Run Integration Tests next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Run Integration Tests this skillmicrosoft/vscode-python-environments | 141 | — | ~1.1k | Automated safety check: Pass | MIT | |
| OpenHarness End-to-End EvalsHKUDS/OpenHarness | 16k | 1 repos | ~2.1k | Automated safety check: Notes | MIT | |
| Interactive CLI Testing With tui-testslopus/happy | 24k | — | ~603 | Automated safety check: Pass | MIT | |
| Promotion Branches E2E Testhardisgroupcom/sfdx-hardis | 400 | — | ~4.7k | Automated safety check: Notes | AGPL-3.0 | |
| Running TestsNangoHQ/nango | 13k | — | ~876 | Automated safety check: Pass | Custom licence | |
| Blockless Extension E2EFreakStudioCN/mpy-hardware-extension | 118 | — | ~1.2k | Automated safety check: Notes | Custom licence |
HKUDS/OpenHarness
Validates OpenHarness features by running real multi-turn agent loops with live LLM calls against an unfamiliar codebase, checking actual tool execution.
slopus/happy
Tests interactive CLI and TUI programs with Microsoft's tui-test, driving prompts, arrow keys and screen output in a real pseudo-terminal.
hardisgroupcom/sfdx-hardis
Runs a full end-to-end test of sfdx-hardis promotion branches and backpromote against real Salesforce orgs and a throwaway repository, then writes a report.
NangoHQ/nango
A skill your agent uses when running tests in the Nango monorepo - knows unit vs integration configs, vitest commands, Docker setup, and common test patterns
FreakStudioCN/mpy-hardware-extension
Run and debug the Blockless VS Code extension release gate: CI-equivalent API and extension tests, V0 protocol smoke, live DeepSeek full-stack e2e, VSIX packaging, local reinstall, direct…
adeze/raindrop-mcp
MCP Testing Strategies with Vitest, Inspector, and Integration Tests
microsoft/vscode-python-environments
Critical patterns for cross-platform path handling in this VS Code extension.
microsoft/vscode-python-environments
Environment manager-specific discovery patterns and known issues.
microsoft/vscode-python-environments
Run E2E tests to verify complete user workflows like environment discovery, creation, and selection.
microsoft/vscode-python-environments
Run smoke tests to verify extension functionality in a real VS Code environment.
microsoft/vscode-python-environments
Debug a failing test using an iterative logging approach, then clean up and document the learning.
microsoft/vscode-python-environments
Generate a codebase health snapshot for technical debt tracking and planning.
Works with
Categories
Run integration tests to verify that extension components work together correctly. Run Integration Tests is an agent skill from microsoft/vscode-python-environments, published by the product's own GitHub organization. Run integration tests to verify that extension components work together correctly.
Run Integration Tests fits situations like: tasks that involve Integration testing.
Run `npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a claude-code`. Or copy the skill folder (.github/skills/run-integration-tests in microsoft/vscode-python-environments) into .claude/skills/run-integration-tests in your project. Claude Code loads it when a task matches its description.
Run `npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a codex`. Or copy the skill folder (.github/skills/run-integration-tests in microsoft/vscode-python-environments) into .agents/skills/run-integration-tests in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add microsoft/vscode-python-environments --skill run-integration-tests -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/run-integration-tests, .gemini/skills/run-integration-tests, .github/skills/run-integration-tests and .opencode/skills/run-integration-tests in your project.
Going by SKILL.md and its folder, Run Integration Tests needs the command-line tools its instructions call (npm).
SKILL.md contains no URLs. Its commands use npm, which can reach the network depending on how they are called. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Run Integration Tests is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
About 1.1k tokens (SKILL.md is roughly 4.3k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Run Integration Tests: OpenHarness End-to-End Evals (HKUDS/OpenHarness, 16k stars), Interactive CLI Testing With tui-test (slopus/happy, 24k stars), Promotion Branches E2E Test (hardisgroupcom/sfdx-hardis, 400 stars) and Running Tests (NangoHQ/nango, 13k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
microsoft (a GitHub organization, an official publisher) maintains it in microsoft/vscode-python-environments, which has 141 GitHub stars. The repository holds 9 skills in this directory. The repository was last updated on October 7, 2026.
Source: microsoft/vscode-python-environments on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.