Official agent skill

Writing Evals

OfficialCustom licenceAuto-check: notesAI & LLM Engineering

Install Writing Evals

skills CLI

$ npx skills add PostHog/posthog --skill writing-evals -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI

$ gh skill install PostHog/posthog writing-evals --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy

$ git clone --depth 1 https://github.com/PostHog/posthog.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/writing-evals .claude/skills/writing-evals && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Paste into Claude Code

Install the "writing-evals" agent skill from https://github.com/PostHog/posthog/tree/master/.agents/skills/writing-evals into .claude/skills/writing-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "writing-evals", then confirm the skill loads.

Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

$skill-installer

$skill-installer install https://github.com/PostHog/posthog/tree/master/.agents/skills/writing-evals

Type this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.

skills CLI

$ npx skills add PostHog/posthog --skill writing-evals -a codex

Project install goes to .agents/skills/; add -g for ~/.codex/skills/.

GitHub CLI

$ gh skill install PostHog/posthog writing-evals --agent codex

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/PostHog/posthog.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.agents/skills/writing-evals .agents/skills/writing-evals && rm -rf skills-src

Use ~/.agents/skills/ instead of .agents/skills for a personal install.

Codex skills documentation · loads skills from .agents/skills/

Paste into Codex

Install the "writing-evals" agent skill from https://github.com/PostHog/posthog/tree/master/.agents/skills/writing-evals into .agents/skills/writing-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "writing-evals", then confirm the skill loads.

Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add PostHog/posthog --skill writing-evals -a cursor

Project install goes to .agents/skills/; add -g for ~/.cursor/skills/.

GitHub CLI

$ gh skill install PostHog/posthog writing-evals --agent cursor

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/PostHog/posthog.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.agents/skills/writing-evals .cursor/skills/writing-evals && rm -rf skills-src

Use ~/.cursor/skills/ instead of .cursor/skills for a personal install.

Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/

Paste into Cursor

Install the "writing-evals" agent skill from https://github.com/PostHog/posthog/tree/master/.agents/skills/writing-evals into .cursor/skills/writing-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "writing-evals", then confirm the skill loads.

Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

gemini skills install

$ gemini skills install https://github.com/PostHog/posthog.git --path .agents/skills/writing-evals

--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.

skills CLI

$ npx skills add PostHog/posthog --skill writing-evals -a gemini-cli

Project install goes to .agents/skills/; add -g for ~/.gemini/skills/.

GitHub CLI

$ gh skill install PostHog/posthog writing-evals --agent gemini-cli

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/PostHog/posthog.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.agents/skills/writing-evals .gemini/skills/writing-evals && rm -rf skills-src

Use ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.

Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/

Paste into Gemini CLI

Install the "writing-evals" agent skill from https://github.com/PostHog/posthog/tree/master/.agents/skills/writing-evals into .gemini/skills/writing-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "writing-evals", then confirm the skill loads.

Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

GitHub CLI

$ gh skill install PostHog/posthog writing-evals

Installs for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).

skills CLI

$ npx skills add PostHog/posthog --skill writing-evals -a github-copilot

Project install goes to .agents/skills/; add -g for ~/.copilot/skills/.

Manual copy

$ git clone --depth 1 https://github.com/PostHog/posthog.git skills-src && mkdir -p .github/skills && cp -r skills-src/.agents/skills/writing-evals .github/skills/writing-evals && rm -rf skills-src

Use ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.

GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/

Paste into GitHub Copilot

Install the "writing-evals" agent skill from https://github.com/PostHog/posthog/tree/master/.agents/skills/writing-evals into .github/skills/writing-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "writing-evals", then confirm the skill loads.

GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

skills CLI

$ npx skills add PostHog/posthog --skill writing-evals -a opencode

OpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.

GitHub CLI

$ gh skill install PostHog/posthog writing-evals --agent opencode

Project scope by default (.agents/skills/); add --scope user for a personal install.

Manual copy

$ git clone --depth 1 https://github.com/PostHog/posthog.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.agents/skills/writing-evals .opencode/skills/writing-evals && rm -rf skills-src

Use ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.

OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/

Paste into OpenCode

Install the "writing-evals" agent skill from https://github.com/PostHog/posthog/tree/master/.agents/skills/writing-evals into .opencode/skills/writing-evals/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "writing-evals", then confirm the skill loads.

OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.

Facts

Skill name: writing-evals
GitHub stars: 40k
Token cost: ~4k tokens
SKILL.md length: 1,686 words
Files: 3 (incl. references)
Skills in repo: 252
Repo updated: Oct 8, 2026
First seen: Oct 7, 2026
Licence: Custom licence

At a glance

Works in 4 steps: hogli evals --list shows your suite… → Run your new case alone: hogli evals… → Unit-test scorer and synthesizer logic… → …
Changing eval suites
SKILL.md covers Writing a suite, One-shot suites, Case anatomy and Seeding data: seeders and…, plus 3 more sections
Calls cargo; needs BRAINTRUST_API_KEY and LLM_GATEWAY_ANTHROPIC_API_KEY

What it does

Writing Evals is an agent skill from PostHog/posthog, published by the product's own GitHub organization. Teaches how to write and run evals on the products/posthogai/evalharness/ harness — sandboxed agent suites that execute the real coding agent in a Docker or Modal sandbox against a seeded Hedgebox project, and one-shot suites that score a single in-process model invocation per case. Use when adding or changing eval suites, cases, scorers, seeders, or synthesizers under products/posthogai/evals/ or products//evals/, when touching the harness under products/posthogai/evalharness/, or when running or debugging those…

Its SKILL.md is about 4k tokens, which your agent loads only when the skill is triggered. The skill folder holds 3 other files, including reference files (for example `references/authoring-reference.md` and `references/running-evals.md`).

It sits in AI & LLM Engineering, covering LLM evaluation. It works with PostHog, pytest and Docker. The repository describes itself as: :hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error…

When your agent uses it

Changing eval suites
Synthesizers under products/posthogai/evals/
Products//evals/
Touching the harness under products/posthogai/evalharness/

Example prompts

“Use the writing-evals skill to teach how to write and run evals on the products/posthogai/evalharness/ harness — sandboxed agent suites that execute…”
“/writing-evals”

Requirements

Python 3
Docker
A credential in BRAINTRUST_API_KEY
A credential in LLM_GATEWAY_ANTHROPIC_API_KEY

Workflow steps

4 steps, taken from the first numbered list in SKILL.md.

hogli evals --list shows your suite (this also import-checks every eval module).
Run your new case alone: hogli evals --eval . Read the transcript named on the final line, then the case summary and scorer metadata, not…
Unit-test scorer and synthesizer logic where it's cheap — but not inside an evals/ tree: keep suite dirs free of conftest.py and pytest…
Prompts and expected values use exact Hedgebox taxonomy names, relative date ranges (-30d, -8w), and shape-based assertions — never…

What it can do on your machine

Read from SKILL.md and the folder at commit 10f9ad7. It shows what the files ask for, not the result of running them.

Tool permissions
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Shell commands in SKILL.md call:
- cargo
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names these keys or tokens, usually read from environment variables:
- BRAINTRUST_API_KEY
- LLM_GATEWAY_ANTHROPIC_API_KEY
- LLM_GATEWAY_OPENAI_API_KEY
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Writing Evals loads about 4k tokens when it runs, and up to ~8.3k if it reads all its reference files. Until then it costs about 202 tokens; SKILL.md has 1,686 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it: ~202
When it runs · the whole SKILL.md, loaded when a task matches: ~4k
With references · SKILL.md plus every file in references/, read only if the agent opens them: ~8.3k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check: notes

The automated check noted patterns worth knowing about, such as sudo or a known installer.

NoteMentions a .env fileSKILL.md:161

- Env is loaded automatically (`.env` by the harness, `.env.local`/`.env.development`/`.env.services` by hogli) and a pr

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

Its licence (Custom licence) doesn't allow us to republish the file, so here is its outline and opening line. It has 1,686 words (~4,047 tokens).

“The harness under products/posthog_ai/eval_harness/ runs two kinds of suites: sandboxed suites execute the real coding agent inside a real sandbox against a seeded Hedgebox project and score what it did, and one-shot suites score a single in-process model invocation per…”

— opening of SKILL.md by PostHog, Custom licence

name: writing-evals

Read the full SKILL.md on GitHub

Files

SKILL.md and 2 other files (references) in .agents/skills/writing-evals of PostHog/posthog.

SKILL.md
references/authoring-reference.md
references/running-evals.md

Open the folder on GitHubat commit 10f9ad7

Used in 1 other repository

We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders. This page covers the copy in PostHog/posthog, which our catalogue first saw on October 7, 2026.

PostHog/posthog-foss.agents/skills/writing-evals

Compare with similar skills

Writing Evals next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Writing Evals compared with similar skills
Skill	Stars	Used in	Tokens	Auto-check	Licence	Repo updated
Writing Evals this skillPostHog/posthog	40k	—	~4k	Automated safety check: Notes	Custom licence	today
Nemo Evaluator SDKOrchestra-Research/AI-Research-SKILLs	13k	3 repos	~3.1k	Automated safety check: Pass	MIT	3 mo ago
Code Model Evaluation HarnessOrchestra-Research/AI-Research-SKILLs	13k	5 repos	~2.9k	Automated safety check: Pass	MIT	3 mo ago
VLM BCQ Gap AnalysisNVIDIA/skills	3.5k	—	~1.3k	Automated safety check: Notes	Apache-2.0	yesterday
Multimodal Dataprep Devopen-edge-platform/edge-ai-libraries	169	—	~1.3k	Automated safety check: Pass	Apache-2.0	today
Suite ConverterMargin-Lab/evals	161	—	~3.7k	Automated safety check: Pass	AGPL-3.0	2 mo ago

Similar skills

Nemo Evaluator SDK
Orchestra-Research/AI-Research-SKILLs
Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution.
13k GitHub starsUsed in 3 repos~3.1k tokens
AI & LLM EngineeringAuto-check passed
Code Model Evaluation Harness
Orchestra-Research/AI-Research-SKILLs
Benchmarks code generation models with the BigCode Evaluation Harness across HumanEval, MBPP, MultiPL-E and other suites using pass@k metrics.
13k GitHub starsUsed in 5 repos~2.9k tokens
AI & LLM EngineeringAuto-check passed
VLM BCQ Gap Analysis
NVIDIA/skills
Official
Compares a vision-language model's yes/no predictions with ground truth and writes the false-positive and false-negative cases to a JSONL file with a summary report.
3.5k GitHub stars~1.3k tokensUpdated yesterday
AI & LLM EngineeringAuto-check: notes
Multimodal Dataprep Dev
open-edge-platform/edge-ai-libraries
Develop and debug the Multimodal DataPrep microservice: its FastAPI media endpoints, in-process embedding pipeline, batch jobs, object detection, telemetry and Metrics Manager publishing, and…
169 GitHub stars~1.3k tokensUpdated today
Backend & APIsAuto-check passed
Suite Converter
Margin-Lab/evals
Converts test suites from external eval frameworks into the Margin Eval suite format.
161 GitHub stars~3.7k tokensUpdated 2 mo ago
Testing & QAAuto-check passed
Clean Code Refactorer
fike/fastapi-blog
Skill for identifying code smells and refactoring code using Clean Code, SOLID, and DRY principles.
101 GitHub stars~402 tokensUpdated yesterday
DevelopmentAuto-check passed

Questions about Writing Evals

What does Writing Evals do?

Teaches how to write and run evals on the products/posthogai/evalharness/ harness — sandboxed agent suites that execute the real coding agent in a Docker or Modal sandbox against a seeded Hedgebox…. Writing Evals is an agent skill from PostHog/posthog, published by the product's own GitHub organization. Teaches how to write and run evals on the products/posthogai/evalharness/ harness — sandboxed agent suites that execute the real coding agent in a Docker or Modal sandbox against a seeded Hedgebox project, and one-shot suites that score a single in-process model invocation per case.

When should I use Writing Evals?

Writing Evals fits situations like: changing eval suites; synthesizers under products/posthogai/evals/; products//evals/; touching the harness under products/posthogai/evalharness/.

How do I install Writing Evals in Claude Code?

Run `npx skills add PostHog/posthog --skill writing-evals -a claude-code`. Or copy the skill folder (.agents/skills/writing-evals in PostHog/posthog) into .claude/skills/writing-evals in your project. Claude Code loads it when a task matches its description.

How do I install Writing Evals in Codex?

Run `npx skills add PostHog/posthog --skill writing-evals -a codex`. Or copy the skill folder (.agents/skills/writing-evals in PostHog/posthog) into .agents/skills/writing-evals in your project. Codex loads it when a task matches its description.

Can I use Writing Evals in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add PostHog/posthog --skill writing-evals -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/writing-evals, .gemini/skills/writing-evals, .github/skills/writing-evals and .opencode/skills/writing-evals in your project.

What does Writing Evals need to run?

Going by SKILL.md and its folder, Writing Evals needs the command-line tools its instructions call (cargo) and credentials named BRAINTRUST_API_KEY, LLM_GATEWAY_ANTHROPIC_API_KEY and LLM_GATEWAY_OPENAI_API_KEY. Our summary lists: Python 3; Docker; A credential in BRAINTRUST_API_KEY; A credential in LLM_GATEWAY_ANTHROPIC_API_KEY.

Does Writing Evals access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Writing Evals safe to install?

Our automated static check of SKILL.md found notes only (mentions a .env file), nothing it rates as a warning. It is not a guarantee. Review the folder before installing.

What licence does Writing Evals use?

Writing Evals has a licence file (the repository's licence) that doesn't match a standard licence. Read it on GitHub before reusing the skill.

How many tokens does Writing Evals use?

About 4k tokens (SKILL.md is roughly 16k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 4.2k tokens, read only when the agent opens those files.

What are the alternatives to Writing Evals?

Skills that share tags, products or a category with Writing Evals: Nemo Evaluator SDK (Orchestra-Research/AI-Research-SKILLs, 13k stars), Code Model Evaluation Harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), VLM BCQ Gap Analysis (NVIDIA/skills, 3.5k stars) and Multimodal Dataprep Dev (open-edge-platform/edge-ai-libraries, 169 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Writing Evals?

PostHog (a GitHub organization, an official publisher) maintains it in PostHog/posthog, which has 40,182 GitHub stars. The repository holds 252 skills in this directory. The repository was last updated on October 8, 2026.

Source: PostHog/posthog on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.

Writing Evals

Install Writing Evals

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Files

Used in 1 other repository

Compare with similar skills

Similar skills

Nemo Evaluator SDK

Code Model Evaluation Harness

VLM BCQ Gap Analysis

Multimodal Dataprep Dev

Suite Converter

Clean Code Refactorer

More from PostHog/posthog

Authoring Log Alerts

Autoresolving PR Conflicts

Diagnosing Stacktrace Symbolication

Exploring Apm Traces

Exploring LLM Traces

Investigate Metric

Questions about Writing Evals

Install Writing Evals

Facts

At a glance

What it does

When your agent uses it

Example prompts

Requirements

Workflow steps

What it can do on your machine

Tool permissions

Runs code

Network

Credentials

Context cost

Safety

SKILL.md

Files

Used in 1 other repository

Compare with similar skills

Similar skills

More from PostHog/posthog

Related

Questions about Writing Evals