New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Official agent skill

Creating Online Evaluations

by PostHog in PostHog/posthog

Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified.

OfficialCustom licenceAuto-check passedAI & LLM Engineering

Install Creating Online Evaluations

skills CLI
$ npx skills add PostHog/posthog --skill creating-online-evaluations -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install PostHog/posthog creating-online-evaluations --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/PostHog/posthog.git skills-src && mkdir -p .claude/skills && cp -r skills-src/products/ai_observability/skills/creating-online-evaluations .claude/skills/creating-online-evaluations && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
creating-online-evaluations
GitHub stars
40k
Token cost
~6.7k tokens
SKILL.md length
3,295 words
Files
2 (incl. references)
Skills in repo
252
Repo updated
First seen
Licence
Custom licence

At a glance

Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified.

  • Works in 2 steps: Decide what to propose, then let the… → Build each accepted eval
  • The user wants evaluations that automatically score new generations
  • SKILL.md covers Tools, Phase 1 — Decide what to…, Phase 2 — Build each accepted… and Scoping with conditions, plus 3 more sections
  • Reaches us.posthog.com

What it does

Creating Online Evaluations is an agent skill from PostHog/posthog, published by the product's own GitHub organization. Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified. Use when the user wants evaluations that automatically score new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn these failures into evals". Covers letting the explored data decide how many evals to create, proposing that set for the user to pick, choosing the target and eval type (hog / llmjudge / sentiment)…

Its SKILL.md is about 6.7k tokens, which your agent loads only when the skill is triggered. The skill folder holds 2 other files, including reference files.

It sits in AI & LLM Engineering, covering LLM evaluation. It works with PostHog. The repository describes itself as: :hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error…

When your agent uses it

  • The user wants evaluations that automatically score new generations
  • Whole traces going forward — create an eval to catch X
  • Continuously check that responses do Y
  • Turn these failures into evals

Example prompts

  • “create an eval to catch X”
  • “continuously check that responses do Y”
  • “turn these failures into evals”
  • “/creating-online-evaluations”

Workflow steps

2 steps, taken from the step headings in SKILL.md.

  1. Decide what to propose, then let the user choose
  2. Build each accepted eval

What it can do on your machine

Read from SKILL.md and the folder at commit 10f9ad7. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md (its code samples are json and sql).

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Hosts in commands or code, which the agent is likely to contact:

    • us.posthog.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Creating Online Evaluations loads about 6.7k tokens when it runs, and up to ~9.5k if it reads all its reference files. Until then it costs about 255 tokens; SKILL.md has 3,295 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~255
When it runs · the whole SKILL.md, loaded when a task matches
~6.7k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~9.5k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

Its licence (Custom licence) doesn't allow us to republish the file, so here is its outline and opening line. It has 3,295 words (~6,680 tokens).

“An online evaluation automatically scores either each matching $ai_generation or the whole trace containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic, not from a guess or a generic metric like…”

— opening of SKILL.md by PostHog, Custom licence
name
creating-online-evaluations

Read the full SKILL.md on GitHub

Files

SKILL.md and 1 other file (references) in products/ai_observability/skills/creating-online-evaluations of PostHog/posthog.

  • SKILL.md
  • references/evaluation-payload.md.j2

Open the folder on GitHubat commit 10f9ad7

Used in 1 other repository

We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders. This page covers the copy in PostHog/posthog, which our catalogue first saw on October 7, 2026.

Compare with similar skills

Creating Online Evaluations next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Creating Online Evaluations compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Creating Online Evaluations this skillPostHog/posthog40k—~6.7kAutomated safety check: PassCustom licence
LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs13k8 repos~3kAutomated safety check: PassMIT
Azure AI Projects Python SDKmicrosoft/skills3.1k6 repos~2.8kAutomated safety check: PassMIT
Fine-Tuning ExpertJeffallan/claude-skills12k1 repos~1.7kAutomated safety check: PassMIT
Looperksimback/looper710—~2.7kAutomated safety check: NotesMIT
Hugging Face Local Model Evalshuggingface/skills11k2 repos~1.6kAutomated safety check: PassApache-2.0

Similar skills

  • LLM Benchmarking with lm-evaluation-harness

    Orchestra-Research/AI-Research-SKILLs

    Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.

    13k GitHub starsUsed in 8 repos~3k tokens
    AI & LLM EngineeringAuto-check passed

  • Azure AI Projects Python SDK

    microsoft/skills

    Official

    Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.

    3.1k GitHub starsUsed in 6 repos~2.8k tokens
    AI & LLM EngineeringAuto-check passed

  • Fine-Tuning Expert

    Jeffallan/claude-skills

    Guides LLM fine-tuning with LoRA and QLoRA through Hugging Face PEFT, from dataset validation and training checks to adapter merging, quantization and deployment.

    12k GitHub starsUsed in 1 repo~1.7k tokens
    AI & LLM EngineeringAuto-check passed

  • Looper

    ksimback/looper

    Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.

    710 GitHub stars~2.7k tokensUpdated 2 mo ago
    AI & LLM EngineeringAuto-check: notes

  • Hugging Face Local Model Evals

    huggingface/skills

    Official

    Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.

    11k GitHub starsUsed in 2 repos~1.6k tokens
    AI & LLM EngineeringAuto-check passed

  • Agent Eval Engineering

    langchain-ai/langchain-skills

    Official

    Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.

    1.3k GitHub stars~4k tokensUpdated 3 days ago
    AI & LLM EngineeringAuto-check passed

More from PostHog/posthog

All 252 skills in this repo

  • Authoring Log Alerts

    PostHog/posthog

    Official

    Author useful, low-noise log alerts on services in a PostHog project.

    40k GitHub stars~3k tokensUpdated today
    Auto-check passed

  • Autoresolving PR Conflicts

    PostHog/posthog

    Official

    Operating procedure for the conflict-autoresolver agent: sweep open PostHog/posthog PRs that conflict with master, resolve the trivial conflicts (generated artifacts deterministically, source…

    40k GitHub stars~4.2k tokensUpdated today
    Auto-check passed

  • Diagnosing Stacktrace Symbolication

    PostHog/posthog

    Official

    Help users debug PostHog Error Tracking stack-trace symbolication for any supported platform — JavaScript/TypeScript web, React Native (Hermes), Android (Proguard / R8), or iOS / macOS (dSYM).

    40k GitHub stars~2.2k tokensUpdated today
    Auto-check passed

  • Exploring Apm Traces

    PostHog/posthog

    Official

    Investigates distributed application performance using PostHog APM (OpenTelemetry span) data via MCP.

    40k GitHub stars~3.5k tokensUpdated today
    Auto-check passed

  • Exploring LLM Traces

    PostHog/posthog

    Official

    Debug and inspect LLM/AI agent traces using PostHog's MCP tools.

    40k GitHub stars~4.4k tokensUpdated today
    Auto-check passed

  • Investigate Metric

    PostHog/posthog

    Official

    Diagnose why a product metric changed (dropped, spiked, or plateaued) by orchestrating breakdowns, actors, paths, lifecycle, retention, and annotations queries.

    40k GitHub stars~1.9k tokensUpdated today
    Auto-check passed

Works with

Categories

Questions about Creating Online Evaluations

What does Creating Online Evaluations do?

Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified. Creating Online Evaluations is an agent skill from PostHog/posthog, published by the product's own GitHub organization. Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified.

When should I use Creating Online Evaluations?

Creating Online Evaluations fits situations like: the user wants evaluations that automatically score new generations; whole traces going forward — create an eval to catch X; continuously check that responses do Y; turn these failures into evals.

How do I install Creating Online Evaluations in Claude Code?

Run `npx skills add PostHog/posthog --skill creating-online-evaluations -a claude-code`. Or copy the skill folder (products/ai_observability/skills/creating-online-evaluations in PostHog/posthog) into .claude/skills/creating-online-evaluations in your project. Claude Code loads it when a task matches its description.

How do I install Creating Online Evaluations in Codex?

Run `npx skills add PostHog/posthog --skill creating-online-evaluations -a codex`. Or copy the skill folder (products/ai_observability/skills/creating-online-evaluations in PostHog/posthog) into .agents/skills/creating-online-evaluations in your project. Codex loads it when a task matches its description.

Can I use Creating Online Evaluations in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add PostHog/posthog --skill creating-online-evaluations -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/creating-online-evaluations, .gemini/skills/creating-online-evaluations, .github/skills/creating-online-evaluations and .opencode/skills/creating-online-evaluations in your project.

What does Creating Online Evaluations need to run?

SKILL.md names no scripts, command-line tools or credentials: Creating Online Evaluations is instructions for the agent only.

Does Creating Online Evaluations access the network?

SKILL.md names 1 domain. In commands or code: us.posthog.com; the agent is likely to contact it when it follows the instructions. This is read from the text; nothing was executed.

Is Creating Online Evaluations safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Creating Online Evaluations use?

Creating Online Evaluations has a licence file (the repository's licence) that doesn't match a standard licence. Read it on GitHub before reusing the skill.

How many tokens does Creating Online Evaluations use?

About 6.7k tokens (SKILL.md is roughly 27k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 2.9k tokens, read only when the agent opens those files.

What are the alternatives to Creating Online Evaluations?

Skills that share tags, products or a category with Creating Online Evaluations: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Azure AI Projects Python SDK (microsoft/skills, 3.1k stars), Fine-Tuning Expert (Jeffallan/claude-skills, 12k stars) and Looper (ksimback/looper, 710 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Creating Online Evaluations?

PostHog (a GitHub organization, an official publisher) maintains it in PostHog/posthog, which has 40,182 GitHub stars. The repository holds 252 skills in this directory. The repository was last updated on October 8, 2026.

Source: PostHog/posthog on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.